Data synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to a data synchronization method and device, computer equipment and a storage medium. The method is used for synchronizing data of a database to a data warehouse, the database is a relational database, and the data warehouse is a non-relational database. The method comprises the following steps: extracting incremental logs in a database in real time; analyzing the incremental log according to preset log analysis configuration information to obtain at least one piece of data change information; extracting first data from a database and extracting second data from a data warehouse according to the data change information; comparing the first data with the second data; and when the first data and the second data are inconsistent, executing a preset data recovery strategy according to the data difference type of the first data and the second data. According to the method and the device, an automatic data synchronization process can be realized, the real-time performance of data synchronization is improved, and the resource consumption required by data synchronization is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method, device, computer equipment and storage medium. Background Art

[0002] With the rapid growth of data volume and the increasing demand for real-time data analysis, data warehouses, as key components of big data processing, are particularly important for ensuring data consistency. However, traditional data synchronization methods, such as regular batch extraction or nightly synchronization, often fail to reflect the latest data changes in the database in real time. This may cause the data in the data warehouse to lag behind the data in the database, thus affecting real-time analysis and decision-making based on the data warehouse. Traditional data synchronization methods may require full synchronization or regular resynchronization of the entire data set, which results in a large amount of data transmission and computing resource consumption. Summary of the invention

[0003] In view of the above-mentioned deficiencies or shortcomings, the present application provides a data synchronization method, apparatus, computer device and storage medium. The embodiments of the present application can realize an automated data synchronization process, improve the real-time performance of data synchronization, and reduce the resource consumption required for data synchronization.

[0004] The present application provides a data synchronization method according to a first aspect. In some embodiments, the method is used to synchronize data from a database to a data warehouse, where the database is a relational database and the data warehouse is a non-relational database. The method includes:

[0005] Extract incremental logs from the database in real time;

[0006] Parse the incremental log according to the preset log parsing configuration information to obtain at least one piece of data change information;

[0007] Extracting first data from a database and extracting second data from a data warehouse according to the data change information;

[0008] comparing the first data with the second data;

[0009] When the first data and the second data are inconsistent, a preset data repair strategy is executed according to the data difference type between the first data and the second data.

[0010] In some embodiments, the incremental log of the database is obtained in real time, and the incremental log is parsed according to the preset log parsing configuration information to obtain at least one piece of data change information, including:

[0011] The incremental log synchronization program is used to extract the incremental logs of the database in real time, convert the format of the extracted incremental logs from binary format to a predefined JSON format, and save them to the message queue;

[0012] The incremental log is extracted from the message queue through the data consumption program, and the extracted incremental log is parsed according to the preset log parsing configuration information to obtain at least one piece of data change information.

[0013] In some embodiments, converting the format of the extracted incremental log from a binary format to a predefined JSON format and saving it to a message queue includes:

[0014] Convert the format of the extracted incremental log from binary format to a predefined JSON format, serialize the incremental log in JSON format to obtain a serialized incremental log in binary format, and transfer the serialized incremental log to a message queue for storage;

[0015] Extract incremental logs from the message queue through the data consumer program, including:

[0016] The serialized incremental log is consumed from the message queue through the data consumer program, and the serialized incremental log is deserialized to obtain the incremental log in JSON format.

[0017] In some embodiments, the log parsing configuration information includes a data table identifier, a field identifier, and a data change type that need to be filtered;

[0018] Parse the extracted incremental log according to the preset log parsing configuration information to obtain at least one piece of data change information, including:

[0019] The extracted incremental log is parsed by a log parsing algorithm, and the parsed data change information is put into a change information set; each piece of data change information includes a data table identifier, a field identifier, and a data change type;

[0020] Traversing the data change information in the change information set; in the traversal process, comparing the data table identifier, field identifier and data change type included in the traversed data change information with the data table identifier, field identifier and data change type included in the log parsing configuration information, respectively; when the data table identifier included in the traversed data change information is consistent with the data table identifier included in the log parsing configuration information, or the field identifier included in the traversed data change information is consistent with the field identifier included in the log parsing configuration information, or the data change type included in the traversed data change information is consistent with the data change type included in the log parsing configuration information, the traversed data change information is removed from the change information set;

[0021] After the traversal is completed, the remaining data change information in the change information set is taken as at least one piece of data change information.

[0022] In some embodiments, the data change information includes a data table identifier and a field identifier; extracting the first data from the database and extracting the second data from the data warehouse according to the data change information includes:

[0023] Querying the database for data matching the data table identifier and the field identifier included in the data change information, and using the query result as the first data;

[0024] The data warehouse is queried for data matching the data table identifier and the field identifier included in the data change information, and the query result is used as the second data.

[0025] In some embodiments, executing a preset data repair strategy according to the data difference type between the first data and the second data includes:

[0026] When the first data is not empty and the second data is empty, performing data completion processing in the data warehouse according to the data table identifier and the field identifier included in the first data and the data change information;

[0027] When both the first data and the second data are not empty, and the first data and the second data are not the same, performing data repair processing in the data warehouse according to the data table identifier and the field identifier included in the first data and the data change information;

[0028] When the first data is empty and the second data is not empty, data deletion processing is performed in the data warehouse according to the data table identifier and the field identifier included in the data change information.

[0029] In some embodiments, the method further comprises:

[0030] Monitor the software and hardware resource status and pre-specified performance indicators of the server where the database is located;

[0031] Determine whether anomalies or errors occur based on real-time monitoring data of software and hardware resource status and performance indicators;

[0032] When an abnormality or error is detected, an alarm is issued.

[0033] According to a second aspect, the present application provides a data synchronization device. In some embodiments, the device is used to synchronize data of a database to a data warehouse, where the database is a relational database and the data warehouse is a non-relational database. The device includes:

[0034] Incremental log extraction module, used to extract incremental logs in the database in real time;

[0035] A log parsing module, used to parse the incremental log according to preset log parsing configuration information to obtain at least one piece of data change information;

[0036] A data extraction module, used to extract first data from a database and second data from a data warehouse according to data change information;

[0037] A comparison module, used for comparing the first data with the second data;

[0038] The data repair module is used to execute a preset data repair strategy according to the data difference type between the first data and the second data when the first data and the second data are inconsistent.

[0039] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data synchronization method provided in any of the above embodiments when executing the computer program.

[0040] According to a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data synchronization method provided in any of the above embodiments are implemented.

[0041] In the above embodiment of the present application, the data synchronization device extracts the incremental log in the database in real time, analyzes the incremental log according to the preset log analysis configuration information, obtains at least one data change information, and for each data change information, extracts the first data from the database according to the data change information and extracts the second data from the data warehouse, compares the first data with the second data, and when the first data and the second data are inconsistent, executes the preset data repair strategy according to the data difference type of the first data and the second data. Among them, by capturing the incremental log of the database in real time, it can ensure that the data in the data warehouse is consistent with the data in the database in real time, thereby solving the problem of insufficient real-time performance of the traditional method; when synchronizing, only the data that has changed in the database is synchronized, rather than the full amount of data, so that the amount of data transmission and computing resource consumption in the data synchronization process can be significantly reduced, and the efficiency of data synchronization can be improved. In addition, during the data synchronization process, due to network delays, data loss, repeated transmission or data write errors, the data in the data warehouse may be inconsistent with the data in the database, and the embodiment of the present application can accurately capture each data change in the database and compare and verify it with the data in the data warehouse, so that the accuracy of data synchronization can be ensured, and the risk of data inconsistency can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a data synchronization method provided by the present application according to one or more embodiments;

[0043] Figure 2 A schematic diagram of a data synchronization architecture provided by the present application according to one or more embodiments;

[0044] Figure 3 A structural block diagram of a data synchronization device provided by the present application according to one or more embodiments;

[0045] Figure 4 This is a diagram of the internal structure of a computer device provided by the present application according to one or more embodiments. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0047] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0048] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0049] In response to the deficiencies or defects of the prior art, the present application provides a data synchronization method, which is used to synchronize data in a database to a data warehouse. The method can ensure that the data in the data warehouse remains consistent with the data in the database in real time by capturing the incremental log of the database in real time, thereby solving the problem of insufficient real-time performance of traditional methods. During synchronization, only the data that has changed in the database is synchronized, rather than the full amount of data. This can significantly reduce the amount of data transmission and computing resource consumption during the data synchronization process, thereby improving the efficiency of data synchronization. In addition, during the data synchronization process, due to network delays, data loss, repeated transmission or data writing errors, the data in the data warehouse may be inconsistent with the data in the database. This method can accurately capture every data change in the database and compare and verify it with the data in the data warehouse. This can ensure the accuracy of data synchronization and reduce the risk of data inconsistency. In some embodiments, the method includes: Figure 1 The steps shown are described in detail below.

[0050] S110: extracting incremental logs in the database in real time.

[0051] In this embodiment, the data synchronization method can be applied to a data synchronization device, which can be a server for synchronizing data in a database to a data warehouse so that the data of the two remain consistent. The database is a relational database, such as a MySQL database, and the data warehouse is a non-relational database, such as ElasticSearch, Mongo database, etc.

[0052] The database log is the operation sequence of all update activities in the database system. It records any update operation of the data (such as adding, modifying, deleting), including the command of the relevant operation, the execution time, the update of the data and other information. The database log is an important means to ensure the normal operation of the system and maintain data consistency. It is of great significance for data recovery, auditing and troubleshooting.

[0053] In this embodiment, the data synchronization device captures the change log information of the database as an incremental log through the database log. The data synchronization device can extract the incremental log of the database in real time to ensure the real-time nature of data synchronization. During data synchronization, only incremental data is synchronized, which can significantly reduce the amount of data transmission and computing resource consumption.

[0054] Furthermore, the data synchronization method of this embodiment supports multiple database types and multiple data warehouse types, and can be flexibly configured and used according to actual needs, so that as the business develops and the amount of data grows, the data synchronization method can maintain good scalability and flexibility to adapt to different data warehouses and databases. Accordingly, the data synchronization device needs to use corresponding log capture technology for different database types to extract incremental logs in the database in real time. For example, if the database is MySQL, the binlog log in MySQL is extracted in real time, and if the database is Oracle, the binary log of Oracle is extracted in real time. On this basis, the data synchronization method can also improve scalability through distributed computing, index acceleration and other technical means.

[0055] S120: Parse the incremental log according to the preset log parsing configuration information to obtain at least one piece of data change information.

[0056] The incremental log records the relevant information of each data change in the database, such as the corresponding data change type (such as addition, modification, deletion), the corresponding data table and field identification, the data value before and after the change, etc.

[0057] In some embodiments, the incremental log is in binary format. After the incremental log is extracted from the database, it is necessary to parse the incremental log using a log parsing algorithm to extract relevant information of each piece of data that has changed in the database.

[0058] The above data change information is the relevant information of the changed data obtained by parsing the incremental log.

[0059] Each piece of data change information extracted is processed through steps S130-S150 to modify the data in the data warehouse related to the data changed in the database. Steps S130-S150 are described below.

[0060] S130: extracting first data from the database and second data from the data warehouse according to the data change information.

[0061] For each piece of data change information, the data synchronization device queries data from the database according to the piece of data change information to obtain first data, and queries data from the data warehouse according to the piece of data change information to obtain second data.

[0062] In some embodiments, the data change information includes a data table identifier and a field identifier. Accordingly, the operations of extracting the first data from the database and extracting the second data from the data warehouse according to the data change information include: querying the database for data matching the data table identifier and the field identifier included in the data change information, and using the query result as the first data; querying the data warehouse for data matching the data table identifier and the field identifier included in the data change information, and using the query result as the second data.

[0063] During the data synchronization process, due to network delays, data loss, repeated transmission or data writing errors, the data in the data warehouse may be inconsistent with the data in the database. To address such problems, this embodiment reads the incremental log in the database in real time, and then determines the relevant information of the data that has changed in the database through log analysis, and then queries the corresponding data from the database and the data warehouse respectively, and compares and verifies the data queried from the database and the data queried from the data warehouse. This ensures the accuracy of data synchronization and reduces the risk of data inconsistency.

[0064] S140: Compare the first data and the second data.

[0065] The first data is the field value that matches the data table identifier and field identifier included in the data change information queried from the database, and the second data is the field value that matches the data table identifier and field identifier included in the data change information queried from the data warehouse. Based on this, the first data and the second data are compared, specifically, the two field values ​​obtained from the database and the data warehouse are compared.

[0066] In some embodiments, in addition to field value comparison, the data synchronization device may also use mechanisms such as data integrity verification to verify the first data and the second data to ensure the accuracy and consistency of the data.

[0067] S150: When the first data and the second data are inconsistent, executing a preset data repair strategy according to the data difference type between the first data and the second data.

[0068] In this embodiment, if the data synchronization device finds that the first data and the second data are inconsistent after comparison, it takes corresponding processing measures according to the specific situation, such as data correction, data completion, data deletion, etc.

[0069] In some embodiments, the operation of executing a preset data repair strategy according to the data difference type between the first data and the second data includes:

[0070] When the first data is not empty and the second data is empty, performing data completion processing in the data warehouse according to the data table identifier and the field identifier included in the first data and the data change information;

[0071] When both the first data and the second data are not empty, and the first data and the second data are not the same, performing data repair processing in the data warehouse according to the data table identifier and the field identifier included in the first data and the data change information;

[0072] When the first data is empty and the second data is not empty, data deletion processing is performed in the data warehouse according to the data table identifier and the field identifier included in the data change information.

[0073] The above-mentioned data completion processing may be to set the field value in the data warehouse that matches the data table identifier and field identifier included in the data change information as the first data. The above-mentioned data repair processing may be to update the field value in the data warehouse that matches the data table identifier and field identifier included in the data change information as the first data. The above-mentioned data deletion processing may be to delete the field value in the data warehouse that matches the data table identifier and field identifier included in the data change information.

[0074] In some embodiments, the method also includes: monitoring the software and hardware resource status and pre-specified performance indicators of the server where the database is located; judging whether an abnormality or error occurs based on real-time monitoring data of the software and hardware resource status and performance indicators; and issuing an alarm when an abnormality or error is determined to occur.

[0075] The above-mentioned hardware and software resource status includes the hardware resource status and software resource status of the server where the database is located (which may be referred to as the database server). The hardware resource status includes but is not limited to the operating status of the database server's CPU (central processing unit), memory and other hardware, and the software resource status includes but is not limited to the operating status of the database server's database link pool, thread pool and other software resources.

[0076] The above performance indicators mainly include the number of incremental data processed per second, the number of data warehouse writes per second, whether the added messages are backlogged, etc.

[0077] In this embodiment, the data synchronization device can be combined with the monitoring and alarm system to track the status and performance of the data synchronization process in real time to ensure the stability and reliability of data synchronization. In addition, when anomalies or errors occur (such as the increasing backlog of incremental messages, or database or thread pool resource alarms, etc.), alarm notifications are sent to relevant persons in charge in a timely manner to facilitate timely processing and resolution of problems.

[0078] In some embodiments, the operation of obtaining the incremental log of the database in real time, parsing the incremental log according to preset log parsing configuration information, and obtaining at least one piece of data change information may include: extracting the incremental log of the database in real time through an incremental log synchronization program, converting the format of the extracted incremental log from binary format to a predefined JSON format and saving it to a message queue; extracting the incremental log from the message queue through a data consumption program, parsing the extracted incremental log according to the preset log parsing configuration information, and obtaining at least one piece of data change information.

[0079] See also Figure 2 In order to achieve high availability of data synchronization, this embodiment implements data comparison through three steps, wherein the first step is to store the incremental log through the incremental log synchronization program, that is, to transmit it to the message queue; the second step is to consume the incremental log in the message queue at different times through the data consumption program according to the pre-set log parsing configuration information; the third step is that the data consumption program queries the matching data in the database and the data warehouse respectively according to the data parsed from the incremental log, compares the queried data, and executes the corresponding processing strategy according to the comparison result.

[0080] In some embodiments, the operation of converting the format of the extracted incremental log from a binary format to a predefined JSON format and saving it in a message queue includes: converting the format of the extracted incremental log from a binary format to a predefined JSON format, serializing the incremental log in the JSON format to obtain a serialized incremental log in a binary format, and transferring the serialized incremental log to the message queue for storage.

[0081] JSON (JavaScript Object Notation) is a text format that is easy to read and write, but its length and parsing overhead can become a performance bottleneck in some scenarios. For this reason, in this embodiment, after the incremental log is extracted from the database in real time, the binary incremental log is first converted to JSON format, and then the incremental log in JSON format is serialized to obtain a serialized incremental log converted to binary format, and finally the serialized incremental log is transferred to the message queue. Such operation can not only make the incremental log maintain the flexibility and scalability of data exchange, but also improve the efficiency of data transmission and data processing. Among them, Protobuf can be used to serialize the incremental log in JSON format. Protobuf uses binary encoding, which can usually reduce the data volume by 3 to 10 times compared with JSON format, which makes it more efficient in the network transmission process. In addition, since the parsing of binary data is faster than text data, the encoding and decoding speed of protobuf is usually far superior to JSON.

[0082] Accordingly, the operation of extracting the incremental log from the message queue through the data consumer program includes: consuming the serialized incremental log from the message queue through the data consumer program, deserializing the serialized incremental log, and obtaining the incremental log in JSON format.

[0083] Since the message queue stores serialized incremental logs in binary format, after the data consumer receives the incremental logs from the message queue, it needs to deserialize them before further processing.

[0084] In some embodiments, the log parsing configuration information includes a data table identifier, a field identifier, and a data change type that need to be filtered; accordingly, the operation of parsing the extracted incremental log according to the preset log parsing configuration information to obtain at least one data change information includes: parsing the extracted incremental log through a log parsing algorithm, and putting the parsed data change information into a change information set; each piece of data change information includes a data table identifier, a field identifier, and a data change type; traversing the data change information in the change information set; during the traversal process, the data table identifier, field identifier, and data change type included in the traversed data change information are respectively The traversed data change information is compared with the data table identifier, field identifier and data change type included in the log parsing configuration information. When the data table identifier included in the traversed data change information is consistent with the data table identifier included in the log parsing configuration information, or the field identifier included in the traversed data change information is consistent with the field identifier included in the log parsing configuration information, or the data change type included in the traversed data change information is consistent with the data change type included in the log parsing configuration information, the traversed data change information is removed from the change information set; after the traversal is completed, the remaining data change information in the change information set is taken as at least one piece of data change information.

[0085] Based on the general log parsing algorithm, this embodiment adds personalized configuration. Users can configure the data table identifiers, field identifiers, and / or data change types that do not need to be synchronized in the database according to actual needs. Therefore, when parsing the incremental log, the data change information containing the pre-configured data table identifiers, field identifiers or data change types will be eliminated, and then the corresponding data repair operations will be performed according to the remaining data change information.

[0086] It should be noted that, with respect to the various steps included in the data synchronization method provided in any of the above embodiments, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of these steps may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0087] Based on the same inventive concept, the present application also provides a data synchronization device, which is used to synchronize data from a database to a data warehouse, wherein the database is a relational database and the data warehouse is a non-relational database. Figure 3 As shown, the data synchronization device includes the following modules:

[0088] The incremental log extraction module 110 is used to extract the incremental logs in the database in real time;

[0089] The log parsing module 120 is used to parse the incremental log according to the preset log parsing configuration information to obtain at least one piece of data change information;

[0090] A data extraction module 130, configured to extract first data from a database and second data from a data warehouse according to data change information;

[0091] A comparison module 140, configured to compare the first data with the second data;

[0092] The data repair module 150 is configured to execute a preset data repair strategy according to a data difference type between the first data and the second data when the first data and the second data are inconsistent.

[0093] In some embodiments, the incremental log extraction module 110 is specifically used to extract the incremental log of the database in real time through the incremental log synchronization program, convert the format of the extracted incremental log from the binary format to the predefined JSON format and save it to the message queue;

[0094] The log parsing module 120 is specifically used to extract incremental logs from the message queue through a data consumption program, and parse the extracted incremental logs according to preset log parsing configuration information to obtain at least one piece of data change information.

[0095] In some embodiments, the operation of the incremental log extraction module 110 converting the format of the extracted incremental log from the binary format to the predefined JSON format and saving it to the message queue includes: converting the format of the extracted incremental log from the binary format to the predefined JSON format, serializing the incremental log in the JSON format to obtain a serialized incremental log in the binary format, and transmitting the serialized incremental log to the message queue for storage;

[0096] Accordingly, the log parsing module 120 extracts the incremental log from the message queue through the data consumption program, including: consuming the serialized incremental log from the message queue through the data consumption program, deserializing the serialized incremental log, and obtaining the incremental log in JSON format.

[0097] In some embodiments, the log parsing configuration information includes a data table identifier, a field identifier, and a data change type to be filtered; accordingly, the log parsing module 120 parses the extracted incremental log according to the preset log parsing configuration information to obtain at least one data change information operation, including:

[0098] The extracted incremental log is parsed by a log parsing algorithm, and the parsed data change information is put into a change information set; each piece of data change information includes a data table identifier, a field identifier, and a data change type;

[0099] Traversing the data change information in the change information set; in the traversal process, comparing the data table identifier, field identifier and data change type included in the traversed data change information with the data table identifier, field identifier and data change type included in the log parsing configuration information, respectively; when the data table identifier included in the traversed data change information is consistent with the data table identifier included in the log parsing configuration information, or the field identifier included in the traversed data change information is consistent with the field identifier included in the log parsing configuration information, or the data change type included in the traversed data change information is consistent with the data change type included in the log parsing configuration information, the traversed data change information is removed from the change information set;

[0100] After the traversal is completed, the remaining data change information in the change information set is taken as at least one piece of data change information.

[0101] In some embodiments, the data change information includes a data table identifier and a field identifier; accordingly, the data extraction module 130 is specifically used to: query the database for data that matches the data table identifier and field identifier included in the data change information, and use the query result as the first data; query the data warehouse for data that matches the data table identifier and field identifier included in the data change information, and use the query result as the second data.

[0102] In some embodiments, the data repair module 150 is specifically configured to:

[0103] When the first data is not empty and the second data is empty, data completion processing is performed in the data warehouse according to the data table identifier and field identifier included in the first data and the data change information; when both the first data and the second data are not empty, and the first data and the second data are not the same, data repair processing is performed in the data warehouse according to the data table identifier and field identifier included in the first data and the data change information; when the first data is empty and the second data is not empty, data deletion processing is performed in the data warehouse according to the data table identifier and field identifier included in the data change information.

[0104] In some embodiments, the device further includes a monitoring alarm module. The monitoring alarm module is used to:

[0105] Monitor the software and hardware resource status and pre-specified performance indicators of the server where the database is located;

[0106] Determine whether anomalies or errors occur based on real-time monitoring data of software and hardware resource status and performance indicators;

[0107] When an abnormality or error is detected, an alarm is issued.

[0108] For the specific definition of the data synchronization device, please refer to the definition of the data synchronization method above, which will not be repeated here. Each module in the above-mentioned data synchronization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0109] In some embodiments, the present application provides a computer device, whose internal structure diagram can be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as log parsing configuration information, and the specific stored data can also be found in the definition in the above method embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0110] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] This embodiment also provides a computer device. In some embodiments, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data synchronization method provided in any of the above embodiments can be implemented.

[0112] The present application also provides a computer-readable storage medium. In some embodiments, a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the data synchronization method provided in any of the above embodiments are implemented.

[0113] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0114] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method, the method is used to synchronize the data of a database to a data warehouse, characterized in that: The database is a relational database, and the data warehouse is a non-relational database; the method comprises: Extracting incremental logs in the database in real time; Parsing the incremental log according to preset log parsing configuration information to obtain at least one piece of data change information; Extracting first data from the database according to the data change information and extracting second data from the data warehouse; comparing the first data and the second data; When the first data and the second data are inconsistent, a preset data repair strategy is executed according to the data difference type between the first data and the second data.

2. The method according to claim 1, characterized in that Acquire the incremental log of the database in real time, parse the incremental log according to the preset log parsing configuration information, and obtain at least one piece of data change information, including: Extracting incremental logs of the database in real time through an incremental log synchronization program, converting the format of the extracted incremental logs from a binary format to a predefined JSON format and saving them to a message queue; The incremental log is extracted from the message queue through a data consumption program, and the extracted incremental log is parsed according to preset log parsing configuration information to obtain at least one piece of data change information.

3. The method according to claim 2, characterized in that Convert the extracted incremental log format from binary format to a predefined JSON format and save it to the message queue, including: Convert the format of the extracted incremental log from binary format to a predefined JSON format, serialize the incremental log in JSON format to obtain a serialized incremental log in binary format, and transfer the serialized incremental log to a message queue for storage; Extracting incremental logs from the message queue through a data consumer program includes: The serialized incremental log is consumed from the message queue through a data consumption program, and the serialized incremental log is deserialized to obtain an incremental log in JSON format.

4. The method according to claim 2 or 3, characterized in that The log parsing configuration information includes the data table identifier, field identifier and data change type to be filtered; The extracted incremental log is parsed according to the preset log parsing configuration information to obtain at least one piece of data change information, including: Parse the extracted incremental logs using a log parsing algorithm, and put the parsed data change information into a change information set; each piece of data change information includes a data table identifier, a field identifier, and a data change type; Traversing the data change information in the change information set; in the traversal process, comparing the data table identifier, field identifier and data change type included in the traversed data change information with the data table identifier, field identifier and data change type included in the log parsing configuration information, respectively; when the data table identifier included in the traversed data change information is consistent with the data table identifier included in the log parsing configuration information, or the field identifier included in the traversed data change information is consistent with the field identifier included in the log parsing configuration information, or the data change type included in the traversed data change information is consistent with the data change type included in the log parsing configuration information, the traversed data change information is removed from the change information set; After the traversal is completed, the remaining data change information in the change information set is used as the at least one piece of data change information.

5. The method according to claim 1, characterized in that The data change information includes a data table identifier and a field identifier; extracting first data from the database and extracting second data from the data warehouse according to the data change information includes: querying the database for data matching the data table identifier and the field identifier included in the data change information, and using the query result as the first data; The data warehouse is queried for data matching the data table identifier and the field identifier included in the data change information, and the query result is used as the second data.

6. The method according to claim 1, characterized in that Executing a preset data repair strategy according to the data difference type between the first data and the second data includes: When the first data is not empty and the second data is empty, performing data completion processing in the data warehouse according to the data table identifier and the field identifier included in the first data and the data change information; When both the first data and the second data are not empty, and the first data and the second data are not the same, performing data repair processing in the data warehouse according to the data table identifier and the field identifier included in the first data and the data change information; When the first data is empty and the second data is not empty, data deletion processing is performed in the data warehouse according to the data table identifier and the field identifier included in the data change information.

7. The method according to claim 1, characterized in that The method further comprises: Monitor the software and hardware resource status and pre-specified performance indicators of the server where the database is located; Determining whether an abnormality or error occurs based on the real-time monitoring data of the software and hardware resource status and the performance indicators; When an abnormality or error is detected, an alarm is issued.

8. A data synchronization device, the device is used to synchronize the data of a database to a data warehouse, characterized in that: The database is a relational database, and the data warehouse is a non-relational database; the device comprises: An incremental log extraction module, used for extracting incremental logs in the database in real time; A log parsing module, used to parse the incremental log according to preset log parsing configuration information to obtain at least one piece of data change information; A data extraction module, used for extracting first data from the database and second data from the data warehouse according to the data change information; A comparison module, used for comparing the first data with the second data; The data repair module is used to execute a preset data repair strategy according to the data difference type between the first data and the second data when the first data and the second data are inconsistent.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images