Data synchronization method, device, electronic device and computer-readable medium

Abstract

The present invention discloses a data synchronization method, device, electronic device and computer-readable medium, which relate to the field of data exchange technology. A specific implementation of the method includes: monitoring the logs in the source database, and when the logs change, parsing the logs according to the log format preset by the source database to obtain multiple data change messages; updating the flow direction sequence based on the message type of each data change message, and then calculating the flow characteristic value based on the numerical value in the flow direction sequence; if the flow characteristic value is within the preset interval, a batch of messages of the same type are written in time sequence, otherwise a single message is written in time sequence to write multiple data change messages to the target database. This implementation method counts the message types of the most recent data change messages to update the flow direction sequence and calculate a better data writing mode, thereby improving the overall writing performance and reducing memory usage.

Description

Technical Field

[0001] The present invention relates to the field of data exchange technology, and in particular to a data synchronization method, device, electronic device and computer-readable medium. Background Art

[0002] With business development and departmental divisions, each department has established independent business submodules and databases. However, data dependencies exist between departments, which requires daily synchronization of large amounts of new data between departments. There are two existing data synchronization solutions: batch writing of messages of the same type in time series (hereinafter referred to as Method 1) and writing of individual messages in time series (hereinafter referred to as Method 2). However, these two methods have the following problems:

[0003] 1. Method 1 requires message type judgment and creation of new precompiled SQL statements when processing different message types. When faced with frequent switching between multiple message types, its performance is very poor and memory consumption is very high.

[0004] 2. Method 2 requires processing each piece of data separately. Although it does not require determining the message type and creating a new precompiled object, it cannot fully utilize the target database's optimization for batch operations. Therefore, its performance is poor when faced with a large number of consecutive messages of the same type.

[0005] The above write modes exhibit inconsistent performance across different combinations of synchronous message types. Batch writes offer better performance and lower memory usage for large, continuous batches of messages of the same type, while single writes offer better performance and lower memory usage for large, frequently switching batches of messages of different types. However, existing write modes cannot be adjusted in real time for different scenarios, resulting in unstable overall performance and high memory usage. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a data synchronization method, device, electronic device and computer-readable medium, which can at least solve the problem that the existing technology cannot adjust the writing mode for different scenarios when facing different message type combinations.

[0007] To achieve the above object, according to one aspect of an embodiment of the present invention, a data synchronization method is provided, comprising:

[0008] Monitoring logs in a source database, and when the logs change, parsing the logs according to a log format preset by the source database to obtain multiple data change messages;

[0009] Updating a traffic direction sequence based on the message type of each data change message, and then calculating a traffic characteristic value based on the values ​​in the traffic direction sequence; wherein the traffic direction indicates whether the message type of adjacent data change messages has changed;

[0010] If the traffic characteristic value is within the preset interval, a batch time-sequential writing mode for messages of the same type is adopted; otherwise, a single time-sequential writing mode is adopted to write the multiple data change messages into the target database.

[0011] Optionally, after obtaining the multiple data change messages, the method further includes:

[0012] Sending the multiple data change messages to a distributed message queue for caching; and

[0013] According to the time sequence of log generation, data change messages are pulled from the distributed message queue one by one; wherein the data change messages include the time sequence of the log generation.

[0014] Optionally, the method further includes: the distributed message queue includes multiple partitions, and the source database corresponds to one or more of the partitions.

[0015] Optionally, before updating the traffic direction sequence based on the message type of each data change message, the method further includes:

[0016] Get the preset number of historical data change messages synchronized since the current one;

[0017] The traffic direction sequence is generated by counting the message types of the preset number of historical data change messages.

[0018] Optionally, the flow direction sequence is composed of the preset number of values;

[0019] The generating the traffic direction sequence by counting the message types of the preset number of historical data change messages includes:

[0020] For the preset number of historical data change messages, when the first historical data change message is monitored, the historical traffic direction sequence is initialized to all 0s;

[0021] When a second historical data change message is monitored, if the second historical data change message and the first data change message have the same message type, the current traffic direction is assigned to the first preset value; otherwise, the current traffic direction is assigned to the second preset value;

[0022] Remove the first value in the flow direction sequence and shift the other values ​​in sequence to write the value of the current flow direction in the last digit;

[0023] The above steps are repeated until the preset number of historical data change messages are processed to obtain the traffic direction sequence.

[0024] Optionally, the first preset value is 1 and the second preset value is -1.

[0025] Optionally, different code values ​​correspond to different message types;

[0026] The method further comprises:

[0027] Determine a first code value corresponding to a message type of a first historical data change message, and initialize the current message type of the traffic direction to the first code value;

[0028] Determine a second code value corresponding to the message type of the second historical data change message, judge whether the first code value and the second code value are the same; and if the judgment result is different, change the current message type of the traffic direction to the second code value.

[0029] Optionally, updating the traffic direction sequence based on the message type of each data change message includes:

[0030] For a first data change message among the multiple data change messages, determining whether the message type is the same as that of the last data change message in the traffic direction sequence;

[0031] If they are the same, the current flow direction is assigned to the first preset value, otherwise it is assigned to the second preset value;

[0032] Remove the first value in the flow direction sequence and shift the remaining values ​​to write the value of the current flow direction in the last digit;

[0033] The above process is repeated until the multiple data change messages are processed, and an updated traffic direction sequence is obtained.

[0034] Optionally, the calculating of the flow characteristic value based on the numerical value in the flow direction sequence includes:

[0035] Counting the sum of all values ​​in the traffic direction sequence, calculating an average of the sum of all values ​​and the preset number, and determining whether the average is less than or equal to 0;

[0036] If yes, determine the standard deviation of the flow direction sequence and set the flow characteristic value to -1*(mean*standard deviation+1); or

[0037] If not, the flow characteristic value is set to (mean value*standard deviation+1).

[0038] Optionally, the method further includes: acquiring a current data writing mode, and if the current data writing mode is inconsistent with the determined data writing mode, switching the current data writing mode to the determined data writing mode.

[0039] Optionally, the method further includes: performing format conversion on the multiple data change messages according to the table structure of the target database, so as to write the multiple data change messages after the format conversion into the target table of the target database.

[0040] Optionally, the preset interval is [1, 1.7].

[0041] To achieve the above object, according to another aspect of an embodiment of the present invention, a data synchronization device is provided, comprising:

[0042] A log monitoring module is used to monitor the logs in the source database and parse the logs according to the log format preset by the source database when the logs change, to obtain multiple data change messages;

[0043] a traffic monitoring module configured to update a traffic direction sequence based on the message type of each data change message, and further calculate a traffic characteristic value based on the values ​​in the traffic direction sequence; wherein the traffic direction indicates whether the message type of adjacent data change messages has changed;

[0044] The data writing module is used to adopt a batch writing mode of messages of the same type in a time-sequential manner if the traffic characteristic value is within a preset interval, otherwise adopt a single message writing mode in a time-sequential manner to write the multiple data change messages into the target database.

[0045] Optionally, it further includes a data sending module, configured to send the plurality of data change messages to a distributed message queue for caching; and

[0046] The data pulling module is used to pull data change messages from the distributed message queue one by one according to the generation time sequence of the logs; wherein the data change messages include the generation time sequence of the logs in which they are located.

[0047] Optionally, the method further includes: the distributed message queue includes multiple partitions, and the source database corresponds to one or more of the partitions.

[0048] Optionally, the flow monitoring module is further configured to:

[0049] Get the preset number of historical data change messages synchronized since the current one;

[0050] The traffic direction sequence is generated by counting the message types of the preset number of historical data change messages.

[0051] Optionally, the flow direction sequence is composed of the preset number of values;

[0052] The flow monitoring module is used to:

[0053] For the preset number of historical data change messages, when the first historical data change message is monitored, the historical traffic direction sequence is initialized to all 0s;

[0054] When a second historical data change message is monitored, if the second historical data change message and the first data change message have the same message type, the current traffic direction is assigned to the first preset value; otherwise, the current traffic direction is assigned to the second preset value;

[0055] Remove the first value in the flow direction sequence and shift the other values ​​in sequence to write the value of the current flow direction in the last digit;

[0056] The above steps are repeated until the preset number of historical data change messages are processed to obtain the traffic direction sequence.

[0057] Optionally, the first preset value is 1 and the second preset value is -1.

[0058] Optionally, different code values ​​correspond to different message types;

[0059] The flow monitoring module is further used for:

[0060] Determine a first code value corresponding to a message type of a first historical data change message, and initialize the current message type of the traffic direction to the first code value;

[0061] Determine a second code value corresponding to the message type of the second historical data change message, judge whether the first code value and the second code value are the same; and if the judgment result is different, change the current message type of the traffic direction to the second code value.

[0062] Optionally, the flow monitoring module is used to:

[0063] For a first data change message among the multiple data change messages, determining whether the message type is the same as that of the last data change message in the traffic direction sequence;

[0064] If they are the same, the current flow direction is assigned to the first preset value, otherwise it is assigned to the second preset value;

[0065] Remove the first value in the flow direction sequence and shift the remaining values ​​to write the value of the current flow direction in the last digit;

[0066] The above process is repeated until the multiple data change messages are processed, and an updated traffic direction sequence is obtained.

[0067] Optionally, the flow monitoring module is used to:

[0068] Counting the sum of all values ​​in the traffic direction sequence, calculating an average of the sum of all values ​​and the preset number, and determining whether the average is less than or equal to 0;

[0069] If yes, determine the standard deviation of the flow direction sequence and set the flow characteristic value to -1*(mean*standard deviation+1); or

[0070] If not, the flow characteristic value is set to (mean value*standard deviation+1).

[0071] Optionally, the traffic monitoring module is further used to: obtain a current data writing mode, and if the current data writing mode is inconsistent with the determined data writing mode, switch the current data writing mode to the determined data writing mode.

[0072] Optionally, the data writing module is configured to convert the format of the multiple data change messages according to the table structure of the target database, so as to write the multiple data change messages after the format conversion into the target table of the target database.

[0073] Optionally, the preset interval is [1, 1.7].

[0074] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data synchronization electronic device is provided.

[0075] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned data synchronization methods.

[0076] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above-mentioned data synchronization methods is implemented.

[0077] According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: by adding a traffic monitoring module to the real-time data synchronization process, the message types of recent data change messages are counted to update the traffic direction sequence and calculate a better data writing mode, thereby improving the overall writing performance and reducing memory usage.

[0078] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0080] Figure 1 This is a schematic diagram of the main flow of a data synchronization method according to an embodiment of the present invention;

[0081] Figure 2 is a flowchart of an optional data synchronization method according to an embodiment of the present invention;

[0082] Figure 3 is a flowchart of another optional data synchronization method according to an embodiment of the present invention;

[0083] Figure 4 2 is a schematic diagram of a data synchronization implementation framework according to an embodiment of the present invention;

[0084] Figure 5 This is a schematic diagram of main modules of a data synchronization device according to an embodiment of the present invention;

[0085] Figure 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0086] Figure 7 It is a schematic diagram of the structure of a computer system of a mobile device or server suitable for implementing the embodiments of the present invention. DETAILED DESCRIPTION

[0087] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0088] See also Figure 1 , which shows a main flow chart of a data synchronization method provided by an embodiment of the present invention, including the following steps:

[0089] S101: monitoring a log in a source database, and when the log changes, parsing the log according to a log format preset by the source database to obtain a plurality of data change messages;

[0090] S102: updating a traffic direction sequence based on the message type of each data change message, and then calculating a traffic characteristic value based on the values ​​in the traffic direction sequence; wherein the traffic direction indicates whether the message type of adjacent data change messages has changed;

[0091] S103: If the traffic characteristic value is within a preset range, a batch writing mode of messages of the same type is adopted in a time-sequential manner; otherwise, a single message writing mode is adopted;

[0092] S104: Writing the multiple data change messages into the target database.

[0093] In the above embodiment, regarding step S101, multiple databases are typically established for the same information processing system based on the source database and different usage requirements. When data in the source database changes, it is necessary to ensure data synchronization between the other databases established based on the source database and the source database. In other words, data synchronization with the other databases is required based on data changes in the source database.

[0094] Monitor the logs in the source database, such as Binlog in MySQL and materialized view logs in Oracle. When the logs in the source database change, parse the logs according to the preset log format of the source database and capture the data change messages in the logs. One log corresponds to a group of messages, and each group of messages consists of multiple data change messages.

[0095] Furthermore, to alleviate data synchronization pressure, data change messages can be sent to a distributed message queue for caching. The generation time sequence of the logs in each data change message is then obtained, and the corresponding data change messages are pulled from the distributed message queue one by one in the order of log generation time.

[0096] Furthermore, considering that there may be multiple source databases, sending all of them to a distributed message queue simultaneously could cause data confusion. Therefore, partitions can be created within the distributed message queue. Each source database can correspond to just one partition or multiple partitions, with flexible configuration. Therefore, data change messages must be sent to the partition corresponding to the source database on the distributed message queue, and then pulled from that partition. Furthermore, different logs correspond to different tables, and different partitions can also correspond to different tables (a table is one or more source tables in the source database that need to be synchronized).

[0097] In step S102, this solution pre-sets four message types: INSERT (I), DELETE (D), UPDATE (U), and DDL (L). These can be set based on specific circumstances. A source database can be connected to multiple target databases. After acquiring data change messages, the message type of each data change message is first determined.

[0098] This solution updates the existing flow direction sequence based on the message type of this data change message, and the existing flow direction sequence is obtained based on the preset number of data change messages synchronized from the current one, such as the distance from the current one hundred thousand data change messages. Figure 2 As shown:

[0099] S201: Obtaining a preset number of data change messages synchronized from the current one;

[0100] S202: For the preset number of data change messages, when the first data change message is monitored, initialize the traffic direction sequence to all 0s;

[0101] S203: When a second data change message is monitored, if the second data change message and the first data change message are of the same message type, assigning the current traffic direction to a first preset value; otherwise, assigning the current traffic direction to a second preset value;

[0102] S204: removing the first value in the flow direction sequence and shifting the other values ​​in sequence, so as to write the value of the current flow direction in the last digit;

[0103] S205: Repeat the above steps until the preset number of data change messages are processed to obtain the traffic direction sequence.

[0104] The above operations are completed in the traffic monitoring module, which caches two intermediate variables: one is the current traffic direction (S), and the other is the traffic direction sequence (SQ). This solution sets the traffic direction sequence to consist of a preset number of values. Taking into account the continuous changes in the traffic direction sequence, the current time is used as the base point, and a preset number of historical data change messages synchronized to the current time are selected, such as 100,000 historical data change messages from the current time. In addition, considering the different amounts of data change messages in different time periods and the instability, the method of selecting data change messages of a certain historical length is not considered.

[0105] For the 100,000 historical data change messages, SQ is initialized to all 0s for the first historical data change message, such as [0, 0, 0, 0…, 0], according to chronological order. For the second historical data change message, if the message type is the same as the first historical data change message, the current traffic direction S is assigned a value of 1 (the first preset value, which is adjustable), indicating that the message type is consistent with the previous data change message. Otherwise, S is assigned a value of -1 (the second preset value, which is adjustable), indicating that the message type is different from the previous data change message.

[0106] Remove the first value in the traffic direction sequence (SQ) and write the refreshed S value in the last bit. Subsequent message monitoring and processing processes are synchronized. After processing these 100,000 historical data change messages, a non-zero traffic direction sequence will be obtained.

[0107] Furthermore, the traffic monitoring module can also set the message type (T) of the previous message. For the first historical data change message, T will be set to the message type of the first historical data change message. Code values ​​can also be pre-set for different message types, such as 0 / 1 / 2 / 3 for I / D / U / L, respectively. Subsequently, T will be set to the first code value corresponding to the message type of the first historical data change message. When a second historical data change message is monitored, T will be refreshed to the second code value corresponding to the message type of the second historical data change message.

[0108] On the premise of obtaining the existing traffic direction sequence, update it based on the data change message obtained this time. For details, see Figure 3 As shown, the operation process is the same as Figure 2 As shown, I will not repeat it here:

[0109] S301: For a first data change message among multiple data change messages, determine whether the message type is the same as that of the last data change message in the traffic direction sequence;

[0110] S302: If they are the same, assigning the current flow direction to the first preset value; otherwise, assigning the current flow direction to the second preset value;

[0111] S303: removing the first value in the flow direction sequence and shifting the remaining values ​​to write the value of the current flow direction in the last digit;

[0112] S304: Repeat the above process until the multiple data change messages are processed, and obtain an updated traffic direction sequence.

[0113] After the traffic direction sequence is updated based on multiple data change messages, the traffic characteristic value is calculated. First, the average value of SQ is calculated. If the average value of SQ is less than or equal to 0, the traffic characteristic value is -1*(SQ average value*standard deviation+1), otherwise it is (SQ average value*standard deviation+1).

[0114] The standard deviation is set between [0 and 0.7] based on empirical values. It indicates the degree of deviation between the traffic direction sample and the mean and can be adjusted based on actual conditions. The closer to 0, the lower the deviation, while the larger the deviation, the higher the deviation. Multiplying by the SQ average is used to distinguish between positive and negative traffic directions. A positive value indicates that more than half of the messages are of the same type, while a negative value indicates that more than half of the messages are of different types. Adding 1 is used to exclude extreme conditions, such as the special case of a standard deviation of 0. When the traffic direction statistics are all 1 or all -1, the standard deviation is 0. Adding 1 and then multiplying by the SQ average determines whether the overall traffic direction is positive or negative.

[0115] For steps S103 to S104, if the calculated traffic characteristic value is in the interval [1,1.7], the batch time-sequential writing mode is used for messages of the same type; when it is in the remaining intervals, the single message time-sequential writing mode is used to write multiple data change messages to the target database.

[0116] The value [1, 1.7] set here is an empirical value. According to statistical experience, nearly 80% of sample message types do not change, that is, a large number of continuous messages of the same type are sent. In this case, it is more optimal to write messages of the same type in batches in a time sequence.

[0117] In addition, the above calculation only calculates the current optimal data writing mode. This mode is relative to different message type combinations. When multiple message types switch frequently, it is better to use the single message time-sequential writing mode. In the case of a large number of consecutive messages of the same type, it is better to use the batch time-sequential writing mode of the same type of messages.

[0118] The current data writing mode can be obtained by querying the data writing module. If it is different from the calculated data writing mode, the data writing mode needs to be switched, otherwise no processing is performed.

[0119] Each database has its own format requirements. Therefore, when writing multiple data change information into the target database, it is also necessary to convert the format of these data change information according to the table structure of the target database so that the converted data change information can be written into the target table of the target database.

[0120] The method provided in the above embodiment adds a traffic monitoring module to the real-time data synchronization process, and adjusts the optimal data writing mode in real time by real-time monitoring, statistics, and analysis of the message types of data change information. When multiple message types are frequently switched, a single message is written in time sequence. In the case of a large number of continuous messages of the same type, batches of messages of the same type are written in time sequence, thereby improving the overall writing performance and stability while reducing memory usage.

[0121] See also Figure 4, which shows a schematic diagram of the main modules of a data synchronization device 400 provided by an embodiment of the present invention, including:

[0122] A log monitoring module is used to monitor the logs in the source database and, when the logs change, parse the logs according to the log format preset by the source database to obtain multiple data change messages;

[0123] a traffic monitoring module configured to update a traffic direction sequence based on the message type of each data change message, and further calculate a traffic characteristic value based on the values ​​in the traffic direction sequence; wherein the traffic direction indicates whether the message type of adjacent data change messages has changed;

[0124] The data writing module is used to adopt a batch writing mode of messages of the same type in a time-sequential manner if the traffic characteristic value is within a preset interval, otherwise adopt a single message writing mode in a time-sequential manner to write the multiple data change messages into the target database.

[0125] The flow monitoring module in the implementation device of the present invention is also used to: obtain a preset number of historical data change messages synchronized from the current one; and generate the flow direction sequence by counting the message types of the preset number of historical data change messages.

[0126] Optionally, the flow direction sequence is composed of the preset number of numerical values; the flow monitoring module is used to: for the preset number of historical data change messages, when the first historical data change message is monitored, initialize the historical flow direction sequence to all 0s; when the second historical data change message is monitored, if the second historical data change message and the first data change message are of the same message type, assign the current flow direction to the first preset numerical value, otherwise assign it to the second preset numerical value; remove the first numerical value in the flow direction sequence and shift the other numerical values ​​in order to write the assigned value of the current flow direction in the last digit; repeat the above steps until the preset number of historical data change messages are processed, and obtain the flow direction sequence. The first preset numerical value here is 1, and the second preset numerical value is -1.

[0127] Optionally, different message types correspond to different code values; the traffic monitoring module is also used to: determine the first code value corresponding to the message type of the first historical data change message, and initialize the current message type of the traffic direction to the first code value; determine the second code value corresponding to the message type of the second historical data change message, and judge whether the first code value and the second code value are the same; and if the judgment result is different, change the current message type of the traffic direction to the second code value.

[0128] Optionally, the traffic monitoring module is used to: for the first data change message among the multiple data change messages, determine whether the message type is the same as the message type of the last data change message in the traffic direction sequence; if the same, assign the current traffic direction to the first preset value, otherwise assign it to the second preset value; remove the first value in the traffic direction sequence and shift the remaining values ​​to write the assigned value of the current traffic direction in the last place; repeat the above process until the multiple data change messages are processed and the updated traffic direction sequence is obtained.

[0129] Optionally, the traffic monitoring module is used to: count the sum of all numerical values ​​in the traffic direction sequence, calculate the sum of all numerical values ​​and the average of the preset number, and determine whether the average is less than or equal to 0; if so, determine the standard deviation of the traffic direction sequence, and set the traffic characteristic value to -1*(average value*standard deviation+1); or if not, set the traffic characteristic value to (average value*standard deviation+1).

[0130] Optionally, the traffic monitoring module is further used to: obtain a current data writing mode, and if the current data writing mode is inconsistent with the determined data writing mode, switch the current data writing mode to the determined data writing mode.

[0131] The data writing module in the implementation device of the present invention is used to: convert the format of the multiple data change messages according to the table structure of the target database, so as to write the multiple data change messages after the conversion into the target table of the target database.

[0132] The preset interval in the device implementing the present invention is [1, 1.7].

[0133] The implementation device of the present invention further includes a data sending module, which is used to send the multiple data change messages to a distributed message queue;

[0134] A distributed message queue, configured to cache the plurality of data change messages sent by the data sending module; further, the distributed message queue includes a plurality of partitions, the source database corresponding to one or more of the partitions; and

[0135] The data pull module is used to pull data change messages from the distributed message queue one by one according to the log generation time sequence; wherein the data change message includes the generation time sequence of the log in which it is located, see the whole Figure 5 shown.

[0136] In addition, the specific implementation content of the device in the embodiment of the present invention has been described in detail in the above method, so the repeated content will not be described again here.

[0137] Figure 6 An exemplary system architecture 600 to which embodiments of the present invention may be applied is shown, including terminal devices 601 , 602 , 603 , a network 604 and a server 605 (only an example).

[0138] Terminal devices 601, 602, and 603 can be various electronic devices with display screens and support web browsing, and are installed with various communication client applications. Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc.

[0139] The network 604 is used to provide a medium for communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0140] Server 605 can be a server that provides various services, used to perform operations such as monitoring logs in a source database, parsing logs to obtain data change information, updating a traffic direction sequence based on the data change information, calculating traffic characteristic values ​​based on the traffic direction sequence, adjusting the optimal data writing mode using the traffic characteristic values, and writing data change messages to a target database.

[0141] It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the device is generally set in the server 605 .

[0142] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0143] Reference below Figure 7 , which shows a schematic structural diagram of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0144] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are also stored in the RAM 703. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0145] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read therefrom can be installed into the storage section 708 as needed.

[0146] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0147] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] The modules described in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as: a processor including a log monitoring module, a traffic monitoring module, and a data writing module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, a data writing module may also be described as a "data change message writing module."

[0150] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently without being incorporated into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device includes:

[0151] Monitoring logs in a source database, and when the logs change, parsing the logs according to a log format preset by the source database to obtain multiple data change messages;

[0152] Updating a traffic direction sequence based on the message type of each data change message, and then calculating a traffic characteristic value based on the values ​​in the traffic direction sequence; wherein the traffic direction indicates whether the message type of adjacent data change messages has changed;

[0153] If the traffic characteristic value is within the preset interval, a batch time-sequential writing mode for messages of the same type is adopted; otherwise, a single time-sequential writing mode is adopted to write the multiple data change messages into the target database.

[0154] According to the technical solution of the embodiments of the present invention, a traffic monitoring module is added to the real-time data synchronization process. By real-time monitoring, statistics, and analysis of the message types of data change information, the optimal data writing mode is adjusted in real time. When multiple message types are frequently switched, a single message is written in time sequence. In the case of a large number of continuous messages of the same type, batches of messages of the same type are written in time sequence, which improves the overall writing performance and stability while reducing memory usage.

[0155] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized in that: include: Monitoring logs in a source database, and when the logs change, parsing the logs according to a log format preset by the source database to obtain multiple data change messages; Updating a traffic direction sequence based on the message type of each data change message, and then calculating a traffic characteristic value based on the values ​​in the traffic direction sequence; wherein the traffic direction indicates whether the message type of adjacent data change messages has changed; If the traffic characteristic value is within the preset interval, a batch time-sequential writing mode for messages of the same type is adopted; otherwise, a single time-sequential writing mode is adopted to write the multiple data change messages into the target database.

2. The method according to claim 1, characterized in that After obtaining the plurality of data change messages, the method further includes: Sending the multiple data change messages to a distributed message queue for caching; and According to the time sequence of log generation, data change messages are pulled from the distributed message queue one by one; wherein the data change messages include the time sequence of the log generation.

3. The method according to claim 2, characterized in that Also includes: The distributed message queue includes multiple partitions, and the source database corresponds to one or more of the partitions.

4. The method according to claim 1, wherein Before updating the traffic direction sequence based on the message type of each data change message, the method further includes: Get the preset number of historical data change messages synchronized since the current one; The traffic direction sequence is generated by counting the message types of the preset number of historical data change messages.

5. The method according to claim 4, characterized in that The flow direction sequence is composed of the preset number of values; The generating the traffic direction sequence by counting the message types of the preset number of historical data change messages includes: For the preset number of historical data change messages, when the first historical data change message is monitored, the historical traffic direction sequence is initialized to all 0s; When a second historical data change message is monitored, if the second historical data change message and the first data change message have the same message type, the current traffic direction is assigned to the first preset value; otherwise, the current traffic direction is assigned to the second preset value; Remove the first value in the flow direction sequence and shift the other values ​​in sequence to write the value of the current flow direction in the last digit; The above steps are repeated until the preset number of historical data change messages are processed to obtain the traffic direction sequence.

6. The method according to claim 5, characterized in that The first preset value is 1, and the second preset value is -1.

7. The method according to claim 5, characterized in that Different message types correspond to different code values; The method further comprises: Determine a first code value corresponding to a message type of a first historical data change message, and initialize the current message type of the traffic direction to the first code value; Determine a second code value corresponding to the message type of the second historical data change message, judge whether the first code value and the second code value are the same; and if the judgment result is different, change the current message type of the traffic direction to the second code value.

8. The method according to claim 5, characterized in that The updating of the traffic direction sequence based on the message type of each data change message includes: For a first data change message among the multiple data change messages, determining whether the message type is the same as that of the last data change message in the traffic direction sequence; If they are the same, the current flow direction is assigned to the first preset value, otherwise it is assigned to the second preset value; Remove the first value in the flow direction sequence and shift the remaining values ​​to write the value of the current flow direction in the last digit; The above process is repeated until the multiple data change messages are processed, and an updated traffic direction sequence is obtained.

9. The method according to claim 4, characterized in that The calculating of the flow characteristic value based on the numerical value in the flow direction sequence includes: Counting the sum of all values ​​in the traffic direction sequence, calculating an average of the sum of all values ​​and the preset number, and determining whether the average is less than or equal to 0; If yes, determine the standard deviation of the flow direction sequence and set the flow characteristic value to -1*(mean*standard deviation+1); or If not, the flow characteristic value is set to (mean value*standard deviation+1).

10. The method according to claim 1, characterized in that Also includes: A current data writing mode is obtained, and if the current data writing mode is inconsistent with the determined data writing mode, the current data writing mode is switched to the determined data writing mode.

11. The method according to claim 1, wherein Also includes: The plurality of data change messages are format-converted according to the table structure of the target database, so as to write the plurality of data change messages after the format conversion into the target table of the target database.

12. The method according to claim 1, characterized in that The preset interval is [1,1.7].

13. A data synchronization device, characterized in that: include: A log monitoring module is used to monitor the logs in the source database and, when the logs change, parse the logs according to the log format preset by the source database to obtain multiple data change messages; a traffic monitoring module configured to update a traffic direction sequence based on the message type of each data change message, and further calculate a traffic characteristic value based on the values ​​in the traffic direction sequence; wherein the traffic direction indicates whether the message type of adjacent data change messages has changed; The data writing module is used to adopt a batch writing mode of messages of the same type in a time-sequential manner if the traffic characteristic value is within a preset interval, otherwise adopt a single message writing mode in a time-sequential manner to write the multiple data change messages into the target database.

14. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 12.

15. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images