Data synchronization method under distributed system, main system, storage medium and system

Abstract

The invention provides a data synchronization method under a distributed system, a main system, a storage medium and a system. The method comprises the steps that an SQL instruction is received and generated in response to preset operation, the preset operation comprises at least one operation of adding, deleting, modifying and searching data of a target table in a database of a main system, and the target table is an operated database table corresponding to the SQL instruction; the SQL instruction and execution information are integrated into a message, the message is sent to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization of the master system and the target slave system, and the execution information at least comprises the target slave system corresponding to the SQL instruction. The target slave system is a system representing that data synchronization needs to be carried out. The problem of poor data synchronization flexibility in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization in a distributed system, and in particular to a data synchronization method in a distributed system, a main system, a computer-readable storage medium, and a data synchronization system in a distributed system. Background Art

[0002] In traditional data synchronization solutions, the sender and receiver often need to synchronize data through thread links within the same system. This direct connection method makes the sender and receiver highly coupled, resulting in poor flexibility in data synchronization. Summary of the Invention

[0003] The main purpose of this application is to provide a data synchronization method in a distributed system, a main system, a computer-readable storage medium and a data synchronization system in a distributed system, so as to at least solve the problem of poor flexibility of data synchronization in the prior art.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data synchronization method in a distributed system is provided, including: receiving and generating SQL instructions in response to preset operations, the preset operations including adding, deleting, modifying and searching data in a target table in the database of the main system, and the target table is the operated database table corresponding to the SQL instruction; integrating the SQL instruction and execution information into a message, and sending the message to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the main system and the target slave system, wherein the execution information includes at least the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the need for data synchronization.

[0005] Optionally, after integrating the SQL instruction and execution information into a message and sending the message to a message queue so that the target slave system obtains and processes the message from the message queue, the method further includes: receiving a receipt message sent by the target slave system, changing the status of the message according to the receipt message, the receipt message being an identifier representing the status of the message, the status identifier being one of the following: a processed identifier, an unprocessed identifier, and a pending synchronization identifier; when the receipt message is the processed identifier, determining that the execution result of the message corresponding to the receipt message is successful execution, and determining that the message status of the message corresponding to the receipt message is processed; when the receipt message is the unprocessed identifier, determining that the execution result of the message corresponding to the receipt message is failed execution, and determining that the message status of the message corresponding to the receipt message is unprocessed; when the receipt message is the pending synchronization identifier, determining that the target slave system has not received the message corresponding to the receipt message, and determining that the message status of the message corresponding to the receipt message is pending synchronization.

[0006] Optionally, the execution information also includes the version number of the module to which the target table belongs, the version number being an identifier representing the version of the module, the module being a collection of tables with data association relationships, and after sending the message to the message queue so that the target slave system can obtain and process the message from the message queue, thereby achieving data synchronization, the method further includes: determining the module to which the target table belongs based on the target table; updating the version number of the module, and each time the preset operation is performed on the table in the module, adjusting the version number to the sum of the current version number and the first preset value, and the initial value of the version number of the module is 1.

[0007] Optionally, after determining that the target slave system has not received the message corresponding to the receipt message, the method further includes: determining that the message corresponding to the receipt message not received by the target slave system is a message to be synchronized; after initializing the number of retries of the message to be synchronized to 0, sending the message to be synchronized to the message queue, so that the target slave system can obtain the message to be synchronized from the message queue, and increasing the number of retries of the message to be synchronized by 1.

[0008] Optionally, after receiving the receipt message sent by the target from the system and changing the status of the message according to the receipt message, the method also includes: when the receipt message is marked as unprocessed, determining the number of retries of the message corresponding to the receipt message; when the number of retries does not reach a second preset value, resending the message with the receipt message marked as unprocessed to the message queue, and increasing the number of retries by 1 based on the original number of retries; when the number of retries reaches the preset value, generating an alarm message to prompt that the synchronization operation of the message whose number of retries reaches the preset value has failed.

[0009] Optionally, the method also includes: every predetermined time period, determining that the message status is unprocessed and the message to be synchronized is a message to be retried, and obtaining the number of retries of the message to be retried. If the number of retries of the message to be retried does not reach a second preset value, sending the message to be retried to the message queue again, and increasing the number of retries by 1 based on the original number of retries. If the number of retries of the message to be retried reaches the preset value, generating an alarm message indicating an execution failure.

[0010] According to another aspect of the present application, a data synchronization method in a distributed system is provided, including: receiving a message queue sent by a main system, parsing the message queue to obtain a message, and processing the message to perform data synchronization, wherein the message includes an SQL instruction and execution information, the SQL instruction is an instruction generated by the main system according to a preset operation, and the preset operation includes at least one of adding, deleting, modifying and searching data in a target table in a database of the main system, and the target table is a database table corresponding to the SQL instruction and being operated.

[0011] According to another aspect of the present application, a main system is provided, including: a receiving unit, for receiving and generating an SQL instruction in response to a preset operation, the preset operation including at least one of adding, deleting, modifying and searching data in a target table in a database of the main system, the target table being an operated database table corresponding to the SQL instruction; a collating unit, for integrating the SQL instruction and execution information into a message, and sending the message to a message queue, so that a target slave system obtains and processes the message from the message queue to achieve data synchronization between the main system and the target slave system, wherein the execution information includes at least the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the need for data synchronization.

[0012] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0013] According to another aspect of the present application, a data synchronization system in a distributed system is provided, comprising: a master system and a target slave system, wherein the master system executes any one of the methods described, and the target slave system executes the method described.

[0014] The technical solution of the present application is applied. First, an SQL instruction is received and generated in response to a preset operation. The preset operation includes at least one operation of adding, deleting, modifying, and searching data in a target table in the database of the master system. The target table is the database table to be operated corresponding to the SQL instruction. Then, the SQL instruction and execution information are integrated into a message, and the message is sent to a message queue so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the master system and the target slave system. The execution information at least includes the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the system to be synchronized. The present application organizes SQL statements and their execution information into messages and sends them to a message queue so that the target slave system can receive and execute messages from the message queue. The master system and the target slave system communicate through the message queue. The message queue acts as an intermediate layer to store messages sent by the master system. The target slave system reads messages from the message queue instead of directly receiving messages from the master system. Therefore, direct thread links or real-time communication are no longer required between the master system and the slave system, making data synchronization more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0016] Figure 1 A schematic diagram of a data synchronization method in a distributed system according to an embodiment of the present application is shown;

[0017] Figure 2 A flow chart of another data synchronization method in a distributed system provided according to an embodiment of the present application is shown;

[0018] Figure 3 A flowchart of a main system sending a message to a message queue according to an embodiment of the present application is shown;

[0019] Figure 4A flowchart of a slave system processing a message according to an embodiment of the present application is shown;

[0020] Figure 5 A flowchart of a main system receipt processing provided according to an embodiment of the present application is shown;

[0021] Figure 6 A flowchart of a main system polling process provided according to an embodiment of the present application is shown;

[0022] Figure 7 A structural block diagram of a main system in a distributed system provided according to an embodiment of the present application is shown;

[0023] Figure 8 A structural block diagram of a target slave system provided according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] As introduced in the background technology, the flexibility of data synchronization in the existing technology is poor. In order to solve the above technical problems, the embodiments of the present application provide a data synchronization method, a main system, a computer-readable storage medium and a data synchronization system in a distributed system.

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0029] In this embodiment, a data synchronization method in a distributed system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 This is a flow chart of a data synchronization method in a distributed system according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0031] Step S201: Receive and generate an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying, and searching data in a target table in a database of the main system, and the target table is the database table to be operated corresponding to the SQL instruction;

[0032] Step S202: Integrate the above-mentioned SQL instructions and execution information into a message, and send the above-mentioned message to a message queue, so that the target slave system obtains and processes the above-mentioned message from the above-mentioned message queue to achieve data synchronization between the above-mentioned master system and the above-mentioned target slave system, wherein the above-mentioned execution information at least includes the above-mentioned target slave system corresponding to the above-mentioned SQL instruction, and the above-mentioned target slave system is a system that represents the need for data synchronization.

[0033] Specifically, the target slave system obtains messages from the message queue, parses the message content, extracts SQL instructions and execution information, and executes the SQL instructions in the target slave system to complete data synchronization. The message queue is a first-in-first-out linked list, that is, the message that enters the message queue first is processed first, and the messages are processed in the order they enter the message list. At the same time, after the master system completes its own task processing, it does not directly call the target slave system, but integrates the operation content into a message and puts it into the message queue. The target slave system receives the message in the message queue and performs subsequent operations according to the message content. The master system includes a master library and a message service, and the slave system includes a slave library and a message service. The message service is used to send and receive messages. The master library provides external data reading and writing operations, and the slave library only provides external data reading operations. When the data in the master library is written and modified externally, the modified content needs to be sent to each slave library so that the slave library can also be updated according to the modified content of the master library, so that the master and slave libraries remain consistent.

[0034] Through the above embodiment, first, an SQL instruction is received and generated in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying and searching data in a target table in the database of the master system, wherein the target table is the database table to be operated corresponding to the SQL instruction, and then the SQL instruction and execution information are integrated into a message, and the message is sent to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the master system and the target slave system, wherein the execution information at least includes the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the system to be synchronized with data. The present application organizes SQL statements and their execution information into messages and sends them to a message queue, so that the target slave system can receive and execute messages from the message queue. The master system and the target slave system communicate through the message queue. The message queue acts as an intermediate layer to store messages sent by the master system. The target slave system reads messages from the message queue instead of directly receiving messages from the master system. Therefore, there is no need for direct thread links or real-time communication between the master system and the slave system, making data synchronization more flexible.

[0035] In an optional solution, after integrating the above-mentioned SQL instructions and execution information into a message and sending the above-mentioned message to the message queue, so that the target slave system obtains and processes the above-mentioned message from the above-mentioned message queue, the above-mentioned method also includes: receiving a receipt message sent by the above-mentioned target slave system, changing the status of the above-mentioned message according to the above-mentioned receipt message, the above-mentioned receipt message is an identifier representing the status of the above-mentioned message, and the identifier of the above-mentioned status is one of the following: a processed identifier, an unprocessed identifier and a pending synchronization identifier; when the above-mentioned receipt message is the above-mentioned processed identifier, determining that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is successful execution, and determining that the message status of the above-mentioned message corresponding to the above-mentioned receipt message is processed; when the above-mentioned receipt message is the above-mentioned unprocessed identifier, determining that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is failed execution, and determining that the above-mentioned message status of the above-mentioned message corresponding to the above-mentioned receipt message is unprocessed; when the above-mentioned receipt message is the above-mentioned pending synchronization identifier, determining that the above-mentioned target slave system has not received the above-mentioned message corresponding to the above-mentioned receipt message, and determining that the above-mentioned message status of the above-mentioned message corresponding to the above-mentioned receipt message is pending synchronization.

[0036] In the above embodiment, by receiving the receipt message from the target slave system, the master system can promptly understand how the target slave system processes the message, which helps the master system to promptly discover potential problems, thereby improving the reliability and stability of message synchronization.

[0037] In another optional scheme, the above-mentioned execution information also includes the version number of the module to which the above-mentioned target table belongs, and the above-mentioned version number is an identifier that represents the version of the above-mentioned module. The above-mentioned module is a collection of tables with data association relationships. After sending the above-mentioned message to the message queue so that the target slave system can obtain and process the above-mentioned message from the above-mentioned message queue, thereby realizing data synchronization, the above-mentioned method also includes: determining the above-mentioned module to which the above-mentioned target table belongs based on the above-mentioned target table; updating the above-mentioned version number of the above-mentioned module, and each time the above-mentioned preset operation is performed on the table in the above-mentioned module, adjusting the above-mentioned version number to the sum of the current version number and the first preset value, and the initial value of the above-mentioned version number of the above-mentioned module is 1.

[0038] In the above embodiment, an identifier with an initial version number of 1 is assigned to each module, the version number is incremented each time a user operation is performed on a table in the module, and the SQL statement and the execution information carrying the module version number of the target table are organized into a message and sent to the message queue, so that the target can accurately identify the version corresponding to the message when obtaining the message in the message queue from the system.

[0039] A specific usage scenario of the version number of the above module is provided. Before data synchronization, a unique module identifier is assigned to each module and the module identifier is carried in the message. In the master system, a version number counter is configured for each module. Each time an SQL operation is performed on a table in the module, the version number counter is incremented and the new version number is included in the message and sent to the slave system. After receiving the message, the slave system first checks the module identifier and version number in the message to ensure that the data update operation in the same module is performed in the order of the version numbers. If the version number of the received message is discontinuous, it means that a message is lost. At this time, the slave system can send a message missing receipt to the master system to request the reissue of the missing message. For messages from different modules, the slave system does not need to execute them in the order of the version numbers and can process them concurrently to improve the overall performance of the system.

[0040] Data updates for different modules do not need to be executed sequentially to improve system concurrency performance;

[0041] In a specific use case, such as data synchronization between an e-commerce platform's main order system and inventory system, a change in order status in the order system (such as a successful payment) triggers the data synchronization process. Due to the strong correlation between order and inventory data, they are divided into the same module. When the order status is updated, the main order system adds an optimistic lock to the module and sends an update message to the inventory system. After receiving the message, the inventory system executes the inventory deduction operation in version order, ensuring the consistency of the order status change and inventory deduction sequence to avoid overselling or understocking. Furthermore, since inventory data for different products belongs to different modules, their updates can be executed concurrently, improving overall system performance.

[0042] In other exemplary embodiments, after determining that the target slave system has not received the message corresponding to the receipt message, the method further includes: determining that the message corresponding to the receipt message not received by the target slave system is a message to be synchronized; after initializing the number of retries of the message to be synchronized to 0, sending the message to be synchronized to the message queue, so that the target slave system can obtain the message to be synchronized from the message queue, and increasing the number of retries of the message to be synchronized by 1.

[0043] In the above embodiment, retrying pending synchronization messages ensures that all pending synchronization messages are ultimately received and processed by the target slave system. By initializing the retry count to 0 and incrementing it by 1 with each retry, the retry status of each pending synchronization message can be tracked until the message is successfully processed or the maximum retry count is reached. This process eliminates the need for manual intervention, improving the system's automatic recovery capabilities.

[0044] According to some further optional schemes of the present application, after receiving the receipt message sent by the above-mentioned target from the system and changing the status of the above-mentioned message according to the above-mentioned receipt message, the above-mentioned method also includes: in the case that the above-mentioned receipt message is the above-mentioned unprocessed mark, determining the number of retries of the above-mentioned message corresponding to the above-mentioned receipt message; when the above-mentioned number of retries does not reach the second preset value, resending the above-mentioned message with the above-mentioned receipt message as the above-mentioned unprocessed mark to the above-mentioned message queue, and increasing the above-mentioned number of retries by 1 based on the original number of retries; when the above-mentioned number of retries reaches the preset value, generating an alarm message to prompt that the synchronization operation of the above-mentioned message whose number of retries reaches the above-mentioned preset value has failed.

[0045] In the above embodiment, by resending messages with an unprocessed message receipt, the resending operation is automatically performed before the number of retries reaches a preset value, eliminating the need for manual intervention and improving the efficiency of message synchronization. When the number of retries for a message reaches the preset value, an alarm is generated, indicating that the synchronization operation for the message has failed, helping operations and maintenance personnel to promptly identify and address synchronization failures. By properly setting the number of retries, system resources can be rationally utilized while ensuring the synchronization success rate. This prevents excessive system resource usage due to unlimited retries, which can affect the normal operation of other services.

[0046] In some other exemplary schemes of the present application, the above method also includes: every predetermined time period, determining that the above message status is unprocessed and the above message to be synchronized is a message to be retried, and obtaining the retry count of the above message to be retried, and when the above retry count of the above message to be retried does not reach the second preset value, sending the above message to be retried to the above message queue again, and increasing the above retry count by 1 based on the original retry count, and generating an alarm message indicating execution failure when the above retry count of the above message to be retried reaches the above preset value.

[0047] In the above embodiment, by regularly scanning and identifying messages in the unprocessed and pending synchronization states as retry targets, the automation and continuity of the message processing process are ensured. By periodically obtaining the current retry count of messages to be retried and automatically incrementing it after each retry, the resource waste caused by unlimited retries is avoided, while providing multiple processing opportunities for messages and improving the synchronization success rate.

[0048] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the data synchronization method under the distributed system of the present application will be described in detail below with reference to specific embodiments.

[0049] This embodiment relates to a specific data synchronization method in a distributed system, such as Figure 2 As shown, the following steps are included:

[0050] Step S1: The main system generates SQL execution according to the preset operation, integrates the above SQL instructions and execution information into a message, and sends it to the message queue;

[0051] Step S2: The target slave system obtains and processes the message from the message queue, and sends the processing result to the master system via a receipt message;

[0052] Step S3: The main system processes the receipt message.

[0053] For example, the process of the main system sending a message to the message queue is as follows: Figure 3As shown, the following steps are included:

[0054] Step S11: When a preset operation is detected, the Java-based persistence layer framework is used to monitor the execution of instructions by the main library in the main system and format the statements.

[0055] Step S12: query the module definition table to obtain all subscription tables;

[0056] Specifically, the module definition table records the relationship between tables and modules, and the subscription table is a customized table that needs attention.

[0057] Step S13: Determine the table corresponding to the preset operation. If it is a preset operation of the subscription table, execute step S14;

[0058] Step S14: Determine the module to which the operated table belongs and add module-level optimistic lock;

[0059] Specifically, the above-mentioned module and optimistic lock manage optimistic lock version numbers based on a set of tables with interrelated data as a whole. The version number is an auto-increment array starting from 1. Each data operation on the table in the module will cause the module to increase the preset value based on the largest old version number, so that the new optimistic lock version number obtained by the module, and subsequent targets are executed in sequence according to the version number from the system message synchronization.

[0060] Step S15: converting the SQL instruction into a database execution model, and registering the database execution model and its operation content into the update content information record table;

[0061] Specifically, the database execution model is the execution method and strategy that the system uses to convert SQL instructions into actual operations when processing SQL instructions.

[0062] Step S16: query the module name association table to obtain the module to which the operated table belongs and the targets to which the slave system needs to be synchronized;

[0063] Specifically, the module name association table records the master system and target slave system corresponding to the operated table, as well as the module to which the target table belongs.

[0064] Step S17: Send the message to the message queue, register the message status in the message queue, and set the message status to "processing".

[0065] For example, the process of processing messages from the system is as follows: Figure 4 As shown, the following steps are included:

[0066] Step S21: Receive the message queue sent by the main system, parse the message queue to obtain the message, determine the sender system number of the message, the target table, the module to which the target table belongs, and the version number carried by the message, which is recorded as seq;

[0067] Step S22: According to the module and the system number of the message sender, the maximum version number value among the messages with the status of "processed" is obtained from the message queue, which is recorded as ycl;

[0068] Step S23: Determine whether seq is greater than ycl. If seq is greater than ycl, execute step S24; if seq is less than or equal to ycl, execute step S28.

[0069] Step S24: Determine whether seq is equal to ycl+1. If seq is equal to ycl+1, execute the SQL instruction in the message and send the execution result to the main system through a receipt message. If seq is not equal to ycl+1, if seq is greater than ycl+1, register the status of the message as unprocessed in the message queue.

[0070] Step S25: Obtain all messages in the "unprocessed" state belonging to this module and system, and add the above messages to the unprocessed message list;

[0071] Step S26: sort the messages in the unprocessed message list by version number from small to large, and obtain the version number of the smallest unprocessed message, which is recorded as wcl;

[0072] Step S27: Re-obtain the version number of the largest processed message, recorded as ycl;

[0073] Step S28: Determine whether ycl+1 is equal to wcl. If ycl+1 is equal to wcl, execute the SQL instructions in the unprocessed message list in sequence in a loop and send the execution results to the main system. If ycl+1 is not equal to wcl, execute the above steps S27 and S28 in a loop at least once until ycl+1 is not equal to wcl.

[0074] For example, the main system receipt processing flow is as follows: Figure 5 As shown, the following steps are included:

[0075] Step S31: The master system receives a receipt message from the slave system;

[0076] Step S32: Determine whether the message status in the receipt message is "processed" or "unprocessed". If the message status in the receipt message is "processed" or "unprocessed", update the message status in the message queue and execute step S33. If the message status in the receipt message is not "processed" or "unprocessed", execute step S34.

[0077] Step S33: Determine whether the message is successfully executed. If so, the process ends. If not, determine whether the target table of the message supports skipping upon failure. If so, the process ends. If not, an alarm is issued.

[0078] Step S34: Determine whether the message status in the receipt message is "pending synchronization". If the message status in the receipt message is "pending synchronization", obtain the content record of this message in the update content information record table, and query the module name association table to obtain the module to which the operated table belongs and which target systems need to be synchronized, send the message to the message queue again, and update the number of retries of the message in the message queue. If the message status in the receipt message is "pending synchronization", end the process.

[0079] For example, the main system polling process is as follows: Figure 6 As shown, the following steps are included:

[0080] Step S41: query the message list in the message queue with the status of "processing";

[0081] Step S42: loop through the message list;

[0082] Step S43: If the number of message retransmissions does not reach the upper limit, the message may be retransmitted;

[0083] Step S431: resending the message;

[0084] Step S432: Update the number of retransmissions in the message queue and add one;

[0085] Step S44: If the number of message retransmissions reaches the upper limit, it is considered that after sufficient waiting and retransmission, the slave system still fails to perform synchronization successfully, and an alarm is issued;

[0086] For example, the system polling process is as follows Figure 7 As shown, the following steps are included:

[0087] Step S51: Obtain all "unprocessed" data under each module of the current system, and process each module in a loop;

[0088] Step S52: Sort the "unprocessed" data of each module in ascending order of the version number, and obtain the minimum version number wcl of the "unprocessed" data under this module;

[0089] Step S53: Obtain the maximum version number ycl of the "processed" data under this module;

[0090] Step S54: If ycl + 1 < wcl, that is, the minimum "unprocessed" version number is greater than the maximum "processed" version number, and the two numbers are not consecutive;

[0091] Step S55: Query all version numbers between ycl and wcl in sequence;

[0092] Step S56: Check whether the message of this version exists;

[0093] Step S561: If the message does not exist, it is considered that the message is missing, and a message missing receipt is sent to the main system;

[0094] Step S562: Register a piece of data with the status of "pending processing" in the [Message Synchronization Record Table];

[0095] Step S57: If the message exists and the status is "pending synchronization", the subsequent steps can be executed;

[0096] Step S58: Check the message synchronization times;

[0097] Step S581: If the synchronization times have not reached the upper limit, send a message missing receipt to the main library;

[0098] Step S582: Register that the synchronization times are incremented by one;

[0099] [[ID=*33]]Step S59: If the message synchronization times have reached the upper limit, it is considered that the synchronization fails, and an alarm processing is performed. <* <*

[0100] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0101] The embodiment of the present application also provides a main system under a distributed system. It should be noted that the main system under the distributed system of the embodiment of the present application can be used to execute the data synchronization method for the distributed system provided by the embodiment of the present application. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and contemplated. Note: There seems to be an asterisk (*) added in the translation of step S59 in the original text you provided. I'm not sure if this is intentional. If it's a mistake, you can correct it according to the actual situation. Also, the tags and

[0100] in step S59 in the original text have an asterisk (*) added in the translation, which may need to be adjusted according to the correct content.

[0102] The following introduces the main system under the distributed system provided in the embodiment of the present application.

[0103] Figure 7 Schematic diagram of the main system under the distributed system according to the embodiment of the present application. Figure 7 As shown, the device includes:

[0104] A first receiving unit 20 is configured to receive and generate an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying, and searching data in a target table in a database of the main system, wherein the target table is a database table to be operated on corresponding to the SQL instruction;

[0105] The collating unit 30 is used to integrate the above-mentioned SQL instructions and execution information into a message, and send the above-mentioned message to a message queue, so that the target slave system obtains and processes the above-mentioned message from the above-mentioned message queue to achieve data synchronization between the above-mentioned master system and the above-mentioned target slave system, wherein the above-mentioned execution information at least includes the above-mentioned target slave system corresponding to the above-mentioned SQL instruction, and the above-mentioned target slave system is a system that represents the system that needs to be synchronized with data.

[0106] Specifically, the target slave system obtains the message from the message queue, parses the message content, extracts the SQL instruction and execution information, and executes the SQL instruction in the target slave system to complete data synchronization.

[0107] Specifically, the message queue is a first-in-first-out linked list, that is, the message that enters the message queue earliest is processed first, and is processed in the order in which the messages enter the message list. At the same time, after the main system completes its own task processing, it does not directly call the target slave system, but integrates the operation content into a message and puts it into the message queue. The target slave system receives the message in the message queue and performs subsequent operations according to the message content.

[0108] Specifically, the master system includes a master database and a message service, while the slave system includes slave databases and a message service. The message service is used to send and receive messages. The master database provides external data read and write operations, while the slave database only provides external data read operations. When data in the master database is modified by external writes, the modified content needs to be sent to each slave database so that the slave database can also update according to the changes in the master database, ensuring consistency between the master and slave databases.

[0109] Through the above embodiment, the first receiving unit first receives and generates an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying and searching data in a target table in the database of the master system, wherein the target table is the database table to be operated corresponding to the SQL instruction, and then the sorting unit integrates the SQL instruction and execution information into a message, and sends the message to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the master system and the target slave system, wherein the execution information at least includes the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the system to be synchronized with data. The present application organizes the SQL statement and its execution information into a message and sends it to a message queue, so that the target slave system can receive and execute the message from the message queue. The master system and the target slave system communicate through the message queue. The message queue acts as an intermediate layer to store the messages sent by the master system. The target slave system reads the message from the message queue instead of directly receiving the message from the master system. Therefore, there is no need for direct thread link or real-time communication between the master system and the slave system, making data synchronization more flexible.

[0110] In an optional scheme, the above-mentioned device also includes: a second receiving unit, used to receive a receipt message sent by the above-mentioned target from the system, and change the status of the above-mentioned message according to the above-mentioned receipt message, the above-mentioned receipt message is an identifier representing the status of the above-mentioned message, and the identifier of the above-mentioned status is one of the following: a processed identifier, an unprocessed identifier and a pending synchronization identifier; a first determination unit, used to determine that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is a successful execution when the above-mentioned receipt message is the above-mentioned processed identifier, and determine that the message status of the above-mentioned message corresponding to the above-mentioned receipt message is processed; a second determination unit, used to determine that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is a failed execution when the above-mentioned receipt message is the above-mentioned unprocessed identifier, and determine that the message status of the above-mentioned message corresponding to the above-mentioned receipt message is unprocessed; a third determination unit, used to determine that the above-mentioned target from the system has not received the above-mentioned message corresponding to the above-mentioned receipt message when the above-mentioned receipt message is the above-mentioned pending synchronization identifier, and determine that the message status of the above-mentioned message corresponding to the above-mentioned receipt message is pending synchronization.

[0111] In the above embodiment, by receiving the receipt message from the target slave system, the master system can promptly understand how the target slave system processes the message, which helps the master system to promptly discover potential problems, thereby improving the reliability and stability of message synchronization.

[0112] In an optional scheme, the above-mentioned device also includes: a fourth determination unit, used to determine the above-mentioned module to which the above-mentioned target table belongs based on the above-mentioned target table; an update unit, used to update the above-mentioned version number of the above-mentioned module, and each time the above-mentioned preset operation is performed on the table in the above-mentioned module, the above-mentioned version number is adjusted to the sum of the current version number and the first preset value, and the initial value of the above-mentioned version number of the above-mentioned module is 1.

[0113] In the above embodiment, an identifier with an initial version number of 1 is assigned to each module, the version number is incremented each time a user operation is performed on a table in the module, and the SQL statement and the execution information carrying the module version number of the target table are organized into a message and sent to the message queue, so that the target can accurately identify the version corresponding to the message when obtaining the message in the message queue from the system.

[0114] In an optional scheme, the above-mentioned third determination unit includes: a first determination module, used to determine that the above-mentioned message corresponding to the above-mentioned receipt message not received by the above-mentioned target slave system is a message to be synchronized; a sending module, used to send the above-mentioned message to be synchronized to the above-mentioned message queue after initializing the number of retries of the above-mentioned message to be synchronized to 0, so that the above-mentioned target slave system can obtain the above-mentioned message to be synchronized from the above-mentioned message queue, and increase the number of retries of the above-mentioned message to be synchronized by 1.

[0115] In the above embodiment, retrying pending synchronization messages ensures that all pending synchronization messages are ultimately received and processed by the target slave system. By initializing the retry count to 0 and incrementing it by 1 with each retry, the retry status of each pending synchronization message can be tracked until the message is successfully processed or the maximum retry count is reached. This process eliminates the need for manual intervention, improving the system's automatic recovery capabilities.

[0116] In an optional scheme, the above-mentioned second receiving unit includes: a second determination module, which is used to determine the number of retries of the above-mentioned message corresponding to the above-mentioned receipt message when the above-mentioned receipt message is marked as the above-mentioned unprocessed mark; a retransmission module, which is used to retransmit the above-mentioned message with the above-mentioned receipt message marked as the above-mentioned unprocessed mark to the above-mentioned message queue when the above-mentioned number of retries does not reach a second preset value, and increase the above-mentioned number of retries by 1 based on the original number of retries; a generation module, which is used to generate an alarm message when the above-mentioned number of retries reaches the preset value, to prompt that the synchronization operation of the above-mentioned message whose number of retries reaches the above-mentioned preset value has failed.

[0117] In the above embodiment, by resending messages with an unprocessed message receipt, the resending operation is automatically performed before the number of retries reaches a preset value, eliminating the need for manual intervention and improving the efficiency of message synchronization. When the number of retries for a message reaches the preset value, an alarm is generated, indicating that the synchronization operation for the message has failed, helping operations and maintenance personnel to promptly identify and address synchronization failures. By properly setting the number of retries, system resources can be rationally utilized while ensuring the synchronization success rate. This prevents excessive system resource usage due to unlimited retries, which can affect the normal operation of other services.

[0118] In an optional scheme, the above-mentioned device also includes: a fifth determination unit, which is used to determine, every predetermined time period, that the message status is unprocessed and the message to be synchronized is a message to be retried, and obtain the retry count of the message to be retried. If the retry count of the message to be retried does not reach a second preset value, the message to be retried is sent to the message queue again, and the retry count is increased by 1 based on the original retry count. If the retry count of the message to be retried reaches the preset value, an alarm message indicating an execution failure is generated.

[0119] In the above embodiment, by regularly scanning and identifying messages in the unprocessed and pending synchronization states as retry targets, the automation and continuity of the message processing process are ensured. By periodically obtaining the current retry count of messages to be retried and automatically incrementing it after each retry, the resource waste caused by unlimited retries is avoided, while providing multiple processing opportunities for messages and improving the synchronization success rate.

[0120] The main system of the distributed system includes a processor and memory. The first receiving unit and the collating unit are stored in the memory as program units, and the processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0121] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and the problem of poor flexibility of data synchronization in the prior art can be at least solved by adjusting kernel parameters.

[0122] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0123] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is running, the device where the computer-readable storage medium is located is controlled to execute the data synchronization method under the distributed system.

[0124] Specifically, the data synchronization method in a distributed system includes:

[0125] Step S201: Receive and generate an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying, and searching data in a target table in a database of the main system, and the target table is the database table to be operated corresponding to the SQL instruction;

[0126] Step S202: Integrate the above-mentioned SQL instructions and execution information into a message, and send the above-mentioned message to a message queue, so that the target slave system obtains and processes the above-mentioned message from the above-mentioned message queue to achieve data synchronization between the above-mentioned master system and the above-mentioned target slave system, wherein the above-mentioned execution information at least includes the above-mentioned target slave system corresponding to the above-mentioned SQL instruction, and the above-mentioned target slave system is a system that represents the need for data synchronization.

[0127] Specifically, the target slave system obtains the message from the message queue, parses the message content, extracts the SQL instruction and execution information, and executes the SQL instruction in the target slave system to complete data synchronization.

[0128] Specifically, the message queue is a first-in-first-out linked list, that is, the message that enters the message queue earliest is processed first, and is processed in the order in which the messages enter the message list. At the same time, after the main system completes its own task processing, it does not directly call the target slave system, but integrates the operation content into a message and puts it into the message queue. The target slave system receives the message in the message queue and performs subsequent operations according to the message content.

[0129] Specifically, the master system includes a master database and a message service, while the slave system includes slave databases and a message service. The message service is used to send and receive messages. The master database provides external data read and write operations, while the slave database only provides external data read operations. When data in the master database is modified by external writes, the modified content needs to be sent to each slave database so that the slave database can also update according to the changes in the master database, ensuring consistency between the master and slave databases.

[0130] In one embodiment of the present application, after integrating the above-mentioned SQL instructions and execution information into a message and sending the above-mentioned message to the message queue, so that the target slave system obtains and processes the above-mentioned message from the above-mentioned message queue, the above-mentioned method also includes: receiving a receipt message sent by the above-mentioned target slave system, changing the status of the above-mentioned message according to the above-mentioned receipt message, the above-mentioned receipt message is an identifier representing the status of the above-mentioned message, and the identifier of the above-mentioned status is one of the following: a processed identifier, an unprocessed identifier and a pending synchronization identifier; when the above-mentioned receipt message is the above-mentioned processed identifier, determining that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is successful execution, and determining that the message status of the above-mentioned message corresponding to the above-mentioned receipt message is processed; when the above-mentioned receipt message is the above-mentioned unprocessed identifier, determining that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is failed execution, and determining that the above-mentioned message status of the above-mentioned message corresponding to the above-mentioned receipt message is unprocessed; when the above-mentioned receipt message is the above-mentioned pending synchronization identifier, determining that the above-mentioned target slave system has not received the above-mentioned message corresponding to the above-mentioned receipt message, and determining that the above-mentioned message status of the above-mentioned message corresponding to the above-mentioned receipt message is pending synchronization.

[0131] In one embodiment of the present application, the above-mentioned execution information also includes the version number of the module to which the above-mentioned target table belongs, the above-mentioned version number is an identifier that represents the version of the above-mentioned module, and the above-mentioned module is a collection of tables with data association relationships. After sending the above-mentioned message to the message queue so that the target slave system can obtain and process the above-mentioned message from the above-mentioned message queue, thereby realizing data synchronization, the above-mentioned method also includes: determining the above-mentioned module to which the above-mentioned target table belongs based on the above-mentioned target table; updating the above-mentioned version number of the above-mentioned module, and each time the above-mentioned preset operation is performed on the table in the above-mentioned module, adjusting the above-mentioned version number to the sum of the current version number and the first preset value, and the initial value of the above-mentioned version number of the above-mentioned module is 1.

[0132] In one embodiment of the present application, after determining that the target slave system has not received the message corresponding to the receipt message, the method further includes: determining that the message corresponding to the receipt message not received by the target slave system is a message to be synchronized; after initializing the number of retries of the message to be synchronized to 0, sending the message to be synchronized to the message queue, so that the target slave system can obtain the message to be synchronized from the message queue, and increasing the number of retries of the message to be synchronized by 1.

[0133] In one embodiment of the present application, after receiving the receipt message sent by the above-mentioned target from the system and changing the status of the above-mentioned message according to the above-mentioned receipt message, the above-mentioned method also includes: when the above-mentioned receipt message is the above-mentioned unprocessed mark, determining the number of retries of the above-mentioned message corresponding to the above-mentioned receipt message; when the above-mentioned number of retries does not reach a second preset value, resending the above-mentioned message with the above-mentioned receipt message as the above-mentioned unprocessed mark to the above-mentioned message queue, and increasing the above-mentioned number of retries by 1 based on the original number of retries; when the above-mentioned number of retries reaches the preset value, generating an alarm message to prompt that the synchronization operation of the above-mentioned message whose number of retries reaches the above-mentioned preset value has failed.

[0134] In one embodiment of the present application, the above method also includes: every predetermined time period, determining that the above message status is unprocessed and the above message to be synchronized is a message to be retried, and obtaining the retry count of the above message to be retried, and when the above retry count of the above message to be retried does not reach a second preset value, sending the above message to be retried to the above message queue again, and increasing the above retry count by 1 based on the original retry count, and generating an alarm message indicating execution failure when the above retry count of the above message to be retried reaches the above preset value.

[0135] This application also provides a second data synchronization method in a distributed system, which includes the following steps:

[0136] Step S301: receiving a message queue sent by the main system, parsing the message queue to obtain a message, and processing the message to perform data synchronization.

[0137] In one embodiment of the present application, the above-mentioned message includes SQL instructions and execution information. The above-mentioned SQL instructions are instructions generated by the main system according to preset operations. The above-mentioned preset operations include at least one operation of adding, deleting, modifying and searching the data of the target table in the database of the above-mentioned main system. The above-mentioned target table is the database table corresponding to the above-mentioned SQL instructions.

[0138] In the above steps, by receiving the message queue sent by the main system, parsing the message queue to obtain the message, and processing the message to perform data synchronization, the message can be received and executed from the message queue. The main system and the target slave system communicate through the message queue. The message queue serves as an intermediate layer to store the messages sent by the main system. The target slave system reads the message from the message queue instead of receiving the message directly from the main system. Therefore, there is no need for direct thread link or real-time communication between the main system and the slave system, making data synchronization more flexible.

[0139] In one embodiment of the present application, after integrating the above-mentioned SQL instructions and execution information into a message and sending the above-mentioned message to the message queue, so that the target slave system obtains and processes the above-mentioned message from the above-mentioned message queue, the above-mentioned method also includes: receiving a receipt message sent by the above-mentioned target slave system, changing the status of the above-mentioned message according to the above-mentioned receipt message, the above-mentioned receipt message is an identifier representing the status of the above-mentioned message, and the identifier of the above-mentioned status is one of the following: a processed identifier, an unprocessed identifier and a pending synchronization identifier; when the above-mentioned receipt message is the above-mentioned processed identifier, determining that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is successful execution, and determining that the message status of the above-mentioned message corresponding to the above-mentioned receipt message is processed; when the above-mentioned receipt message is the above-mentioned unprocessed identifier, determining that the execution result of the above-mentioned message corresponding to the above-mentioned receipt message is failed execution, and determining that the above-mentioned message status of the above-mentioned message corresponding to the above-mentioned receipt message is unprocessed; when the above-mentioned receipt message is the above-mentioned pending synchronization identifier, determining that the above-mentioned target slave system has not received the above-mentioned message corresponding to the above-mentioned receipt message, and determining that the above-mentioned message status of the above-mentioned message corresponding to the above-mentioned receipt message is pending synchronization.

[0140] In one embodiment of the present application, the above-mentioned execution information also includes the version number of the module to which the above-mentioned target table belongs, the above-mentioned version number is an identifier that represents the version of the above-mentioned module, and the above-mentioned module is a collection of tables with data association relationships. After sending the above-mentioned message to the message queue so that the target slave system can obtain and process the above-mentioned message from the above-mentioned message queue, thereby realizing data synchronization, the above-mentioned method also includes: determining the above-mentioned module to which the above-mentioned target table belongs based on the above-mentioned target table; updating the above-mentioned version number of the above-mentioned module, and each time the above-mentioned preset operation is performed on the table in the above-mentioned module, adjusting the above-mentioned version number to the sum of the current version number and the first preset value, and the initial value of the above-mentioned version number of the above-mentioned module is 1.

[0141] In one embodiment of the present application, after determining that the target slave system has not received the message corresponding to the receipt message, the method further includes: determining that the message corresponding to the receipt message not received by the target slave system is a message to be synchronized; after initializing the number of retries of the message to be synchronized to 0, sending the message to be synchronized to the message queue, so that the target slave system can obtain the message to be synchronized from the message queue, and increasing the number of retries of the message to be synchronized by 1.

[0142] In one embodiment of the present application, after receiving the receipt message sent by the above-mentioned target from the system and changing the status of the above-mentioned message according to the above-mentioned receipt message, the above-mentioned method also includes: when the above-mentioned receipt message is the above-mentioned unprocessed mark, determining the number of retries of the above-mentioned message corresponding to the above-mentioned receipt message; when the above-mentioned number of retries does not reach a second preset value, resending the above-mentioned message with the above-mentioned receipt message as the above-mentioned unprocessed mark to the above-mentioned message queue, and increasing the above-mentioned number of retries by 1 based on the original number of retries; when the above-mentioned number of retries reaches the preset value, generating an alarm message to prompt that the synchronization operation of the above-mentioned message whose number of retries reaches the above-mentioned preset value has failed.

[0143] In one embodiment of the present application, the above method also includes: every predetermined time period, determining that the above message status is unprocessed and the above message to be synchronized is a message to be retried, and obtaining the retry count of the above message to be retried, and when the above retry count of the above message to be retried does not reach a second preset value, sending the above message to be retried to the above message queue again, and increasing the above retry count by 1 based on the original retry count, and generating an alarm message indicating execution failure when the above retry count of the above message to be retried reaches the above preset value.

[0144] This application also provides a target slave system, such as Figure 8 As shown, the slave system includes: a third receiving unit 40, which is used to receive a message queue sent by the master system, parse the message queue to obtain a message, and process the message to perform data synchronization, wherein the message includes an SQL instruction and execution information, the SQL instruction is an instruction generated by the master system according to a preset operation, and the preset operation includes at least one of adding, deleting, modifying and searching the data of the target table in the database of the master system, and the target table is the operated database table corresponding to the SQL instruction.

[0145] The target slave system first receives and generates an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying and searching the data of the target table in the database of the master system, wherein the target table is the database table to be operated corresponding to the SQL instruction, and then the SQL instruction and execution information are integrated into a message, and the message is sent to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the master system and the target slave system, wherein the execution information at least includes the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the need for data synchronization. The present application organizes SQL statements and their execution information into messages and sends them to a message queue, so that the target slave system can receive and execute messages from the message queue. The master system and the target slave system communicate through the message queue. The message queue acts as an intermediate layer to store messages sent by the master system. The target slave system reads messages from the message queue instead of directly receiving messages from the master system. Therefore, there is no need for direct thread links or real-time communication between the master system and the slave system, making data synchronization more flexible.

[0146] In one embodiment of the present application, after the target slave system obtains and processes the message from the message queue, the method further includes: sending a receipt message, where the receipt message is an identifier representing a status of the message, where the status identifier is one of the following: a processed identifier, an unprocessed identifier, and a pending synchronization identifier;

[0147] The present application also provides a data synchronization system in a distributed system, including: a master system and a target slave system, the master system executes any one of the above methods, and the target slave system executes the above method.

[0148] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0149] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0153] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0154] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0155] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0156] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0157] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0158] 1) The data synchronization method under the distributed system of the present application first receives and generates an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying and searching data in a target table in the database of the master system, wherein the target table is the database table to be operated corresponding to the SQL instruction, and then the SQL instruction and execution information are integrated into a message, and the message is sent to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the master system and the target slave system, wherein the execution information at least includes the target slave system corresponding to the SQL instruction, and the target slave system is a system representing the system to be synchronized. The present application organizes the SQL statement and its execution information into a message and sends it to a message queue, so that the target slave system can receive and execute the message from the message queue. The master system and the target slave system communicate through the message queue. The message queue acts as an intermediate layer to store the messages sent by the master system. The target slave system reads the message from the message queue instead of directly receiving the message from the master system. Therefore, there is no need for direct thread link or real-time communication between the master system and the slave system, making data synchronization more flexible.

[0159] 2) In the main system of the present application, the first receiving unit first receives and generates an SQL instruction in response to a preset operation. The preset operation includes at least one of adding, deleting, modifying and searching data in a target table in the database of the main system. The target table is the database table to be operated corresponding to the SQL instruction. Then, the sorting unit integrates the SQL instruction and execution information into a message and sends the message to a message queue so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the main system and the target slave system. The execution information at least includes the target slave system corresponding to the SQL instruction. The target slave system is a system that represents the system that needs to synchronize data. The present application organizes the SQL statement and its execution information into a message and sends it to a message queue so that the target slave system can receive and execute the message from the message queue. The main system and the target slave system communicate through the message queue. The message queue acts as an intermediate layer to store the messages sent by the main system. The target slave system reads the message from the message queue instead of directly receiving the message from the main system. Therefore, there is no need for direct thread link or real-time communication between the main system and the slave system, making data synchronization more flexible.

[0160] 3) The data synchronization system under the distributed system of the present application receives the message queue sent by the main system, parses the above message queue to obtain messages, and processes the above messages to perform data synchronization. It can receive and execute messages from the message queue. The main system and the target slave system communicate through the message queue. The message queue serves as an intermediate layer to store messages sent by the main system. The target slave system reads messages from the message queue instead of receiving messages directly from the main system. Therefore, there is no need for direct thread links or real-time communication between the main system and the slave system, making data synchronization more flexible.

[0161] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A data synchronization method in a distributed system, characterized in that: include: Receive and generate an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying, and searching data in a target table in a database of the main system, and the target table is a database table to be operated corresponding to the SQL instruction; The SQL instruction and execution information are integrated into a message, and the message is sent to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the master system and the target slave system, wherein the execution information at least includes the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the system that needs to be synchronized with data.

2. The method according to claim 1, characterized in that After integrating the SQL instruction and execution information into a message and sending the message to a message queue so that the target slave system obtains and processes the message from the message queue, the method further includes: receiving a receipt message sent by the target slave system, and changing the state of the message according to the receipt message, wherein the receipt message is an identifier representing the state of the message, and the identifier of the state is one of the following: a processed identifier, an unprocessed identifier, and a pending synchronization identifier; In a case where the receipt message indicates the processed identifier, determining that the execution result of the message corresponding to the receipt message is successful, and determining that the message status of the message corresponding to the receipt message is processed; In a case where the receipt message indicates the unprocessed flag, determining that the execution result of the message corresponding to the receipt message is execution failure, and determining that the message status of the message corresponding to the receipt message is unprocessed; In a case where the receipt message is the pending synchronization identifier, it is determined that the target slave system has not received the message corresponding to the receipt message, and the message status of the message corresponding to the receipt message is determined to be pending synchronization.

3. The method according to claim 1, characterized in that The execution information also includes a version number of the module to which the target table belongs, where the version number is an identifier representing the version of the module. The module is a collection of tables with data associations. After sending the message to a message queue so that the target slave system can obtain and process the message from the message queue, thereby achieving data synchronization, the method further includes: Determining the module to which the target table belongs according to the target table; The version number of the module is updated, and each time the preset operation is performed on the table in the module, the version number is adjusted to the sum of the current version number and a first preset value, and the initial value of the version number of the module is 1.

4. The method according to claim 2, characterized in that After determining that the target slave system has not received the message corresponding to the receipt message, the method further includes: Determining that the message corresponding to the receipt message not received by the target slave system is a message to be synchronized; After initializing the number of retries of the message to be synchronized to 0, the message to be synchronized is sent to the message queue so that the target slave system can obtain the message to be synchronized from the message queue, and the number of retries of the message to be synchronized is increased by 1.

5. The method according to claim 2, characterized in that After receiving the receipt message sent by the target slave system and changing the status of the message according to the receipt message, the method further includes: In a case where the receipt message indicates the unprocessed mark, determining the number of retries of the message corresponding to the receipt message; When the number of retries does not reach a second preset value, resending the message with the receipt message being the unprocessed flag to the message queue, and increasing the number of retries by 1 based on the original number of retries; When the number of retries reaches a preset value, an alarm message is generated to prompt that the synchronization operation of the message whose number of retries reaches the preset value has failed.

6. The method according to claim 2, characterized in that The method further comprises: Every predetermined time period, determine that the message status is unprocessed and the message to be synchronized is a message to be retried, and obtain the retry count of the message to be retried. If the retry count of the message to be retried does not reach a second preset value, send the message to be retried to the message queue again, and increase the retry count by 1 based on the original retry count. If the retry count of the message to be retried reaches the preset value, generate an alarm message indicating execution failure.

7. A data synchronization method in a distributed system, characterized in that: include: Receive the message queue sent by the main system, parse the message queue to obtain the message, and process the message to synchronize data. The message includes an SQL instruction and execution information. The SQL instruction is an instruction generated by the main system according to a preset operation. The preset operation includes at least one of adding, deleting, modifying and searching the data of the target table in the database of the main system. The target table is the database table corresponding to the SQL instruction.

8. A main system, characterized in that: include: a receiving unit, configured to receive and generate an SQL instruction in response to a preset operation, wherein the preset operation includes at least one of adding, deleting, modifying, and searching data in a target table in a database of the main system, the target table being the database table to be operated on corresponding to the SQL instruction; A collating unit is used to integrate the SQL instruction and execution information into a message, and send the message to a message queue, so that the target slave system obtains and processes the message from the message queue to achieve data synchronization between the master system and the target slave system, wherein the execution information at least includes the target slave system corresponding to the SQL instruction, and the target slave system is a system that represents the system that needs to be synchronized with data.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 4.

10. A data synchronization system in a distributed system, characterized in that: include: A master system and a target slave system, wherein the master system executes the method according to any one of claims 1 to 6, and the target slave system executes the method according to claim 7.

Images