A data synchronization method, system and storage medium

Abstract

Embodiments of the present application disclose a data synchronization method, which comprises: if a target operation statement belonging to a target operation type is detected for a target data table, determining at least one second storage node having an association relationship with the target data table; wherein the target operation statement is used to change content in the target data table; determining a data synchronization mode of each second storage node; and based on each data synchronization mode, sending the target operation statement to the corresponding second storage node to realize synchronized updating of the target data table in the first storage node and the second storage node. Embodiments of the present application also disclose a data synchronization system and a storage medium.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data synchronization method, system and storage medium. Background Technology

[0002] With the rapid development of computer technology, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology. As user needs continue to grow, distributed storage systems are being widely used to ensure a good user experience. In distributed storage system applications, when data in one storage system needs to be modified (e.g., adding, deleting, or updating), the corresponding replicas in other systems must also be modified accordingly to ensure data consistency.

[0003] Currently, when modifications need to be made to data in one storage system, and corresponding modifications are also required to replicas in other systems, the common approach is to simultaneously send modification requests to the other systems. However, in this implementation, each of the other systems needs to maintain a copy of the same code. Because business systems iterate rapidly, the maintenance personnel for these other systems are often not the same. This leads to inconsistencies in the code maintained by the systems, resulting in delayed responses to modification requests, data inconsistencies in the distributed storage system, and low data synchronization efficiency.

[0004] Application content

[0005] To address the aforementioned technical problems, this application aims to provide a data synchronization method, device, and storage medium, which solves the current data synchronization efficiency problem in distributed storage systems, effectively ensures data consistency in distributed storage systems, and improves data synchronization efficiency in distributed storage systems.

[0006] The technical solution of this application is implemented as follows:

[0007] In a first aspect, a data synchronization method is applied to a first storage node, the method comprising:

[0008] If a target operation statement of the target operation type is detected targeting a target data table, at least one second storage node associated with the target data table is determined; wherein the target operation statement is used to change the content of the target data table;

[0009] Determine the data synchronization method for each of the second storage nodes;

[0010] Based on each of the aforementioned data synchronization methods, the target operation statement is sent to the corresponding second storage node to achieve synchronous updates of the target data table in the first and second storage nodes.

[0011] Secondly, a data synchronization system, the system comprising: a first storage node and at least one second storage node; wherein:

[0012] The first storage node is configured to, if a target operation statement of the target operation type is detected targeting a target data table, determine at least one second storage node associated with the target data table; wherein the target operation statement is used to change the content of the target data table; determine the data synchronization method for each second storage node; and send the target operation statement to the corresponding second storage node based on each data synchronization method.

[0013] The second storage node is used to receive the target operation statement; and to perform an update operation on the target data table corresponding to the target operation statement in the second storage node using the target operation statement to obtain an update result.

[0014] Thirdly, a storage medium storing a data synchronization program, which, when executed by a processor, implements the steps of the data synchronization method as described in any of the preceding claims.

[0015] In this embodiment, if a target operation statement of the target operation type is detected for the target data table, the first storage node determines at least one second storage node that is associated with the target data table, determines the data synchronization method for each second storage node, and sends the target operation statement to the corresponding second storage node based on each data synchronization method, so that the second storage node executes the target operation statement and realizes the synchronous update operation of the first storage node for the target data table. This solves the problem of data synchronization efficiency in the current distributed storage system, effectively ensures the consistency of data in the distributed storage system, and improves the data synchronization efficiency in the distributed storage system. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a data synchronization method provided in an embodiment of this application;

[0017] Figure 2 A flowchart illustrating another data synchronization method provided in an embodiment of this application;

[0018] Figure 3 A flowchart illustrating yet another data synchronization method provided in an embodiment of this application;

[0019] Figure 4A flowchart illustrating a data synchronization method provided in another embodiment of this application;

[0020] Figure 5 A flowchart illustrating another data synchronization method provided in another embodiment of this application;

[0021] Figure 6 A flowchart illustrating yet another data synchronization method provided in another embodiment of this application;

[0022] Figure 7 A flowchart illustrating a data synchronization method provided in another embodiment of this application;

[0023] Figure 8 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0024] Figure 9 This is a schematic diagram illustrating an alarm notification implementation process provided in an embodiment of this application.

[0025] Figure 10 This is a schematic diagram of the structure of a data synchronization system provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0027] The embodiments of this application provide a data synchronization method, referring to... Figure 1 As shown, the method is applied to the first storage node, and the method includes the following steps:

[0028] Step 101: If a target operation statement of the target operation type is detected for the target data table, determine at least one second storage node that is associated with the target data table.

[0029] The target operation statement is used to change the contents of the target data table.

[0030] In this embodiment, the first storage node is a distributed storage node where the user modifies the content of the target data table. The target operation type is mainly an operation type used to modify the content in the data table. At least one second storage node associated with the target data table refers to the storage node that stores the target data table.

[0031] Step 102: Determine the data synchronization method for each second storage node.

[0032] In this embodiment, different data synchronization methods are configured for different second storage nodes. During the synchronization process, the data synchronization method for each second storage node can be adjusted according to the actual synchronization situation. The data synchronization methods include at least real-time synchronization and asynchronous synchronization.

[0033] Step 103: Based on each data synchronization method, send the target operation statement to the corresponding second storage node.

[0034] Specifically, based on each data synchronization method, the target operation statement is sent to the corresponding second storage node to achieve synchronous updates of the target data table in the first and second storage nodes.

[0035] In this embodiment, after the first storage node determines the data synchronization method of each second storage node, it sends the target operation statement to the corresponding second storage node based on the data synchronization method of each second storage node. This allows the second storage node to directly execute the target operation statement after receiving it, thereby adjusting and updating the data content of the target data table in the second storage node.

[0036] In this embodiment, if a target operation statement of the target operation type is detected for the target data table, the first storage node determines at least one second storage node that is associated with the target data table, determines the data synchronization method for each second storage node, and sends the target operation statement to the corresponding second storage node based on each data synchronization method, so that the second storage node executes the target operation statement and realizes the synchronous update operation of the first storage node for the target data table. This solves the problem of data synchronization efficiency in the current distributed storage system, effectively ensures the consistency of data in the distributed storage system, and improves the data synchronization efficiency in the distributed storage system.

[0037] Based on the foregoing embodiments, embodiments of this application provide a data synchronization method, which is applied to a first storage node, and includes the following steps:

[0038] Step 201: If a target operation statement of the target operation type is detected for the target data table, determine at least one second storage node that is associated with the target data table.

[0039] The target operation statement is used to change the contents of the target data table.

[0040] In this embodiment, the target operation statement is a statement belonging to the target operation type in the operation instruction statement of the first storage node when the user performs corresponding operation instructions on the target data table of the first storage node. This statement can be directly responded to by the database in the storage node. The target operation type includes at least the following types: Insert, Delete, and Update, and the determined target operation statement belongs to at least one of these target operation types. When determining at least one second storage node associated with the target operation table, it can be determined from the relationship between pre-stored data tables and storage nodes. For example, after determining the target data table, the identification information of the target data table can be determined, and the storage node with the identification information of the target data table can be determined from the relationship between pre-stored data tables and storage nodes, thus obtaining at least one second storage node. The target operation statement can be, for example, a Structured Query Language (SQL) type statement that can directly respond to the database in the storage node, such as an InsertSQL statement, a DeleteSQL statement, or an Update statement for the target data table.

[0041] Step 202: Determine the data synchronization method for each second storage node.

[0042] In this embodiment of the application, the configuration parameters of each second storage node are determined to obtain the data synchronization method of each second storage node, thereby obtaining the data synchronization method of at least one second storage node.

[0043] Step 203: Based on each data synchronization method, send the target operation statement to the corresponding second storage node.

[0044] The target operation statement is used to update the target data table in the corresponding second storage node, so as to realize the synchronous update of the target data table in the first storage node and the second storage node.

[0045] In this embodiment of the application, when the data synchronization method of the second storage node is real-time synchronization, the target operation statement can be sent to the corresponding second storage node in real-time synchronization. When the data synchronization method of the second storage node is asynchronous synchronization, the target operation statement can be sent to the second storage node in asynchronous synchronization according to a preset asynchronous time period.

[0046] Based on the foregoing embodiments, in other embodiments of this application, step 203 can be implemented by steps 203a to 203c:

[0047] Step 203a: If each data synchronization method is real-time synchronization, determine the table creation statement corresponding to the target data table.

[0048] In this embodiment of the application, when the data synchronization mode of the second storage node is real-time synchronization mode, the node configuration information of the second storage node is read, and the table creation statement corresponding to the target data table is obtained from the node configuration information.

[0049] Step 203b: Generate the first request message based on the table creation statement and the target operation statement.

[0050] In this embodiment of the application, the table creation statement and the target operation statement are respectively filled into the request message to obtain the first request message.

[0051] Step 203c: Send the first request message to the corresponding second storage node via real-time synchronization.

[0052] The first request message includes a target operation statement. The table creation statement included in the first request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the first request message.

[0053] In this embodiment of the application, the first request message is sent to the corresponding second storage node in real time through real-time synchronization, thereby realizing the sending of the target operation statement to the second storage node.

[0054] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 2 As shown, before the first storage node executes step 203, it is also used to execute step 204:

[0055] Step 204: Determine the target data volume corresponding to the target operation statement.

[0056] In this embodiment of the application, the first storage node determines the amount of data included in the data table content corresponding to the target operation statement. After determining the number of fields included in the target data table, the target data amount is obtained by calculating the quotient between the amount of data included in the data table content corresponding to the target operation statement and the number of fields.

[0057] Correspondingly, step 203 can be implemented by step 203d:

[0058] Step 203d: Based on the target data volume and the synchronization method of each data, send the target operation statement to the corresponding second storage node.

[0059] In this embodiment, the default data synchronization mode of the second storage node can be real-time synchronization. Under different data synchronization modes, different preprocessing methods can be used to preprocess the target operation statements according to the different target data volumes, so as to improve the implementation efficiency of the data synchronization method and the sending efficiency of the target operation statements to the corresponding second storage node.

[0060] Based on the foregoing embodiments, in other embodiments of this application, step 203d can be implemented by steps a11 to a13:

[0061] Step a11: If each data synchronization method is real-time synchronization and the target data volume is less than the preset threshold, determine the table creation statement corresponding to the target data table.

[0062] In this embodiment, the preset threshold can be an empirical value obtained from numerous experiments, or an empirical value set by the user according to actual needs. During use, the preset threshold can also be dynamically adjusted based on the response time of the target operation statement. The preset threshold does not affect the performance of the data synchronization method.

[0063] Step a12: Generate a second request message based on the table creation statement and the target operation statement.

[0064] Step a13: Send the second request message to the corresponding second storage node via real-time synchronization.

[0065] The second request message includes a target operation statement. The table creation statement included in the second request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the second request message.

[0066] Thus, when the data synchronization mode corresponding to the second storage node is real-time synchronization, if the target data volume of the target operation statement is less than the preset threshold, the target operation statement and the table creation statement are directly generated into a second request message.

[0067] Based on the foregoing embodiments, in other embodiments of this application, the step "determine the table creation statement corresponding to the target data table" in step 203a or step a11 can be implemented through steps b11 to b17:

[0068] Step b11: Determine the call information corresponding to the target operation statement.

[0069] In this embodiment of the application, the calling information is the data information of the data service layer responding to the target operation statement.

[0070] Step b12: From the call information, determine the reference field information used to define the structure of the target data table.

[0071] In this embodiment of the application, the corresponding node configuration information is determined from the call information, and the reference field information corresponding to the structure of the target data table can be parsed from the node configuration information.

[0072] Step b13: Replace the reference character in the reference field information with the target character to obtain the target field information.

[0073] In this embodiment of the application, the target characters can be, for example, "*%*" and the blank character "" in an SQL statement, and the corresponding reference characters are "?" and "[\\s]+", so that the target field information is obtained after the replacement process.

[0074] After the data synchronization device executes step b13, it can choose to execute steps b14 to b16 or step b17. If the target field information includes table field entity information, steps b14 to b16 are selected; if the target field information does not include table field entity information, step b17 is selected.

[0075] Step b14: If the target field information includes table field entity information, iterate through the target field information to obtain the target field name.

[0076] Step b15: Obtain the reference field type corresponding to the target field name.

[0077] In this embodiment of the application, the corresponding reference field type is obtained from the table field entity information based on the target field name.

[0078] Step b16: Process the reference field types using the target statement rules to obtain the table creation statement.

[0079] In this embodiment of the application, the target statement rule is the SQL statement format requirement. For example, if the field is a string type, add single quotes (") to the field; if the field is a date type, convert the field to the date format stored in the target data table and add single quotes ("); if the field is a large number with more than 16 significant digits, convert the field to a string type using the large number's built-in string conversion method and add single quotes ("). In this way, the table creation statement can be obtained.

[0080] Step b17: If the target field information does not include table field entity information, retrieve the statement corresponding to the target operation statement from the target field storage area to obtain the table creation statement.

[0081] In this embodiment of the application, the target field storage area may be, for example, a JavaBean.

[0082] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 3As shown, after the first storage node executes step 204, it is also used to execute step 205:

[0083] Step 205: If each data synchronization method is real-time synchronization and the target data volume is greater than or equal to the preset threshold, update the corresponding data synchronization method to asynchronous synchronization.

[0084] In this embodiment of the application, when the data synchronization method of the second storage node is real-time synchronization, if the target data volume is greater than or equal to a preset threshold, directly sending the target operation statement to the second storage node through real-time synchronization will seriously affect the operating performance of the second storage node. Therefore, the data synchronization method of the second storage node is adjusted and updated to asynchronous synchronization.

[0085] Based on the foregoing embodiments, in other embodiments of this application, step 203 or step 203d can also be implemented by steps c11 to c16:

[0086] Step c11: If each data synchronization method is asynchronous synchronization and the target data volume is greater than or equal to a preset threshold, generate a target operation file based on the target operation statement.

[0087] In this embodiment, when the data synchronization method of the second storage node is asynchronous, if the target data volume is greater than or equal to a preset threshold, a file is created, and the target operation statement is placed in the created file to obtain the target operation file. This file can be named using the identification information of the target data table. Furthermore, since the update operation on the target data table may be continuous, in order to distinguish the target operation statements corresponding to different time periods, in addition to naming the file using the identification information of the target data table, it can also include the current time and a randomly generated number. That is, the file storing the target operation statement is named using the identification information of the target data table, the current time, and a randomly generated number to obtain the target operation file.

[0088] If each data synchronization method is asynchronous, and the target data volume is less than a preset threshold, the asynchronous synchronization method is used to directly send the target operation statement to the corresponding second storage node. For example, if the data synchronization method of the second storage node is asynchronous, and the target data volume of the target operation statement is less than the preset threshold, the first storage node sends the target operation statement to the second storage node every preset asynchronous time period, so that the second storage node executes the target operation statement, achieving synchronous update operations between the target data table and the target data table in the first storage node.

[0089] Step c12: Send the target operation file to the file server.

[0090] In this embodiment of the application, the file server is a server used for storing files between a first storage node and at least one second storage node.

[0091] Step c13: Receive the indication information sent by the file server, which indicates the storage location of the target operation file.

[0092] In this embodiment, the indication information may be the identity document (ID) and corresponding hash value of the storage file used to identify the target operation file, generated by the file server after receiving the target operation file. Correspondingly, the generated ID and corresponding hash of the storage file used to identify the target operation file may be determined by the upload frame rate (Frames Per Second, FPS).

[0093] Step c14: Store the instruction information.

[0094] Step c15: Generate an acquisition command according to the preset asynchronous time period.

[0095] The retrieval instruction is used to instruct the retrieval of the target operation file from the instruction information on the file server, obtain the target operation statement, and execute the corresponding update operation.

[0096] In this embodiment, the preset asynchronous time period can be an empirical value set according to actual conditions, or it can be an empirical value obtained from a large number of experiments. For example, the preset asynchronous time period could be every 5 minutes, every 30 minutes, or at 6:00 AM every day.

[0097] Step c16: Send the retrieval command to the corresponding second storage node.

[0098] In this embodiment, the first storage node generates an acquisition instruction every preset asynchronous time period and immediately sends the acquisition instruction to the corresponding second storage node, so that the second storage node can obtain the target operation file at the storage location indicated by the instruction information from the file server, so that the second storage node can execute the target operation statement corresponding to the target operation file.

[0099] In some application scenarios, after receiving the instruction information, the first storage node generates a retrieval command and sends the retrieval command to the corresponding second storage node only at preset asynchronous time intervals, thereby realizing an asynchronous synchronization method between the first storage node and at least one second storage node for the target data table.

[0100] It should be noted that steps c11 to c16 can also be executed after step 205, that is, after the data synchronization mode corresponding to the second storage node is updated from real-time synchronization mode to asynchronous synchronization mode, steps c11 to c16 can be selected to be executed.

[0101] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 4 As shown, after the first storage node executes step 203, it is also used to execute steps 206 to 207:

[0102] Step 206: Receive the update result sent by the corresponding second storage node.

[0103] The update result is the result of the second storage node executing the target operation statement.

[0104] In this embodiment of the application, the update result may be sent from the second storage node to the first storage node in the form of a response message.

[0105] When the data synchronization method is real-time synchronization, after receiving the second request message, the second storage node parses the message to obtain the target operation statement and the table creation statement. Based on the target operation statement, it determines whether the second storage node possesses the corresponding target data table. If the target data table does not exist in the second storage node, it executes the parsed table creation statement to create the corresponding target data table, and then executes the target operation statement on the target data table to obtain the operation result, i.e., the update result. If the target data table exists in the second storage node, it determines whether the field information in the existing target data table is consistent with the field information in the table creation statement. The field information includes at least the field name, field type, and field length. The second storage node caches the target data table and executes the target operation statement to update the content of the target data table, obtaining the update result. The purpose of caching the target data table before executing the target operation statement is to allow for rollback operations in case other second storage nodes encounter execution errors in the target operation statement. If the field information in the target data table is inconsistent with the field information in the table creation statement, the second storage node retrieves the table creation statement of the target data table from its own storage. The second storage node then executes the table creation statement and target operation statement based on the target data table in its storage, but the execution fails, resulting in an update result.

[0106] When the data synchronization method is asynchronous, if the second storage node receives the target operation statement in each preset asynchronous time period, the execution process of the target operation statement by the second storage node is the same as that in the real-time synchronization method. No specific limitation is made here.

[0107] When the data synchronization method is asynchronous, if the second storage node receives the target operation file every preset asynchronous time period, the second storage node parses the target operation file and retrieves a preset number of operation statements from the target operation file in the order of the target operation statements. The execution process is the same as that of the second storage node in the real-time synchronization method. No specific limitation is made here.

[0108] Step 207: Based on the update results, update the data synchronization method of the corresponding second storage node.

[0109] In the embodiments of this application, under different data synchronization methods, different update results correspond to different update methods of the data synchronization method of the second storage node.

[0110] Based on the foregoing embodiments, in other embodiments of this application, step 207 can be implemented by step 207a or step 207b:

[0111] Step 207a: If the data synchronization method is real-time synchronization and the update result indicates that the second storage node is executing the target operation statement abnormally, change the data synchronization method to asynchronous synchronization.

[0112] In this embodiment of the application, when the data synchronization method is real-time synchronization, if the update result indicates that the second storage node has successfully executed the target operation statement, there is no need to update the data synchronization method of the corresponding second storage node, and the data synchronization method of the corresponding second storage node will continue to be real-time synchronization.

[0113] Step 207b: If the data synchronization method is asynchronous, and the update result indicates that the second storage node has successfully executed the target operation statement, then the data synchronization method is changed to real-time synchronization.

[0114] In this embodiment of the application, when the data synchronization method is asynchronous synchronization, if the update result indicates that the second storage node failed to execute the target operation statement, the data synchronization method is not updated, and the corresponding second storage node continues to maintain the data synchronization method as asynchronous synchronization.

[0115] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 5As shown, after the first storage node executes step 206, it is also used to execute steps 208 to 209:

[0116] Step 208: If the update result indicates that the second storage node encountered an error while executing the target operation statement, determine the error priority corresponding to the update result.

[0117] In this embodiment, the synchronization process ends when the update result indicates that the second storage node successfully executed the target operation statement. If the update result indicates that the second storage node encountered an error while executing the target operation statement, the error priority corresponding to the update result is determined based on the update result content.

[0118] Step 209: Execute the alarm operation corresponding to the exception priority.

[0119] In this embodiment, the anomaly priority is assumed to include levels from highest to lowest as: Level 1, Level 2, ..., n. Assuming a total of three levels of anomaly priority, Level 1 represents the highest priority, and the corresponding alarm operation could be, for example, establishing a voice communication connection with the relevant person in charge's communication device to remind them to handle the anomaly promptly. Level 2 represents the next highest priority, and the corresponding alarm operation could be, for example, establishing a communication connection with the relevant person in charge's communication device to remind them to handle the anomaly when they have time by sending a notification message. Level 3 represents the lowest priority, and the corresponding alarm operation could be, for example, generating a notification message and recording it for the relevant person in charge to review.

[0120] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 6 As shown, after the first storage node executes step 206, it is also used to execute steps 210 to 211:

[0121] Step 210: If the update result indicates that the second storage node failed to execute the target operation statement, determine the second storage node whose update result indicates that the second storage node successfully executed the target operation statement from at least one second storage node, and obtain at least one third storage node.

[0122] In this embodiment of the application, if the update result indicates that the second storage node failed to execute the target operation statement, it means that the synchronization operation on the second storage node was not successfully executed. Therefore, the synchronization of the target data table was unsuccessful. At this time, from at least one second storage node that is associated with the target data table, the second storage node whose update result indicates that the target operation statement was successfully executed is determined, and at least one third storage node is obtained.

[0123] Step 211: Send a data rollback control command to at least one third storage node.

[0124] Among them, the data rollback control instruction is used to instruct the third storage node to roll back to the data before the execution of the target operation statement.

[0125] In this embodiment, the first storage node sends a data rollback instruction to at least one third storage node, so that at least one third storage node responds to the data rollback instruction and replaces the target data table after the execution of the target operation statement with the target data table cached before the execution of the target operation statement, thereby realizing the data rollback operation.

[0126] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 7 As shown, after the first storage node executes step 206, it is also used to execute steps 212 to 213:

[0127] Step 212: If the update result sent by each second storage node indicates that the second storage node has successfully executed the target operation statement, calculate the data processing time consumed by each second storage node to obtain at least one data processing time consumed.

[0128] Step 213: Update the preset threshold based on the time consumed by at least one data processing step.

[0129] In this embodiment, the first storage node analyzes the processing time of at least one data item to determine whether the current preset threshold is reasonable. If it is unreasonable, it analyzes the processing time according to an instruction to determine a new threshold to update the preset threshold. If it is reasonable, there is no need to update the preset threshold. In some application scenarios, after obtaining at least one data processing time, it can also be sent to the relevant person in charge so that they can analyze the at least one data processing time and reset the preset threshold.

[0130] Based on the foregoing embodiments, in other embodiments of this application, step 201 can be implemented by steps 201a to 201d:

[0131] Step 201a: Obtain the mapper declaration parameters and parameter type parameters in the data service layer.

[0132] Step 201b: Process the mapper declaration parameters and parameter type parameters using the target statement rules to obtain the reference operation statement.

[0133] In this embodiment of the application, the target statement rules, namely the SQL statement rules, are used to parse the mapper declaration parameters and parameter type parameters, and the parsed content is concatenated to obtain the SQL statement.

[0134] Step 201c: If the operation type of the reference operation statement is the target operation type, determine the target operation statement as the reference operation statement.

[0135] In this embodiment of the application, the reference operation statement is analyzed. If the reference operation statement is determined to be the target operation type, that is, Insert, Delete or Update, the reference operation statement is determined to be the target operation statement.

[0136] Step 201d: Identify at least one second storage node that is associated with the target data table.

[0137] Based on the foregoing embodiments, this application provides a schematic diagram of the structure of a data synchronization system, referring to... Figure 8 As shown, it includes: a primary operation database system, target backup database system 1, target backup database system 2, and target backup database system 3. Each of these systems includes: a business logic application layer, a data service layer, an interceptor, and a primary operation database (DB). The interceptor includes: a data synchronization module, a synchronization result processing module, a data distribution anomaly monitoring and feedback processing module, and a threshold setting module. It should be noted that... Figure 8 The diagram only fully illustrates the structure of the main operating system library. The target backup library system 1, target backup library system 2, and target backup library system 3 have the same structure as the main operating system library. The main operating system library system corresponds to the aforementioned first storage node, and the target backup library system 1, target backup library system 2, and target backup library system 3 correspond to at least one of the aforementioned second storage nodes. The roles and functions of the first storage node and the second storage node can be interchanged.

[0138] based on Figure 8 The diagram shows the structure of a data synchronization system. This application embodiment provides a data synchronization method, the specific implementation process of which is shown below:

[0139] After receiving an operation request from a user requesting an update operation on a target data table in the main operation database, the business logic layer of the main operation database system forwards the request to the data service layer. The corresponding data operation statements in the data service layer respond to the operation request, implementing the operation on the target data table. An interceptor in the main operation database system intercepts the corresponding data operation statements in the data service layer, analyzes the intercepted statements, and determines the target operation statement for the target data table, belonging to the target operation type such as Insert, Delete, or Update. After determining the target operation statement, the interceptor in the main operation database system identifies at least one second storage node associated with the target data table as target backup database system 1, target backup database system 2, and target backup database system 3. It then determines the corresponding data synchronization method for each of these three backup database systems, synchronizes the target operation statement using their respective data synchronization methods, and receives the update results from each backup database system. The master operation database system can also update the data synchronization methods of target backup database 1, target backup database 2 and target backup database 3 based on the update results sent by target backup database 1, target backup database 2 and target backup database 3, and adjust the preset threshold, thus realizing a complete closed-loop feedback system and ensuring an effective synchronization strategy.

[0140] Identifying at least one second storage node associated with the target data table can be derived from the relationships between pre-stored data tables and storage nodes. These relationships can be determined from annotations included in the corresponding framework scripts. For example, they can be determined from annotations included in Spring's Aspect-Oriented Programming (AOP) framework, which identify the corresponding data table to be distributed and specify the target backup database system. The relationships between pre-stored data tables and storage nodes can be represented as a list, as shown in Table 1. Here, DcnRouter represents the identifier of the target data table, SysName represents the system name of the system containing the target data table, DcnNo represents the system ID of the system containing the target data table, and ServiceId represents the mapping relationship between systems containing the target data table when sending information.

[0141] Interceptors in the primary operation database system can be plugins provided by persistence frameworks such as MyBatis. Remote calls can be used between the primary operation database system and the target backup database system.

[0142] The process by which an interceptor obtains the target operation statement can be as follows: The interceptor retrieves the parameters from the data operation statement in the data service layer's response to the operation request, and parses the command type, such as SqlCommandTypeSql, included in the mapper declaration from the parameters. Based on the command type, such as SqlCommandTypeSql, the interceptor determines the target operation statement of type Insert, Delete, and Update, etc.

[0143] Table 1

[0144]

[0145] Determining at least one second storage node that is associated with the target data table can also be achieved by an interceptor obtaining the class name based on the intercepted mapping statement identifier, such as MappedStatementId, mapping the class object information based on the class name, and determining whether the class has added annotations and annotation parameters based on the class object information. If annotations and annotation parameters are added, at least one second storage node that is associated with the target data table can be determined.

[0146] The interceptor determines the target operation statement (Insert, Delete, Update, etc.) based on the command type, such as SqlCommandTypeSql. The specific process is as follows: The interceptor obtains the boundSQL from the MappedStatement and parameter type (ParamObject) in the data service layer. It parses the MappedStatement and ParamObject parameters and concatenates them into an executable SQL statement of the target operation type, thus obtaining the target operation statement. Further, by dividing the amount of data in the boundSQL's parameter list by the number of fields in the table structure, the amount of target data to be executed in the SQL statement is obtained.

[0147] The interceptor parses the node configuration information in the data operation statement of the data service layer's response operation request. Since the node configuration information includes information about each field defining the target data table structure, it can iterate through each field in the node configuration information and concatenate the table creation statement for the target data table. The specific implementation process can be as follows: replace all placeholders "?" with "*%*", and replace all whitespace characters "[\\s]+" with "", to obtain the replaced node configuration information; determine whether the replaced node configuration information uses table field entity classes as parameters; if the replaced node configuration information uses table field entity classes as parameters, iterate through the parameter list to determine the set of parameters passed by the user and the set of parameter mapping relationships; obtain the field names and field types from the set of parameters passed by the user and the set of parameter mapping relationships; and then, based on the obtained field names, retrieve the corresponding... The process of retrieving field types from a collection is as follows: If the field is a string, add single quotes ("); if the field is a date, convert it to the date format stored in the target data table and add single quotes ("); if the field is a large number with more than 16 significant digits, convert it to a string using the built-in large number conversion method and add single quotes ("). This will yield the table creation statement. If the replaced node configuration information does not use the table field entity class as a parameter, retrieve the batch parameters of the entity class from the JavaBean to obtain the table creation statement. The table creation statement is also an SQL statement.

[0148] The interceptor's data synchronization module determines the data synchronization method for each target backup database system. The initial data synchronization method for each target backup database system defaults to real-time synchronization. The interceptor's data synchronization module verifies whether the target data volume reaches a preset threshold. When the data synchronization method is real-time, if the target data volume is less than the preset threshold, the data synchronization method remains real-time; if the target data volume is greater than or equal to the preset threshold, the data synchronization method is updated from real-time to asynchronous synchronization. When the data synchronization method is asynchronous, if the target data volume is less than the preset threshold, the specific data synchronization method is determined to be asynchronous data synchronization; if the target data volume is greater than or equal to the preset threshold, the specific data synchronization method is determined to be asynchronous file synchronization. The data synchronization method can be implemented using two parameters: table name and data synchronization method. Furthermore, during the asynchronous synchronization process, the specific asynchronous synchronization method can be implemented using the format shown in Table 2. In Table 2, the `table_name` parameter represents the name of the target data table; the `dcn_no` parameter represents the ID of the target backup database system receiving the target operation data; the `sql` parameter records the SQL script to be executed for synchronization when `process_type` is `SQL`, otherwise it is empty; the `id_hash` parameter records the ID and hash value of the target operation file obtained when the target operation file is uploaded to the file server when `process_type` is `FILE`; `process_type` identifies the specific execution type of the asynchronous synchronization method, using `FILE` for asynchronous file synchronization and `SQL` for asynchronous data synchronization; `create_time` represents the current time when the asynchronous synchronization method was generated, and this event can be used as the logical order for subsequent synchronization; `sys_seq` represents the current sequence number corresponding to the asynchronous synchronization method, which is used by operations personnel to find the call chain of the entire process after an anomaly is detected, thereby quickly locating the problem, executing the handling plan, and assessing the scope of impact.

[0149] Table 2

[0150] table_name SQL id_hash dcn_no process_type create_time sys_seq

[0151] For example, if the interceptor's data synchronization module determines that the data synchronization method of the target backup database system 1 is real-time synchronization, the interceptor will fill the determined target operation statement and table creation statement into the real-time request message, and publish the real-time request message to the real-time message queue corresponding to the system number of the target backup database system 1, and send the real-time request message to the target backup database system 1 in the form of a queue message. After receiving the real-time request message, the target backup database system 1 will perform the following judgments on the real-time request message: determine whether there is a target data table with a table structure corresponding to the table creation statement in the target backup database system 1; if there is no matching target data table in the target backup database system 1, execute the table creation statement in the real-time request message to create the target data table, and then execute the target operation statement in the real-time request message to obtain the update result of the executed target operation statement, and fill the update result into the response message and send it to the main operation database system; if there is a matching target data table in the target backup database system 1, further check whether the field name, field type, and field length in the table creation statement are the same as the field name, field type, and field length of the target data table in the target backup database system 1. If the segment type and field length are consistent, and the field name, field type, and field length are consistent, the target data table in the target backup database system 1 is first cached, then the target operation statement is executed to obtain the update result, and the update result is filled into the response message and sent to the main operation database system. If at least one of the field name, field type, and field length is inconsistent, the table structure information of the target data table in the target backup database system 1 is obtained through a specific statement such as dataSource, the table creation statement of the target data table in the target backup database system 1 is obtained, the execution failure, the table creation statement of the target data table in the target backup database system 1, and the specific reason for the failure are filled into the response message and sent to the main operation database system.

[0152] After receiving the response message from the target backup system 1, the synchronization result processing module of the master operation database system interceptor will, if the response message indicates that the target backup system 1 has successfully executed the synchronization (i.e., the aforementioned execution of the target operation statement was successful), complete the data synchronization operation for the target backup system 1. If the response message indicates that the target backup system 1 has failed to execute the synchronization (i.e., the aforementioned execution of the target operand statement was abnormal), firstly, the data synchronization mode corresponding to the target backup system will be updated to asynchronous synchronization mode; then, the backup system subsystem that has already been successfully executed will be found through the system number to which the current update result belongs; finally, the target operation statement of this synchronization and the system number corresponding to the target backup system 1 will be recorded according to Table 2, and the process_type field in Table 2 will be updated to SQL. Meanwhile, when the response message indicates that the target standby system 1 failed to synchronize, the exception information also needs to be persisted. The information structure of the persisted exception information can be shown in Table 3. Some of the contents in Table 3 are the same as those in Table 2, and will not be described in detail here. In Table 3, route_sys_name is used to identify the name of the standby system to which the execution exception occurred, which is convenient for operation and maintenance personnel to locate the system log; remark is used to indicate the remark, which is mainly used to record the reason for the execution failure as a key log for alarms; urgent_type is used to identify the urgency type, where H indicates high priority, L indicates low priority, and D indicates that the processing has been completed.

[0153] Table 3

[0154] table_name SQL id_hash dcn_no route_sys_name remark create_time urgent_type

[0155] If the synchronization failure is due to inconsistency in at least one of the field names, field types, or field lengths, the interceptor's data transmission anomaly monitoring and feedback processing module determines the `urgent_type` to be a high-priority (H) anomaly record. For high-priority `urgent_type` records, voice communication can be triggered, for example, to remind the relevant person in charge, so that they can confirm whether the inconsistencies between upstream and downstream fields can be adjusted to be consistent. After the consistency is adjusted, the `urgent_type` can be changed to `D` (processed). If the failure is caused during the synchronization process, the interceptor's data transmission anomaly monitoring and feedback processing module determines the `urgent_type` to be a low-priority (L) anomaly record. For low-priority `urgent_type` records, prompt messages, such as text messages or voice messages, can be sent to remind the relevant person in charge, so that they can allocate time to handle the issue. After the relevant person in charge handles the issue, the `urgent_type` can be changed to `D`.

[0156] If target backup system 1 fails to synchronize, but target backup systems 2 and 3 have successfully synchronized, the interceptor's data delivery anomaly monitoring and feedback processing module still needs to perform data rollback operations on target backup systems 2 and 3 because target backup system 1 failed to synchronize. The main operation database system determines the system numbers of target backup systems 2 and 3, assembles the SQL regression statements to be executed into rollback notification messages, and publishes them to the data rollback queue for distribution to the corresponding target backup systems 2 and 3. Upon receiving the rollback notification messages, target backup systems 2 and 3 perform the corresponding data rollbacks: for example, if the target operation statement is of type INSERT, the successfully added data is deleted; if the target operation statement is of type UPDATE, the backup data before the update is retrieved using the update conditions, and the data is restored; if the target operation statement is of type DELETE, the backup data before the deletion is retrieved using the conditions, and the data is restored.

[0157] When the interceptor's data synchronization module determines that the target backup database system 1's data synchronization mode is asynchronous data synchronization, the main operation database system interceptor's data synchronization module fills the target operation statement into the asynchronous request message and publishes the asynchronous request message to the asynchronous message queue corresponding to the system number of the target backup database system 1. The target operation data is sent to the target backup database system 1 according to a preset asynchronous time period, for example, every 30 minutes. The implementation process of the target backup database system 1 after receiving the asynchronous request message is the same as the processing steps of the target backup database system 1 after receiving the real-time request message, and will not be described in detail here.

[0158] When the interceptor's data synchronization module determines that the data synchronization method of the target backup system 1 is asynchronous file synchronization, the main operation database system interceptor's data synchronization module first creates a file on the local system, naming the file with the target data table name, the current date, and a randomly generated number; it then writes the target operation statement into the created file; and finally uploads the file containing the target operation statement to the file server. The file server returns a unique identifier for the storage location of the file containing the target operation statement, specifically the id_hash field in Tables 2 and 3. The interceptor's data synchronization module then fills the returned unique identifier into the asynchronous request message and publishes the asynchronous request message to the asynchronous message queue corresponding to the system number of the target backup system 1. The target operation data is then sent to the target backup system 1 according to a preset asynchronous time period, for example, every 30 minutes. The implementation process of the target backup system 1 after receiving the asynchronous request message is the same as the processing steps of the target backup system 1 when receiving a real-time request message, and will not be elaborated here. The specific implementation process of writing the target operation statement into the created file can be either to write all the statements in the target operation statement into the created file, or to group the operation statements in the target operation statement according to a certain number and generate at least one file to achieve asynchronous synchronization.

[0159] The processing of asynchronous results in asynchronous synchronous mode is basically the same as that in synchronous mode. The difference is that after receiving the data synchronization success return message, the data synchronization mode is updated to real-time synchronization update.

[0160] The threshold setting module of the main operation library system interceptor is based on Spring's AOP annotation @CollectionReccateTime() to collect the time elapsed from detecting a user's operation request to detecting successful synchronization, as well as the target data volume included in this synchronization. Data collection and observation are performed on these two values ​​for three types of data operations, resulting in the data elapsed time records shown in Table 4. This allows relevant personnel to observe the statistical results shown in Table 4 to adjust the preset threshold based on the different data volume evoked by different operation types. The adjusted preset threshold is then sent to the main operation library system through dynamic configuration items to enable the subsequent synchronization process.

[0161] Table 4

[0162] op_type data_count timeout

[0163] Here, `op_type` identifies the operation type, `data_count` identifies the data volume for the corresponding operation type, and `timeout` identifies the time consumed for that operation type. The time consumed for an operation type = request parsing time + ∑(data retrieval time + data processing time) + time assembling the returned result, where ∑(data retrieval time + data processing time) represents the time consumed for the data volume and operation statements corresponding to that operation type.

[0164] In the data distribution anomaly monitoring and feedback processing module of the main operation library system, the anomaly alarm process can be implemented based on a text messaging platform and a virtual telephone outbound calling platform, as referred to... Figure 9 As shown, for each new high-priority emergency message, an immediate telephone alert can be triggered to notify the relevant person in charge. If the person in charge processes the high-priority emergency message, the emergency type in the emergency message is changed to "processed". If the person in charge does not process the high-priority emergency message, the emergency type in the emergency message remains high-priority, and the triggering logic is placed in a delay queue. After a certain delay, such as 5 minutes, the virtual telephone outbound call platform is triggered again to remind the relevant person in charge via telephone, until the emergency type in the emergency message record changes to "processed". For records with low-priority emergency types, an emergency message scan operation can be triggered after data synchronization is completed. That is, a timed polling method is used to scan the emergency message records for records with low-priority emergency types. The scanned records with low-priority emergency types are converted into request message format and sent to the text message platform to achieve text message alert, until the emergency type in the emergency message record changes to "processed". The text message platform can be, for example, an SMS platform.

[0165] The descriptions of the same steps and contents as in other embodiments in this example can be found in the descriptions of other embodiments, and will not be repeated here.

[0166] In this embodiment, if a target operation statement of the target operation type is detected for the target data table, the first storage node determines at least one second storage node that is associated with the target data table, determines the data synchronization method for each second storage node, and sends the target operation statement to the corresponding second storage node based on each data synchronization method, so that the second storage node executes the target operation statement and realizes the synchronous update operation of the first storage node for the target data table. This solves the problem of data synchronization efficiency in the current distributed storage system, effectively ensures the consistency of data in the distributed storage system, and improves the data synchronization efficiency in the distributed storage system.

[0167] Based on the foregoing embodiments, embodiments of this application provide a data synchronization system that can be applied to... Figures 1-7 In the data synchronization method provided in the corresponding embodiment, refer to Figure 10 As shown, the data synchronization system 3 may include: a first storage node 31 and at least one second storage node 32, wherein:

[0168] The first storage node 31 is used to store the target data table and execute a data synchronization program to achieve the following steps: if a target operation statement of the target operation type is detected, at least one second storage node with an association with the target data table is determined; wherein the target operation statement is used to change the content of the target data table; the data synchronization method of each second storage node is determined; based on each data synchronization method, the target operation statement is sent to the corresponding second storage node to achieve synchronous updates of the target data table in the first storage node and the second storage node;

[0169] The second storage node 32 is used to receive the target operation statement; using the target operation statement, an update operation is performed on the target data table in the second storage node corresponding to the target operation statement, and the update result is obtained.

[0170] In other embodiments of this application, when the first storage node 31 executes the step of sending the target operation statement to the corresponding second storage node based on each data synchronization method, it can be achieved through the following steps:

[0171] If each data synchronization method is real-time synchronization, determine the table creation statement corresponding to the target data table;

[0172] Based on the table creation statement and the target operation statement, generate the first request message;

[0173] The first request message is sent to the corresponding second storage node in real time synchronization. The first request message includes a target operation statement. The table creation statement included in the first request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the first request message.

[0174] In other embodiments of this application, before the first storage node 31 executes the step of sending the target operation statement to the corresponding second storage node based on each data synchronization method, it is also used to execute the following steps:

[0175] Determine the target data volume corresponding to the target operation statement;

[0176] Correspondingly, based on each data synchronization method, the target operation statement is sent to the corresponding second storage node, including:

[0177] Based on the target data volume and the synchronization method of each data, the target operation statement is sent to the corresponding second storage node.

[0178] In other embodiments of this application, when the first storage node 31 executes the step of sending the target operation statement to the corresponding second storage node based on the target data volume and each data synchronization method, it can be achieved through the following steps:

[0179] If each data synchronization method is real-time synchronization and the target data volume is less than the preset threshold, determine the table creation statement corresponding to the target data table.

[0180] Based on the table creation statement and the target operation statement, generate a second request message;

[0181] The second request message is sent to the corresponding second storage node in real time synchronization; wherein the second request message includes the target operation statement, and the table creation statement included in the second request message is used to generate the target data table when the target data table is detected to not exist after the second storage node receives the second request message.

[0182] In other embodiments of this application, when the first storage node 31 executes the step of determining the table creation statement corresponding to the target data table, it can be achieved through the following steps:

[0183] Determine the invocation information corresponding to the target operation statement;

[0184] From the call information, determine the reference field information used to define the structure of the target data table;

[0185] Replace the reference character in the reference field information with the target character to obtain the target field information;

[0186] If the target field information includes table field entity information, iterate through the target field information to obtain the target field name;

[0187] Retrieve the reference field type corresponding to the target field name;

[0188] The reference field types are processed using the target statement rules to obtain the table creation statement;

[0189] If the target field information does not include table field entity information, retrieve the statement corresponding to the target operation statement from the target field storage area to obtain the table creation statement.

[0190] In other embodiments of this application, the first storage node 31 is further configured to perform the following steps:

[0191] If each data synchronization method is real-time synchronization and the target data volume is greater than or equal to a preset threshold, the corresponding data synchronization method is updated to asynchronous synchronization.

[0192] In other embodiments of this application, the first storage node 31 is further configured to perform the following steps:

[0193] If each data synchronization method is asynchronous synchronization and the target data volume is greater than or equal to a preset threshold, a target operation file is generated based on the target operation statement;

[0194] Send the target file to the file server;

[0195] Receive indication information sent by the file server, indicating the storage location of the target file;

[0196] Store instruction information;

[0197] According to a preset asynchronous time period, a retrieval instruction is generated; wherein, the retrieval instruction is used to instruct the retrieval of the target operation file from the instruction information of the file server, and to obtain the target operation statement to execute the corresponding update operation;

[0198] Send the retrieval command to the corresponding second storage node.

[0199] In other embodiments of this application, after the first storage node 31 sends the target operation statement to the corresponding second storage node based on each data synchronization method, it is also used to perform the following steps:

[0200] Receive the update result sent by the corresponding second storage node; wherein, the update result is the operation result of the second storage node executing the target operation statement;

[0201] Based on the update results, update the data synchronization method of the corresponding second storage node.

[0202] In other embodiments of this application, when the first storage node 31 performs the step of updating the data synchronization method of the corresponding second storage node based on the update result, it can be achieved through the following steps:

[0203] If the update result indicates that the second storage node failed to execute the target operation statement when the data synchronization method is real-time, the data synchronization method should be changed to asynchronous synchronization.

[0204] If the data synchronization method is asynchronous, and the update result indicates that the second storage node has successfully executed the target operation statement, then the data synchronization method is real-time.

[0205] In other embodiments of this application, after the first storage node 31 receives the update result sent by the corresponding second storage node, it is further configured to perform the following steps:

[0206] If the update result indicates that the second storage node encountered an error while executing the target operation statement, determine the exception priority corresponding to the update result;

[0207] Execute the alarm operation corresponding to the exception priority.

[0208] In other embodiments of this application, after the first storage node 31 receives the update result sent by the corresponding second storage node, it is further configured to perform the following steps:

[0209] If the update result indicates that the second storage node failed to execute the target operation statement, determine the second storage node whose update result indicates that the second storage node successfully executed the target operation statement from at least one second storage node, and obtain at least one third storage node;

[0210] Send a data rollback control instruction to at least one third storage node; wherein the data rollback control instruction is used to instruct the third storage node to roll back the data before the execution of the target operation statement.

[0211] In other embodiments of this application, the first storage node 31 is further configured to perform the following steps:

[0212] If the update result sent by each second storage node indicates that the second storage node has successfully executed the target operation statement, the data processing time of each second storage node is calculated to obtain at least one data processing time.

[0213] Update the preset threshold based on at least one data processing time.

[0214] In other embodiments of this application, if the first storage node 31 detects a target operation statement of the target operation type for the target data table and determines at least one second storage node associated with the target data table, the following steps can be used:

[0215] Retrieve the mapper declaration parameters and parameter type parameters in the data service layer;

[0216] The mapper declaration parameters and parameter type parameters are processed using target statement rules to obtain reference operation statements;

[0217] If the operation type of the reference operation statement is the same as the target operation type, then the target operation statement is determined to be the reference operation statement.

[0218] Identify at least one second storage node that is associated with the target data table.

[0219] It should be noted that, in the embodiments of this application, when one or more programs are executed by the processor, they can achieve... Figures 1-7 The methods provided in the corresponding embodiments will not be described in detail here.

[0220] In this embodiment, if a target operation statement of the target operation type is detected for the target data table, the first storage node determines at least one second storage node that is associated with the target data table, determines the data synchronization method for each second storage node, and sends the target operation statement to the corresponding second storage node based on each data synchronization method, so that the second storage node executes the target operation statement and realizes the synchronous update operation of the first storage node for the target data table. This solves the problem of data synchronization efficiency in the current distributed storage system, effectively ensures the consistency of data in the distributed storage system, and improves the data synchronization efficiency in the distributed storage system.

[0221] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium, simply referred to as a storage medium, which stores one or more programs that can be executed by one or more processors to achieve, as follows: Figures 1-7 The implementation process of the data synchronization method provided in the corresponding embodiment will not be described in detail here.

[0222] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0223] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable first storage node to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable first storage node, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0224] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable first storage node to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0225] These computer program instructions may also be loaded onto a computer or other programmable first storage node, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0226] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data synchronization method, characterized in that, The method is applied to a first storage node, and the method includes: If a target operation statement of the target operation type is detected targeting a target data table, at least one second storage node associated with the target data table is determined; wherein the target operation statement is used to change the content of the target data table; Determine the data synchronization method for each of the second storage nodes; Determine the target data volume corresponding to the target operation statement; Based on the target data volume and each of the data synchronization methods, the target operation statement is sent to the corresponding second storage node to achieve synchronous updates of the target data table in the first storage node and the second storage node; Wherein, if each of the data synchronization methods is a real-time synchronization method, the table creation statement corresponding to the target data table is determined; based on the table creation statement and the target operation statement, a first request message is generated; the first request message is sent to the corresponding second storage node through the real-time synchronization method; wherein, the first request message includes the target operation statement, and the table creation statement included in the first request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the first request message; Wherein, if each of the data synchronization methods is the real-time synchronization method and the target data volume is greater than or equal to a preset threshold, the corresponding data synchronization method is updated to the asynchronous synchronization method.

2. The method according to claim 1, characterized in that, The step of sending the target operation statement to the corresponding second storage node based on the target data volume and each of the data synchronization methods includes: If each of the data synchronization methods is the real-time synchronization method, and the target data volume is less than a preset threshold, determine the table creation statement corresponding to the target data table; Based on the table creation statement and the target operation statement, a second request message is generated; The second request message is sent to the corresponding second storage node through the real-time synchronization method; wherein, the second request message includes the target operation statement, and the table creation statement included in the second request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the second request message.

3. The method according to claim 1 or 2, characterized in that, The step of determining the table creation statement corresponding to the target data table includes: Determine the invocation information corresponding to the target operation statement; From the call information, determine the reference field information used to define the structure of the target data table; The target field information is obtained by replacing the reference character in the reference field information with the target character. If the target field information includes table field entity information, iterate through the target field information to obtain the target field name; Obtain the reference field type corresponding to the target field name; The reference field type is processed using target statement rules to obtain the table creation statement; If the target field information does not include the table field entity information, the statement corresponding to the target operation statement is obtained from the target field storage area to obtain the table creation statement.

4. The method according to claim 1, characterized in that, The method further includes: If each of the data synchronization methods is an asynchronous synchronization method, and the target data volume is greater than or equal to the preset threshold, a target operation file is generated based on the target operation statement; Send the target operation file to the file server; Receive indication information sent by the file server, indicating the storage location of the target operation file; Store the instruction information; According to a preset asynchronous time period, an acquisition instruction is generated; wherein, the acquisition instruction is used to instruct the acquisition of the target operation file from the instruction information of the file server, and to obtain the target operation statement to execute the corresponding update operation; Send the acquisition instruction to the corresponding second storage node.

5. The method according to claim 1, characterized in that, After sending the target operation statement to the corresponding second storage node based on the target data volume and each of the data synchronization methods, the method further includes: Receive the update result sent by the corresponding second storage node; wherein, the update result is the operation result of the second storage node executing the target operation statement; Based on the update result, the data synchronization method of the corresponding second storage node is updated.

6. The method according to claim 5, characterized in that, The step of updating the data synchronization method of the corresponding second storage node based on the update result includes: If the data synchronization method is real-time synchronization and the update result indicates that the second storage node is executing the target operation statement abnormally, the data synchronization method is updated to asynchronous synchronization. If the data synchronization method is the asynchronous synchronization method, and the update result indicates that the second storage node has successfully executed the target operation statement, then the data synchronization method is updated to the real-time synchronization method.

7. The method according to claim 5, characterized in that, After receiving the update result sent by the corresponding second storage node, the method further includes: If the update result indicates that the second storage node encountered an error while executing the target operation statement, determine the exception priority corresponding to the update result; Execute the alarm operation corresponding to the aforementioned exception priority.

8. The method according to claim 5, characterized in that, After receiving the update result sent by the corresponding second storage node, the method further includes: If the update result indicates that the second storage node failed to execute the target operation statement, determine from at least one of the second storage nodes the second storage node whose update result indicates that the second storage node successfully executed the target operation statement, and obtain at least one third storage node; Send a data rollback control instruction to the at least one third storage node; wherein the data rollback control instruction is used to instruct the third storage node to roll back to the data before the execution of the target operation statement.

9. The method according to claim 5, characterized in that, The method further includes: If the update result sent by each of the second storage nodes indicates that the second storage node has successfully executed the target operation statement, the data processing time of each of the second storage nodes is calculated to obtain at least one data processing time. Update the preset threshold based on the data processing time of at least one of the data processing steps.

10. The method according to claim 1, characterized in that, If a target operation statement of the target operation type is detected targeting the target data table, determining at least one second storage node associated with the target data table includes: Retrieve the mapper declaration parameters and parameter type parameters in the data service layer; The mapper declaration parameters and parameter type parameters are processed using target statement rules to obtain reference operation statements; If the operation type of the reference operation statement is the target operation type, then the target operation statement is determined to be the reference operation statement; Identify at least one second storage node that is associated with the target data table.

11. A data synchronization system, characterized in that, The system includes: a first storage node and at least one second storage node; wherein: The first storage node is configured to, if a target operation statement of the target operation type is detected, determine at least one second storage node associated with the target data table; wherein the target operation statement is used to change the content of the target data table; determine the data synchronization mode of each second storage node; determine the target data volume corresponding to the target operation statement; and send the target operation statement to the corresponding second storage node based on the target data volume and each data synchronization mode; wherein, if each data synchronization mode is a real-time synchronization mode, determine the table creation statement corresponding to the target data table; generate a first request message based on the table creation statement and the target operation statement; and send the first request message to the corresponding second storage node through the real-time synchronization mode; wherein the first request message includes the target operation statement, and the table creation statement included in the first request message is used to generate the target data table when the second storage node detects that the target data table does not exist after receiving the first request message; wherein, if each data synchronization mode is the real-time synchronization mode and the target data volume is greater than or equal to a preset threshold, update the corresponding data synchronization mode to an asynchronous synchronization mode; The second storage node is used to receive the target operation statement; and to perform an update operation on the target data table corresponding to the target operation statement in the second storage node using the target operation statement to obtain an update result.

12. A storage medium, characterized in that, The storage medium stores a data synchronization program, which, when executed by a processor, implements the steps of the data synchronization method as described in any one of claims 1 to 10.

Images