Data synchronization method, device and equipment among multiple server nodes in isolated environment

Abstract

The application belongs to the technical field of data synchronization, and specifically discloses a data synchronization method, device and equipment among multiple server nodes in an isolated environment. According to the data synchronization instruction in the isolated environment, new operation data is merged, and log data that has been synchronized to all other server nodes is deleted; an update operation is performed according to to-be-processed data; and the processed log time is updated to the latest source data time of target record data. In the above manner, in the isolated environment, multiple server nodes do not communicate with each other, and when data of a server node changes, a data update log is generated; when data synchronization is needed, a synchronization operation log is recorded; a log import / export mechanism is used; data update logs of other server nodes are parsed and executed, data synchronization of the other server nodes is realized, and thus data synchronization in the isolated environment can be realized, and the consistency of data of all server nodes is ensured.

Description

Technical Field

[0001] This application belongs to the field of data synchronization technology, and more specifically, relates to a method, apparatus and equipment for data synchronization between multiple server nodes in an isolated environment. Background Technology

[0002] In modern information technology applications, data synchronization between multiple server nodes is a common operation. Currently, the common methods for data synchronization rely on network connectivity or a central scheduling node. However, due to requirements such as network security, data isolation, or business partitioning, it is often necessary to maintain data consistency among multiple server nodes in an isolated environment. Obviously, whether relying on network connectivity or a central scheduling node, both require a continuous network connection between multiple server nodes. In an isolated environment, multiple server nodes do not communicate with each other, and a continuous network connection cannot be guaranteed. Therefore, the above methods cannot achieve data synchronization in an isolated environment. Summary of the Invention

[0003] In view of the shortcomings of the prior art, the purpose of this application is to provide a method, apparatus and device for data synchronization between multiple server nodes in an isolated environment, which aims to solve the problem that the prior art cannot achieve data synchronization in an isolated environment.

[0004] To achieve the above objectives, firstly, this application provides a method for data synchronization between multiple server nodes in an isolated environment, comprising:

[0005] In response to a data synchronization command in an isolated environment, the source synchronization operation log and source data update log of the source server node are imported from the external medium according to the data synchronization command.

[0006] When it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirements, the source synchronization operation log is merged into the local synchronization operation log of the target server node.

[0007] When it is determined that the log identifier of the source data update log meets the data update log merging requirements, the source data update log is merged into the local data update log of the target server node, and the log data in the local data update log that has been synchronized to all other server nodes is deleted.

[0008] The data to be processed is determined based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and a data update operation is performed based on the data to be processed.

[0009] Obtain the target record data identified as the local identifier in the local synchronization operation log after merging the data, and update the processed log time of the target record data to the latest source data time of the target record data.

[0010] In one embodiment, the step of merging the source synchronization operation log into the local synchronization operation log of the target server node when it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirements includes:

[0011] Find the current record data in the source synchronization operation log that has the same source data location identifier as the target data location identifier and the largest latest source log time. The source data location identifier of the current record data is the exported data location identifier.

[0012] A first identifier combination is generated based on the source data location identifier and the target data location identifier of each data entry in the source synchronization operation log.

[0013] If the first identifier combination is in the local synchronization operation log, then a second identifier combination is generated based on the source data location identifier and the target data location identifier of each data entry in the local synchronization operation log, and data in the local synchronization operation log that has the same second identifier combination as the first identifier combination is obtained.

[0014] When the latest source log time of each data entry in the source synchronization operation log is greater than the latest source log time of the data in the local synchronization operation log, the latest source log time of the current data in the local synchronization operation log is updated to the latest source log time of each data entry in the source synchronization operation log, and the processed log time of the current data in the local synchronization operation log is updated to the processed log time of each data entry in the source synchronization operation log.

[0015] After the first loop ends, filter out the data in the source synchronization operation log whose target data location is identified by the exported data location, and modify the target data location identifier of the data in the source synchronization operation log whose target data location is identified by the exported data location to the identifier of the local server node.

[0016] A third identifier combination is generated based on the source data location identifier and the target data identifier of the data whose target data location identifier is the exported data location identifier in the source synchronization operation log;

[0017] If the third identifier combination is in the local synchronization operation log, then retrieve the data in the synchronization operation log that is the same as the second identifier combination and the third identifier combination;

[0018] When the latest source log time of the data whose target data location is identified as the exported data location in the source synchronization operation log is greater than the latest source log time of the data in the synchronization operation log that has the same second identifier combination and the same third identifier combination, the latest source log time of the data in the synchronization operation log that has the same second identifier combination and the same third identifier combination is updated to the latest source log time of the data in the source synchronization operation log whose target data location is identified as the exported data location, until the second loop ends, completing the merging of the synchronization operation logs.

[0019] In one embodiment, the step of merging the source data update log into the local data update log of the target server node and deleting log data that has been synchronized to all other server nodes from the local data update log when it is determined that the log identifier of the source data update log meets the data update log merging requirements includes:

[0020] If the log identifier of the source data update log is not in the local data update log, then the update log merging requirement is met, and the source data update log is merged into the local data update log of the target server node.

[0021] Group the data in the merged local data update log by target field name to count the number of records.

[0022] When the number is greater than 1, a list alarm is generated for the local data update log, and unique data is selected from the record data with a number greater than 1 according to the preset data selection rules;

[0023] The source data location identifier in the updated local data update log is grouped and statistically analyzed, and the local synchronization operation log is grouped and statistically analyzed according to the source data location identifier to obtain the minimum processed log time corresponding to each source data location identifier.

[0024] In the updated local data update log, data with timestamps less than the minimum processed log time corresponding to the source data location identifier of the recorded data are taken as log data that has been synchronized to all other server nodes in the local data update log, and the log data is deleted.

[0025] In one embodiment, the step of determining the data to be processed based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and performing a data update operation based on the data to be processed, includes:

[0026] Based on the source data location identifier, the inserted log is associated with the local synchronization operation log after merging the data, and the first record data in the local synchronization operation log after merging the data with the target data location identifier as the local identifier is obtained according to the insertion association result;

[0027] Extract the first processed log time from the first record data;

[0028] Obtain the first data to be processed from the insertion log whose insertion timestamp is greater than the first processed log time, and perform an insertion data update operation based on the first data to be processed;

[0029] The update log is associated with the local synchronization operation log after merging the data based on the source data location identifier, and the second record data with the target data location identifier as the local identifier is obtained in the local synchronization operation log after merging the data according to the update association result.

[0030] Extract the second processed log time from the second record data;

[0031] Based on the timestamp relationship between the local synchronization operation log and the update log after merging the data, obtain the second data to be processed in the update log whose update timestamp is greater than the second processed log time, and perform field value update operation based on the second data to be processed.

[0032] In one embodiment, the step of obtaining the target record data identified as a local identifier in the local synchronization operation log after the data is merged, and updating the processed log time of the target record data to the latest source data time of the target record data, includes:

[0033] Obtain the target data location identifier list from the local synchronization operation log after merging the data;

[0034] Based on the target matching algorithm, the local identifier is matched with each target data location identifier in the target data location identifier list from multiple ends;

[0035] Obtain the identifier of the successfully matched target data, and determine the target record data corresponding to the successfully matched target data identifier;

[0036] Update the processed log time of the target record data to the latest source data time of the target record data.

[0037] In one embodiment, after the step of updating the processed log time of the target record data to the latest source data time of the target record data, the method further includes:

[0038] Check if there is a third record in the updated local synchronization operation log where both the source data location identifier and the target data location identifier are local identifiers;

[0039] If it exists, obtain the current time and set the processed log time and the latest source log time in the third record data to the current time; if it does not exist, generate the third record data, set the processed log time and the latest source log time in the third record data to the current time, and then insert it into the local synchronization operation log.

[0040] Export the updated local synchronization operation log and the updated data update log to external media.

[0041] Secondly, this application provides a data synchronization device between multiple server nodes in an isolated environment, comprising:

[0042] The import module is used to respond to data synchronization instructions in an isolated environment and import the source synchronization operation log and source data update log of the source server node from the external medium according to the data synchronization instructions.

[0043] The merging module is used to merge the source synchronization operation log into the local synchronization operation log of the target server node when it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirements.

[0044] The merging module is further configured to merge the source data update log into the local data update log of the target server node when it is determined that the log identifier of the source data update log meets the data update log merging requirements, and delete the log data in the local data update log that has been synchronized to all other server nodes.

[0045] The update module is used to determine the data to be processed based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and to perform a data update operation based on the data to be processed.

[0046] The update module is further configured to obtain the target record data identified as local in the local synchronization operation log after the data is merged, and update the processed log time of the target record data to the latest source data time of the target record data.

[0047] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0048] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0049] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0050] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0051] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0052] (1) This application uses a log import and export mechanism to import the source synchronization operation log and source data update log of the source server node from the external medium. After the data synchronization is completed, the updated local synchronization operation log and the updated data update log are exported to the external medium. At this time, there is no need for direct communication between server nodes, which is suitable for data synchronization in an isolated environment.

[0053] (2) In this application, only the fields that have changed are synchronized during data synchronization, which can effectively improve the efficiency of data synchronization compared to the full field update in the prior art. Moreover, the synchronization operation log in this application does not need to be configured in advance. It adopts a self-learning mechanism to learn automatically and dynamically adapt to newly added server nodes, which can effectively improve applicability. By merging the data update logs, data consistency is finally achieved across all server nodes.

[0054] (3) This application also provides a multi-version log merging and conflict alarm mechanism. For local data update logs with a number of records exceeding a preset threshold, a list alarm is generated, and unique data is selected from the records exceeding the preset threshold according to preset data selection rules, thereby ensuring data consistency and effectively improving the accuracy of data synchronization. In addition, this application also sets up a log cleanup strategy. Using the log cleanup strategy, log data that has been synchronized to all other server nodes in the local data update log is deleted, preventing the logs from growing indefinitely and improving system operating efficiency. The identifiers of each target data location in the identifier list can be grouped and arranged into multiple columns, and then matching can begin simultaneously from multiple ends, thereby effectively improving the efficiency of identifier matching.

[0055] In summary, this application responds to a data synchronization command in an isolated environment by importing the source synchronization operation log and source data update log of the source server node from an external medium according to the data synchronization command; when it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirement, the source synchronization operation log is merged into the local synchronization operation log of the target server node; when it is determined that the log identifier of the source data update log meets the data update log merging requirement, the source data update log is merged into the local data update log of the target server node, and log data that has been synchronized to all other server nodes in the local data update log is deleted; the data to be processed is determined according to the timestamp relationship between the merged local synchronization operation log and the data update log, and a data update operation is performed according to the data to be processed; the target record data with the target data location identifier as the local identifier in the merged local synchronization operation log is obtained, and the processed log time of the target record data is updated to the latest source data time of the target record data. In this isolated environment, multiple server nodes do not communicate with each other. When data changes occur on a server node, a data update log is generated. During data synchronization between server nodes, a synchronization operation log is recorded. Using a log import / export mechanism, the data of other server nodes is synchronized to the local machine by parsing and executing their data update logs, thus achieving data synchronization in the isolated environment. This ensures the consistency of data across all server nodes. Furthermore, the introduction of multi-version log merging and conflict alarm mechanisms improves the accuracy of data synchronization, and the efficiency of data synchronization is improved by synchronizing changed fields. Therefore, efficient and accurate data synchronization is achieved in the isolated environment. Attached Figure Description

[0056] Figure 1 This is one of the flowcharts illustrating the data synchronization method between multiple server nodes in an isolated environment provided in the embodiments of this application;

[0057] Figure 2 This is the second flowchart illustrating the data synchronization method between multiple server nodes in an isolated environment provided in the embodiments of this application;

[0058] Figure 3 This is a schematic diagram of the module structure of the data synchronization device between multiple server nodes in an isolated environment provided in the embodiments of this application;

[0059] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0062] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0063] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0064] Based on this, embodiments of this application provide a method for data synchronization between multiple server nodes in an isolated environment, referring to... Figure 1 , Figure 1 This is one of the flowcharts illustrating a data synchronization method between multiple server nodes in an isolated environment provided in this application embodiment. In this embodiment, the data synchronization method between multiple server nodes in an isolated environment includes steps S10 to S50:

[0065] Step S10: In response to the data synchronization command in the isolated environment, import the source synchronization operation log and source data update log of the source server node from the external medium according to the data synchronization command.

[0066] It should be noted that the data synchronization environment in this embodiment is an isolated environment, meaning that multiple server nodes cannot communicate with each other in an isolated environment. In this case, data synchronization can be achieved through a log import / export mechanism, suitable for isolated environments. Responding to data synchronization commands in an isolated environment indicates the need for data synchronization. At this time, the source synchronization operation log and source data update log of the source server node are imported from external media, with the server node as the reference point. , as well as Taking this as an example, the local identifiers of each server node are A, B, and C, respectively. If the data synchronization operation order is A->B->C->A->B…, in the first stage, we can determine that the source server node is A and the target server node is B. For each source server node, it is necessary to synchronize the data to all other server nodes.

[0067] It should be understood that, for server nodes, during data synchronization, the synchronization operation log is initialized. The fields of the synchronization operation log include: source data location identifier, target data location identifier, processed log time, and latest source log time. The latest source log time refers to the last time the data update log of the server node with the source data location identifier was updated. The processed log time refers to the last time the server node with the target data location identifier has processed the data update log of the server node with the source data location identifier. If the processed log time is empty, it indicates that the data of the server node with the source data location identifier has not yet been synchronized to the server node with the target data location identifier. Furthermore, the synchronization operation log in this embodiment does not require prior configuration; it uses a self-learning mechanism to automatically learn and dynamically adapt to newly added server nodes. For example, after adding server node D, the self-learning mechanism is triggered, and the synchronization operation log of server node D is automatically learned.

[0068] Step S20: When it is determined that the combined identifier in the source synchronization operation log meets the requirements for merging synchronization operation logs, the source synchronization operation log is merged into the local synchronization operation log of the target server node.

[0069] It should be understood that when merging the synchronization operation logs, the combined identifiers in the source synchronization operation logs include the first identifier combination and the third identifier combination, while the combined identifiers in the local synchronization operation logs are the second identifier combination.

[0070] Further, step S20 includes: finding the current record data in the source synchronization operation log that has the same source data location identifier and target data location identifier and the largest latest source log time, wherein the source data location identifier of the current record data is the exported data location identifier; generating a first identifier combination based on the source data location identifier and target data location identifier of each data in the source synchronization operation log; if the first identifier combination is in the local synchronization operation log, generating a second identifier combination based on the source data location identifier and target data location identifier of each data in the local synchronization operation log, and obtaining data in the local synchronization operation log whose second identifier combination is the same as the first identifier combination; when the latest source log time of each data in the source synchronization operation log is greater than the latest source log time of the data in the local synchronization operation log, updating the latest source log time of the current data in the local synchronization operation log to the latest source log time of each data in the source synchronization operation log, and updating the processed log time of the current data in the local synchronization operation log to the processed log time of each data in the source synchronization operation log; after the first loop ends... The process involves filtering out data in the source synchronization operation log whose target data location identifier matches the exported data location identifier, and modifying the target data location identifier of this data to the identifier of the local server node. A third identifier combination is generated based on the source data location identifier and target data identifier of the data in the source synchronization operation log whose target data location identifier matches the exported data location identifier. If the third identifier combination exists in the local synchronization operation log, data in the synchronization operation log whose second identifier combination matches the third identifier combination is retrieved. When the latest source log time of the data in the source synchronization operation log whose target data location identifier matches the exported data location identifier is greater than the latest source log time of the data in the synchronization operation log whose second identifier combination matches the third identifier combination, the latest source log time of the data in the synchronization operation log whose second identifier combination matches the third identifier combination is updated to the latest source log time of the data in the source synchronization operation log whose target data location identifier matches the exported data location identifier. This process continues until the second loop ends, completing the merging of the synchronization operation logs.

[0071] It is important to emphasize that when merging the synchronization operation logs, the export data location identifier needs to be identified. This identifier can be the source data location identifier of the current record in the synchronization operation log that has the same source data location identifier as the target data location identifier and the largest latest source log time. For example, when exporting synchronization data from server node A and importing it into server node B for data synchronization, the export data location identifier is A. For each piece of data in the local synchronization operation log, the first loop processing needs to be performed. Furthermore, if the first identifier combination is in the local synchronization operation log, a second identifier combination is generated based on the source data location identifier and the target data location identifier of each piece of data in the local synchronization operation log, and data in the local synchronization operation log whose second identifier combination matches the first identifier combination is retrieved. Conversely, if the first identifier combination is not in the local synchronization operation log, each piece of data in the local synchronization operation log needs to be inserted into the local synchronization operation log, and the step of determining whether the first loop has ended needs to be performed. When the latest source log time of each piece of data in the source synchronization operation log is less than or equal to the latest source log time of the data in the local synchronization operation log, the step of determining whether the first loop has ended needs to be performed.

[0072] It should be understood that before the first loop ends, it is necessary to return to the step of determining whether the first identifier combination is in the local synchronization operation log. Additionally, after filtering out data in the source synchronization operation log whose target data location identifier is the same as the export data location identifier, a second loop is needed for each data entry. This involves modifying the target data location identifier of the data in the source synchronization operation log whose target data location identifier is the export data location identifier to the identifier of the local server node. For example, when exporting synchronization data from server node A and importing that synchronization data into server node B for data synchronization, the identifier of the local server node at this time is B. Additionally, it's necessary to determine if the third identifier combination is in the local synchronization operation log. If so, retrieve the data in the synchronization operation log where the second and third identifier combinations are identical. Otherwise, if not, set the processed log time of the data in the source synchronization operation log whose target data location is the export data location identifier to empty, and insert the data in the source synchronization operation log whose target data location is the export data location identifier into the local synchronization operation log. Then, continue executing the step of determining if the second loop has ended. If yes, complete the merging of the synchronization operation logs; otherwise, return to the step of modifying the target data location identifier of the data in the source synchronization operation log whose target data location is the export data location identifier to the identifier of the local server node. The step of determining if the second loop has ended must be executed when the latest source log time of the data in the source synchronization operation log whose target data location is the export data location identifier is less than or equal to the latest source log time of the data in the synchronization operation log where the second and third identifier combinations are identical.

[0073] Step S30: When it is determined that the log identifier of the source data update log meets the data update log merging requirements, the source data update log is merged into the local data update log of the target server node, and log data that has been synchronized to all other server nodes in the local data update log is deleted.

[0074] It should be understood that source data update logs include, but are not limited to, insert logs and update logs. Since data changes mainly include inserting new data and updating existing data, server nodes need to generate separate insert logs and update logs to record these operations. To ensure data consistency, inserting data and recording the insert log must be performed as atomic operations. Similarly, updating data and recording the update log must also be performed as atomic operations. The fields in the insert log include: log identifier, source data location identifier, table name, record identifier, insert SQL, and timestamp. The fields in the update log include: log identifier, source data location identifier, table name, record identifier, field name, field value, and timestamp. Furthermore, the log identifier can be a log ID, and the record identifier can be a record ID.

[0075] It should be noted that, to avoid log redundancy and reduce the amount of data that needs to be transmitted and processed during data synchronization, the update log recording mechanism is overwrite-based. That is, for the same field with the same record ID within the same data table, the log only records the last update value of that field. After merging the source data update log into the local data update log of the target server node, it is also necessary to delete the log data in the local data update log that has been synchronized to all other server nodes.

[0076] Further, step S30 includes: if the log identifier of the source data update log is not in the local data update log, then it is determined that the update log merging requirement is met, and the source data update log is merged into the local data update log of the target server node; the number of records in the merged local data update log is counted by grouping by target field name; when the number is greater than 1, a list alarm is set for the local data update log, and unique data is selected from the records with a number greater than 1 according to a preset data selection rule; the local synchronization operation log is grouped and counted according to the source data location identifier to obtain the minimum processed log time corresponding to each source data location identifier; the data in the updated local data update log whose timestamp is less than the minimum processed log time corresponding to the source data location identifier of the record data is taken as the log data that has been synchronized to all other server nodes in the local data update log, and the log data is deleted.

[0077] Understandably, before merging the source data update log into the target server node's local data update log, it's necessary to determine whether the update log merging requirements are met based on the log identifier. If so, the source data update log is merged. For the target server node's local data update log, the number of records in the merged local data update log is counted by grouping by target field name. This target field name includes: table name, record identifier, and field name. Additionally, this embodiment also provides a list alert for local data update logs with more than one record, and selects unique data from these records according to a preset data selection rule, thereby ensuring data consistency. This preset rule can be automatically selecting the record with the largest update timestamp, or other selection methods; this embodiment does not limit this selection.

[0078] It should be noted that this embodiment also includes a log cleanup strategy. This strategy deletes log data in the local data update log that has been synchronized to all other server nodes. Specifically, the local synchronization operation logs are grouped and statistically analyzed based on the source data location identifier. The minimum processed log time corresponding to the source data location identifier is determined. Then, data in the updated local data update log with timestamps less than the minimum processed log time is filtered out. The filtered data is the log data in the local data update log that has been synchronized to all other server nodes, and this log data is directly deleted. This avoids the logs from growing indefinitely and improves system operating efficiency.

[0079] Step S40: Determine the data to be processed based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and perform a data update operation based on the data to be processed.

[0080] It is understandable that the data to be processed refers to the data that needs to be inserted, updated, and have its field values ​​updated. This data to be processed includes a first data to be processed and a second data to be processed. The first data to be processed is used to perform the insert and update operation, and the second data to be processed is used to perform the field value update operation. That is, only the fields that have changed are synchronized, which can effectively improve the efficiency of data synchronization.

[0081] Further, step S40 includes: associating the insertion log with the merged local synchronization operation log based on the source data location identifier, and obtaining the first record data in the merged local synchronization operation log with the target data location identifier as the local identifier according to the insertion association result; extracting the first processed log time from the first record data; obtaining the first unprocessed data in the insertion log with the insertion timestamp greater than the first processed log time, and performing an insertion update operation based on the first unprocessed data; associating the update log with the merged local synchronization operation log based on the source data location identifier, and obtaining the second record data in the merged local synchronization operation log with the target data location identifier as the local identifier according to the update association result; extracting the second processed log time from the second record data; obtaining the second unprocessed data in the update log with the update timestamp greater than the second processed log time according to the timestamp relationship between the merged local synchronization operation log and the update log, and performing a field value update operation based on the second unprocessed data.

[0082] It should be understood that both the insertion log and the local synchronization operation log have a field identifying the source data location. Therefore, the insertion log can be associated with the local synchronization operation log after merging the data based on the source data location identifier. The first processed log time refers to the processed log time in the first record data in the local synchronization operation log after merging the data, where the target data location identifier is the local identifier. After extracting the first processed log time from the first record data, the first data to be processed can be filtered out from the insertion log, where the insertion timestamp is greater than the first processed log time. At this time, the insertion update operation can be performed based on the first data to be processed, that is, the insertion data processing can be performed.

[0083] It is understandable that both the update log and the local synchronization operation log have a field identifying the source data location. Therefore, the update log can be associated with the local synchronization operation log after merging the data based on the source data location identifier. The second processed log time refers to the processed log time in the second record data in the local synchronization operation log after merging the data, where the target data location identifier is the local identifier. After extracting the second processed log time from the second record data, the second data to be processed can be filtered from the update log by combining the timestamp relationship between the local synchronization operation log and the update log, where the update timestamp is greater than the second processed log time. At this time, the field value update operation can be performed based on the second data to be processed, i.e., field value update processing is performed. Compared with the full field update of the prior art, this embodiment only synchronizes the fields that have changed, thereby effectively improving the efficiency of data synchronization.

[0084] Step S50: Obtain the target record data identified as the local identifier in the local synchronization operation log after merging the data, and update the processed log time of the target record data to the latest source data time of the target record data.

[0085] It should be understood that the target record data refers to the record data in the local synchronization operation log after data merging that is identified by the local identifier. In the data synchronization of this embodiment, the processed log time will also be updated. After the target record data is obtained, the latest source data time will be extracted from the target record data, and the processed log time of the target record data will be updated to the latest source data time of the target record data, thereby realizing the update processing of the processed log time and completing the data synchronization between server nodes.

[0086] Further, after step S50, the method includes: detecting whether there is a third record data in the updated local synchronization operation log where both the source data location identifier and the target data location identifier are local identifiers; if it exists, obtaining the current time and setting the processed log time and the latest source log time in the third record data to the current time; if it does not exist, generating the third record data, setting the processed log time and the latest source log time in the third record data to the current time, and inserting it into the local synchronization operation log; and exporting the updated local synchronization operation log and the updated data update log to an external medium.

[0087] It should be noted that the third record data refers to the record data where both the source data location and the target data location are identified as local. It is necessary to determine whether this record data exists in the local synchronization operation log. If it exists, the processed log time and the latest source log time in the third record data are directly set to the current time. If it does not exist, the processed log time and the latest source log time in the third record data are set to the current time before being inserted into the local synchronization operation log. To facilitate subsequent data synchronization, this embodiment also uses a log import / export mechanism to export the updated local synchronization operation log and the updated data update log to external media.

[0088] It is important to emphasize that the data synchronization process in this embodiment forms a complete closed loop. For the source server node, the generated source synchronization operation log and source data update log are exported to an external medium. Then, the source synchronization operation log and source data update log of the source server node are imported from the external medium to start the current round of data synchronization update process, such as merging synchronization operation logs, merging data update logs, deleting data update logs, etc., until the current round of data synchronization update process ends.

[0089] In response to a data synchronization command in an isolated environment, this embodiment imports the source synchronization operation log and source data update log of the source server node from an external medium according to the data synchronization command. When it is determined that the combined identifier in the source synchronization operation log meets the requirements for merging synchronization operation logs, the source synchronization operation log is merged into the local synchronization operation log of the target server node. When it is determined that the log identifier of the source data update log meets the requirements for merging data update logs, the source data update log is merged into the local data update log of the target server node, and log data that has been synchronized to all other server nodes in the local data update log is deleted. The data to be processed is determined according to the timestamp relationship between the merged local synchronization operation log and the data update log, and a data update operation is performed according to the data to be processed. The target record data with the target data location identifier set as the local identifier in the merged local synchronization operation log is obtained, and the processed log time of the target record data is updated to the latest source data time of the target record data. In this isolated environment, multiple server nodes do not communicate with each other. When data changes on a server node, a data update log is generated. When data is synchronized between server nodes, a synchronization operation log is recorded. Using a log import / export mechanism, the data of other server nodes is synchronized to the local machine by parsing and executing the data update logs of other server nodes. This enables data synchronization in an isolated environment and ensures the consistency of data across all server nodes.

[0090] In one specific implementation, this application provides steps for updating the processed log time of the local synchronization operation log. Please refer to... Figure 2 , Figure 2 This is the second flowchart illustrating the data synchronization method between multiple server nodes in an isolated environment provided in this application embodiment. Step S50 includes steps S501 to S504:

[0091] Step S501: Obtain the target data location identifier list from the local synchronization operation log after merging the data.

[0092] It should be noted that the target data location identifier list refers to a list consisting of the various target data location identifiers in the local synchronization operation log after the data is merged, such as target data location identifier A, target data location identifier B, and target data location identifier C, etc.

[0093] Step S502: Based on the target matching algorithm, the local identifier is matched with each target data location identifier in the target data location identifier list from multiple ends.

[0094] It is understandable that a target matching algorithm refers to an algorithm that matches a local identifier with each target identifier in a list of target identifiers. This target matching algorithm is a multi-dimensional feature matching algorithm. To effectively improve the efficiency of identifier matching, the target identifiers in the list of target identifiers can be grouped and arranged into multiple columns, and then matching can begin simultaneously from multiple ends.

[0095] Step S503: Obtain the identifier of the target data that has been successfully matched, and determine the target record data corresponding to the identifier of the target data that has been successfully matched.

[0096] It should be understood that the target record data refers to the record data in the local synchronization operation log after the data is merged, where the target data location is identified as the local identifier. After the matching is completed, the target data location identifier that is successfully matched is extracted from the matching result, and the target record data corresponding to the successfully matched target data location identifier is determined. This target record data is the record data where the target data location is identified as the local identifier.

[0097] Step S504: Update the processed log time of the target record data to the latest source data time of the target record data.

[0098] Understandably, after obtaining the target record data, the latest source data time will be extracted from the target record data, and the processed log time of the local synchronization operation log will be updated to the latest source data time to complete the data synchronization between the source server node and the target server node.

[0099] This embodiment obtains a list of target data location identifiers from the local synchronization operation log after merging the data; based on a target matching algorithm, it matches the local identifiers with each target data location identifier in the target data location identifier list from multiple endpoints; it obtains the successfully matched target data location identifiers and determines the target record data corresponding to the successfully matched target data location identifiers; and it updates the processed log time of the target record data to the latest source data time of the target record data. Through the above method, after obtaining the list of target data location identifiers composed of various target data location identifiers, it uses a multi-end simultaneous matching method based on the target matching algorithm to match the local identifiers with each target data location identifier in the target data location identifier list, which can effectively improve the identifier matching efficiency. Then, it extracts the successfully matched target data location identifiers from the matching results, and updates the processed log time of the target record data with the latest source data time of the target record data using the target data location identifier, thereby effectively improving the efficiency and accuracy of updating data in an isolated environment.

[0100] The following describes the data synchronization device for multiple server nodes in an isolated environment provided in this application. The data synchronization device for multiple server nodes in an isolated environment described below can be referred to in conjunction with the data synchronization method for multiple server nodes in an isolated environment described above. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of the module structure of a data synchronization device between multiple server nodes in an isolated environment provided in this application embodiment, including:

[0101] The import module T10 is used to respond to data synchronization instructions in an isolated environment and import the source synchronization operation log and source data update log of the source server node from the external medium according to the data synchronization instructions.

[0102] The merging module T20 is used to merge the source synchronization operation log into the local synchronization operation log of the target server node when it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirements.

[0103] The merging module T20 is further configured to merge the source data update log into the local data update log of the target server node when it is determined that the log identifier of the source data update log meets the data update log merging requirements, and delete the log data in the local data update log that has been synchronized to all other server nodes.

[0104] The update module T30 is used to determine the data to be processed based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and to perform a data update operation based on the data to be processed.

[0105] The update module T30 is also used to obtain the target record data identified as local in the local synchronization operation log after the data is merged, and update the processed log time of the target record data to the latest source data time of the target record data.

[0106] In response to a data synchronization command in an isolated environment, this embodiment imports the source synchronization operation log and source data update log of the source server node from an external medium according to the data synchronization command. When it is determined that the combined identifier in the source synchronization operation log meets the requirements for merging synchronization operation logs, the source synchronization operation log is merged into the local synchronization operation log of the target server node. When it is determined that the log identifier of the source data update log meets the requirements for merging data update logs, the source data update log is merged into the local data update log of the target server node, and log data that has been synchronized to all other server nodes in the local data update log is deleted. The data to be processed is determined according to the timestamp relationship between the merged local synchronization operation log and the data update log, and a data update operation is performed according to the data to be processed. The target record data with the target data location identifier set as the local identifier in the merged local synchronization operation log is obtained, and the processed log time of the target record data is updated to the latest source data time of the target record data. In this isolated environment, multiple server nodes do not communicate with each other. When data changes on a server node, a data update log is generated. When data is synchronized between server nodes, a synchronization operation log is recorded. Using a log import / export mechanism, the data of other server nodes is synchronized to the local machine by parsing and executing the data update logs of other server nodes. This enables data synchronization in an isolated environment and ensures the consistency of data across all server nodes.

[0107] It is understood that the detailed functional implementation of each of the above modules can be found in the description of the aforementioned method embodiments, and will not be repeated here.

[0108] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0109] Based on the methods in the above embodiments, this application provides an electronic device, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0110] It should be noted that the system may include: a processor 10, a communication interface 20, a memory 30, and a communication bus 40. The processor 10, communication interface 20, and memory 30 communicate with each other via the communication bus 40. The processor 10 can invoke logical instructions stored in the memory 30 to execute the methods described in the above embodiments.

[0111] Furthermore, the logical instructions in the aforementioned memory 30 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0112] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0113] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0114] It is understood that the processor in the embodiments of this application can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0115] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor.

[0116] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. Those skilled in the art will readily understand that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for data synchronization among multiple server nodes in an isolated environment, characterized in that, include: In response to a data synchronization command in an isolated environment, the source synchronization operation log and source data update log of the source server node are imported from an external medium according to the data synchronization command. The source synchronization operation log includes a source data location identifier, a target data location identifier, the processed log time, and the latest source log time. This source synchronization operation log does not require prior configuration and automatically adapts to newly added server nodes using a self-learning mechanism. The self-learning mechanism is triggered to automatically learn the synchronization operation logs of the server nodes. The source data update log includes a log identifier, a source data location identifier, a table name, a record identifier, field names, field values, and a timestamp. When it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirements, the source synchronization operation log is merged into the local synchronization operation log of the target server node. When it is determined that the log identifier of the source data update log meets the data update log merging requirements, the source data update log is merged into the local data update log of the target server node, and the log data in the local data update log that has been synchronized to all other server nodes is deleted. The data to be processed is determined based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and a data update operation is performed based on the data to be processed. Obtain the target record data with the local identifier in the local synchronization operation log after merging the data, and update the processed log time of the target record data to the latest source data time of the target record data. The steps of determining the data to be processed based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and performing a data update operation based on the data to be processed, include: The update log is associated with the local synchronization operation log after merging the data based on the source data location identifier, and the second record data with the target data location identifier as the local identifier is obtained in the local synchronization operation log after merging the data according to the update association result. Extract the second processed log time from the second record data; Based on the timestamp relationship between the local synchronization operation log and the update log after merging the data, obtain the second data to be processed in the update log whose update timestamp is greater than the second processed log time, and perform field value update operation based on the second data to be processed.

2. The method as described in claim 1, characterized in that, The step of merging the source synchronization operation log into the local synchronization operation log of the target server node when it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirements includes: Find the current record data in the source synchronization operation log that has the same source data location identifier as the target data location identifier and the largest latest source log time. The source data location identifier of the current record data is the exported data location identifier. A first identifier combination is generated based on the source data location identifier and the target data location identifier of each data entry in the source synchronization operation log. If the first identifier combination is in the local synchronization operation log, then a second identifier combination is generated based on the source data location identifier and the target data location identifier of each data entry in the local synchronization operation log, and data in the local synchronization operation log that has the same second identifier combination as the first identifier combination is obtained. When the latest source log time of each data entry in the source synchronization operation log is greater than the latest source log time of the data in the local synchronization operation log, the latest source log time of the current data in the local synchronization operation log is updated to the latest source log time of each data entry in the source synchronization operation log, and the processed log time of the current data in the local synchronization operation log is updated to the processed log time of each data entry in the source synchronization operation log. After the first loop ends, filter out the data in the source synchronization operation log whose target data location is identified by the exported data location, and modify the target data location identifier of the data in the source synchronization operation log whose target data location is identified by the exported data location to the identifier of the local server node. A third identifier combination is generated based on the source data location identifier and the target data identifier of the data whose target data location identifier is the exported data location identifier in the source synchronization operation log; If the third identifier combination is in the local synchronization operation log, then retrieve the data in the synchronization operation log that is the same as the second identifier combination and the third identifier combination; When the latest source log time of the data whose target data location is identified as the exported data location in the source synchronization operation log is greater than the latest source log time of the data in the synchronization operation log that has the same second identifier combination and the same third identifier combination, the latest source log time of the data in the synchronization operation log that has the same second identifier combination and the same third identifier combination is updated to the latest source log time of the data in the source synchronization operation log whose target data location is identified as the exported data location, until the second loop ends, completing the merging of the synchronization operation logs.

3. The method as described in claim 1, characterized in that, The step of merging the source data update log into the local data update log of the target server node when the log identifier of the source data update log meets the data update log merging requirements, and deleting log data in the local data update log that has been synchronized to all other server nodes, includes: If the log identifier of the source data update log is not in the local data update log, then the update log merging requirement is met, and the source data update log is merged into the local data update log of the target server node. Group the data in the merged local data update log by target field name to count the number of records. When the number is greater than 1, a list alarm is generated for the local data update log, and unique data is selected from the record data with a number greater than 1 according to the preset data selection rules; The source data location identifier in the updated local data update log is grouped and statistically analyzed, and the local synchronization operation log is grouped and statistically analyzed according to the source data location identifier to obtain the minimum processed log time corresponding to each source data location identifier. In the updated local data update log, data with timestamps less than the minimum processed log time corresponding to the source data location identifier of the recorded data are taken as log data that has been synchronized to all other server nodes in the local data update log, and the log data is deleted.

4. The method as described in claim 1, characterized in that, The steps of determining the data to be processed based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and performing a data update operation based on the data to be processed, include: Based on the source data location identifier, the inserted log is associated with the local synchronization operation log after merging the data, and the first record data in the local synchronization operation log after merging the data with the target data location identifier as the local identifier is obtained according to the insertion association result; Extract the first processed log time from the first record data; Obtain the first data to be processed from the insertion log whose insertion timestamp is greater than the first processed log time, and perform an insertion data update operation based on the first data to be processed.

5. The method as described in claim 1, characterized in that, The step of obtaining the target record data identified as the local identifier in the local synchronization operation log after the merged data, and updating the processed log time of the target record data to the latest source data time of the target record data, includes: Obtain the target data location identifier list from the local synchronization operation log after merging the data; Based on the target matching algorithm, the local identifier is matched with each target data location identifier in the target data location identifier list from multiple ends; Obtain the identifier of the successfully matched target data, and determine the target record data corresponding to the successfully matched target data identifier; Update the processed log time of the target record data to the latest source data time of the target record data.

6. The method according to any one of claims 1 to 5, characterized in that, After the step of updating the processed log time of the target record data to the latest source data time of the target record data, the method further includes: Check if there is a third record in the updated local synchronization operation log where both the source data location identifier and the target data location identifier are local identifiers; If it exists, obtain the current time and set the processed log time and the latest source log time in the third record data to the current time; if it does not exist, generate the third record data, set the processed log time and the latest source log time in the third record data to the current time, and then insert it into the local synchronization operation log. Export the updated local synchronization operation log and the updated data update log to external media.

7. A data synchronization device for multiple server nodes in an isolated environment, characterized in that, include: The import module is used to respond to data synchronization commands in an isolated environment, and import the source synchronization operation logs and source data update logs of the source server node from an external medium according to the data synchronization commands. The source synchronization operation logs include source data location identifiers, target data location identifiers, processed log times, and the latest source log times. These source synchronization operation logs do not require prior configuration and automatically adapt to newly added server nodes using a self-learning mechanism. The self-learning mechanism is triggered to automatically learn the synchronization operation logs of the server nodes. The source data update logs include log identifiers, source data location identifiers, table names, record identifiers, field names, field values, and timestamps. The merging module is used to merge the source synchronization operation log into the local synchronization operation log of the target server node when it is determined that the combined identifier in the source synchronization operation log meets the synchronization operation log merging requirements. The merging module is further configured to merge the source data update log into the local data update log of the target server node when it is determined that the log identifier of the source data update log meets the data update log merging requirements, and delete the log data in the local data update log that has been synchronized to all other server nodes. The update module is used to determine the data to be processed based on the timestamp relationship between the local synchronization operation log and the data update log after merging the data, and to perform a data update operation based on the data to be processed. The update module is also used to obtain the target record data identified as local in the local synchronization operation log after the data is merged, and update the processed log time of the target record data to the latest source data time of the target record data. The update module is further configured to associate the update log with the local synchronization operation log after merging the data based on the source data location identifier, and obtain the second record data in the local synchronization operation log after merging the data whose target data location identifier is the local identifier according to the update association result; extract the second processed log time from the second record data; obtain the second unprocessed data in the update log whose update timestamp is greater than the second processed log time according to the timestamp relationship between the local synchronization operation log after merging the data and the update log, and perform a field value update operation according to the second unprocessed data.

8. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on the processor, it causes the processor to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-6.

Images