A data processing method and terminal

Abstract

This invention discloses a data processing method and terminal. The method includes the following steps: S1, receiving synchronization data sent by other master nodes, determining whether the synchronization data is conflicting data, if not, storing the synchronization data in the data source storage area, if so, storing the synchronization data and the conflicting data corresponding to the synchronization data in the data source storage area in the conflict data cache area; S2, receiving a data access request, determining whether the accessed data is in the conflict data cache area, if so, reading all corresponding data from the conflict data cache area, performing conflict processing on all read data, returning the conflict-processed data, and updating all read data in the conflict data cache area with the conflict-processed data, if not, reading and returning the corresponding data from the data source storage area; by adopting the above-mentioned conflict data cache and conflict data cache area priority access mechanism, the data synchronization efficiency is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data processing method and terminal. Background Technology

[0002] Many systems currently incorporate data synchronization functionality. Most systems employ a master-slave architecture for synchronization, where data is written to the master node, and multiple slave nodes then pull data from the master node to achieve synchronization. However, this architecture suffers from a single point of failure: with only one master node, its failure renders the entire system unusable. To overcome this, a multi-master architecture emerged, where multiple master nodes exist, each with identical functionality, eliminating the concept of a central master node. This architecture effectively eliminates the single point of failure. However, it introduces new challenges: in a multi-master model, data synchronization between master nodes is crucial to ensure eventual consistency. To achieve this, data synchronization between master nodes is necessary, and during this process, data conflicts may arise between them.

[0003] Currently, a common approach to resolving data conflicts in a multi-master architecture is to set up a conflict resolution mechanism to handle conflicting data one by one. Once the conflict resolution is complete, the node then updates and synchronizes its own data. However, because conflict resolution takes time, accessing the master node's own data will only allow access to older data. Until the conflict resolution is complete and the data is updated, the latest data cannot be obtained, thus impacting data synchronization efficiency. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a data processing method and terminal that can improve the data synchronization efficiency in a multi-master node architecture.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A data processing method includes the following steps:

[0007] S1. Receive synchronization data sent by other master nodes, determine whether the synchronization data is conflicting data, if not, store the synchronization data in the data source storage area, if so, store the synchronization data and the conflicting data corresponding to the synchronization data in the data source storage area in the conflicting data cache area.

[0008] S2. Receive a data access request, determine whether the accessed data is in the conflict data cache area, if so, read all the corresponding data from the conflict data cache area, perform conflict processing on all the read data, return the conflict-processed data, and update all the read data in the conflict data cache area with the conflict-processed data; if not, read and return the corresponding data from the data source storage area.

[0009] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0010] A data processing terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the data processing method described above.

[0011] The beneficial effects of this invention are as follows: During data synchronization, when the synchronized data is conflicting data, it is preferentially written to the conflicting data cache. When data is accessed, the conflicting data cache is accessed first. If the data to be accessed is stored in the conflicting data cache, the data to be accessed is directly processed for conflict before access, and the conflicting data cache is updated based on the data after conflict processing. Through the conflicting data cache and the mechanism of preferential access to the conflicting data cache, it is ensured that conflicting data can be accessed directly before the conflicting data is processed. Furthermore, by accessing the conflicting data cache, conflict processing can also be performed on the data simultaneously, realizing the synchronous execution of data conflict processing and conflicting data access. This avoids the problem in the prior art where, if conflicting data exists, only old data can be accessed. If new data is to be accessed, data synchronization can only be performed after the conflicting data is processed, and only then can the new data be accessed, resulting in the inability to access the latest data in a timely manner. By adopting the above-mentioned conflicting data cache and the mechanism of preferential access to the conflicting data cache, the latest data can be accessed immediately, and data conflicts are also processed during the access process, thereby improving data synchronization efficiency. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the steps of a data processing method according to an embodiment of the present invention;

[0013] Figure 2 This is a schematic diagram of the structure of a data processing terminal according to an embodiment of the present invention. Detailed Implementation

[0014] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0015] Please refer to Figure 1A data processing method, comprising the following steps:

[0016] S1. Receive synchronization data sent by other master nodes, determine whether the synchronization data is conflicting data, if not, store the synchronization data in the data source storage area, if so, store the synchronization data and the conflicting data corresponding to the synchronization data in the data source storage area in the conflicting data cache area.

[0017] S2. Receive a data access request, determine whether the accessed data is in the conflict data cache area, if so, read all the corresponding data from the conflict data cache area, perform conflict processing on all the read data, return the conflict-processed data, and update all the read data in the conflict data cache area with the conflict-processed data; if not, read and return the corresponding data from the data source storage area.

[0018] As described above, the beneficial effects of this invention are as follows: During data synchronization, when the synchronized data is conflicting data, it is preferentially written to the conflicting data cache. When data is accessed, the conflicting data cache is accessed preferentially. If the data to be accessed is stored in the conflicting data cache, the data to be accessed is directly processed for conflict before access, and the conflicting data cache is updated based on the processed data. Through the conflicting data cache and the mechanism of preferential access to the conflicting data cache, it is ensured that conflicting data can be directly accessed before the conflicting data is processed, and the access to the conflicting data cache can also simultaneously process the conflicting data. The conflict handling mechanism enables simultaneous handling of data conflicts and access to conflicting data. This avoids the problem in existing technologies where, when conflicting data exists, only old data can be accessed. If new data is needed, data synchronization can only begin after the conflict is resolved, leading to the inability to access the latest data in a timely manner. By adopting the conflict data caching and conflict data cache priority access mechanism, the latest data can be accessed immediately, and data conflicts are handled during the access process, thereby improving data synchronization efficiency. This mechanism is particularly suitable for data synchronization in multi-master node architectures.

[0019] Furthermore, the synchronization data includes key values, and the key values ​​are different for different synchronization data;

[0020] The determination of whether the synchronized data is conflicting data includes:

[0021] The data source storage area is searched according to the key value of the synchronized data to determine whether target data containing the key value exists. If not, the synchronized data is not conflicting data. If so, it is determined whether the data update time of the synchronized data is consistent with the data update time of the target data. If not, the synchronized data is conflicting data.

[0022] As described above, by setting a key value that can uniquely identify each piece of data to be synchronized, it is possible to conveniently and quickly search the data source storage area to see if the data to be synchronized is stored based on the key value. In addition, combined with the data update time, it is possible to quickly and accurately determine whether the data is conflicting. Furthermore, the data update time can also ensure the consistency of data after synchronization between different master nodes.

[0023] Furthermore, if the data update time of the synchronized data is consistent with the data update time of the target data, then the synchronized data is not conflicting data, and the step of storing the synchronized data in the data source storage area is not performed.

[0024] As described above, by using both key-value pairs and data update time, it is possible to accurately and quickly determine whether the received synchronization data has already been synchronized. If it has already been synchronized, there is no need to perform the step of storing it in the data source storage area; it can be discarded directly, reducing resource consumption and thus improving resource utilization.

[0025] Furthermore, the synchronization data also includes a timestamp, which is added to the key value as a suffix;

[0026] The step of searching the data source storage area based on the key value of the synchronized data includes:

[0027] The data source storage area is searched based on the key value of the synchronized data after removing the timestamp suffix.

[0028] As described above, adding the timestamp to the key-value pair as a suffix means that the key-value pair contains the timestamp. The timestamp is stored in the spare space of the key-value pair, eliminating the need for additional storage space. This effectively reduces the storage space occupied by referencing the key-value pair and the timestamp. Furthermore, by using the timestamp, the time synchronization between different master nodes can be guaranteed, further improving the accuracy and reliability of data synchronization.

[0029] Furthermore, the conflict handling for all read data includes:

[0030] All data are merged based on their update time, retaining only the latest data.

[0031] As described above, by comparing the latest data in the conflicting data based on the data update time of each data point, the conflicting data can be quickly merged to resolve the data conflict problem.

[0032] Furthermore, it also includes the following steps:

[0033] A preset asynchronous process is triggered to perform conflict resolution on the data in the conflict data cache. The data in the data source storage area is updated based on the conflict-resolved data, and the conflict-resolved data is deleted from the conflict data cache.

[0034] As described above, by setting up an asynchronous process to handle data conflicts in the conflict data buffer, and by combining data access and asynchronous processes, the efficiency of data conflict handling is improved. Furthermore, since the conflict data buffer centrally stores conflicting data, all conflicting data can be processed at once when the asynchronous process is triggered. This avoids the current practice of processing conflicting data sequentially for each new conflict. The centralized storage of conflicting data in the conflict data buffer, combined with the centralized processing by the asynchronous process, further improves the efficiency of conflict handling, thereby further enhancing synchronization efficiency.

[0035] Furthermore, a preset asynchronous process is triggered periodically.

[0036] As described above, the timed triggering of the preset asynchronous process can promptly resolve conflicts in the conflict data cache, thereby improving the update speed of the data in the data source storage area.

[0037] Furthermore, the conflict handling of data in the conflict data buffer includes:

[0038] For each piece of data to be processed in the conflict data buffer, perform the following operations in sequence:

[0039] Search the remaining data in the conflict data buffer based on the key value of the data to be processed, and determine whether the data corresponding to the key value is found. If not, the data to be searched is the data after conflict processing. Otherwise, conflict processing is performed based on the data to be processed and all the data with the corresponding key value found.

[0040] As described above, when accessing conflicting data, conflict resolution is performed and the data is updated in the conflict data cache. Therefore, when an asynchronous process is triggered to perform conflict data processing, if it encounters data that has already undergone conflict resolution during data access, further conflict resolution is unnecessary; the data can be directly updated to the data source storage area. This avoids resource waste and further improves synchronization efficiency.

[0041] Furthermore, for each conflict-resolved data point identified, the data in the data source storage area is updated based on the identified conflict-resolved data.

[0042] Alternatively, after all data in the conflict data cache has been conflict-handled, the data in the data source storage area can be updated based on the conflict-handled data.

[0043] As described above, when performing conflict data processing through an asynchronous process, it is possible to either update the data in real time after each conflict-resolved data is determined, or update the data in the data source storage area all at once after all the data in the conflict data cache has been conflict-resolved. The processing method is flexible and can be adapted to the specific application scenario.

[0044] Please refer to Figure 2 A data processing terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement each step of the above-described data processing method.

[0045] The data processing method and terminal described in this application are applicable to various data synchronization scenarios, and are particularly suitable for data synchronization applications under a multi-master node architecture. The following is a detailed description of the implementation method:

[0046] Example 1

[0047] Please refer to Figure 1 A data processing method, comprising the following steps:

[0048] S1. Receive synchronization data sent by other master nodes, determine whether the synchronization data is conflicting data, if not, store the synchronization data in the data source storage area, if so, store the synchronization data and the conflicting data corresponding to the synchronization data in the data source storage area in the conflicting data cache area.

[0049] This embodiment is a data synchronization system with a multi-master node architecture. For example, if there are three master nodes A, B and C in the system, other terminals in the data synchronization system can randomly write data to these three master nodes. Each master node has a data synchronization function and can synchronize data with other master nodes.

[0050] In this embodiment, a conflict data cache is added to each master node, which is used to store conflicting data;

[0051] For example, when master node A needs to synchronize data with master node C, master node A sends a data synchronization request to master node C. This data synchronization request contains the data records to be synchronized, such as three data records: Record1, Record2, and Record3. Each synchronized data record includes a key and a timestamp. The key values ​​of different synchronized data are different, that is, each synchronized data record has a key used to uniquely identify it. The key can be used to locate the corresponding synchronized data. The timestamp is used to ensure the consistency of time between different master nodes and to achieve time synchronization between different master nodes.

[0052] The determination of whether the synchronized data is conflicting data includes:

[0053] The data source storage area is searched according to the key value of the synchronized data to determine whether there is target data containing the key value. If not, the synchronized data is not conflicting data and can be directly stored in the data source storage area. If yes, it is determined whether the data update time of the synchronized data is consistent with the data update time of the target data. If not, the synchronized data is conflicting data.

[0054] If the data update time of the synchronized data is consistent with the data update time of the target data, then the synchronized data is not conflicting data, and the step of storing the synchronized data in the data source storage area is not performed;

[0055] In other words, by using the key value and data update time, it is easy and quick to determine whether the received synchronized data is conflicting data. For example, for data record Record1, a data record Record1' with the same key value is found in the data source storage area. At this time, the data update time of Record1 and Record1' is further checked. If the data update times of the two are the same, it means that the master node C has already synchronized the data corresponding to the key value. For example, it may have been synchronized from the master node B before, so there is no need to synchronize again and the data can be ignored directly.

[0056] In an optional implementation, the timestamp is added to the key value as a suffix;

[0057] The step of searching the data source storage area based on the key value of the synchronized data includes:

[0058] The key value of the synchronized data after removing the timestamp suffix is ​​searched in the data source storage area;

[0059] S2. Receive a data access request, determine whether the accessed data is in the conflict data cache area, if so, read all the corresponding data from the conflict data cache area, perform conflict processing on all the read data, return the conflict-processed data, and update all the read data in the conflict data cache area with the conflict-processed data; if not, read and return the corresponding data from the data source storage area.

[0060] The conflict handling for all read data includes:

[0061] All data are merged based on their update time, retaining only the latest data.

[0062] Each piece of data has a corresponding update time and update field. Based on the update time and update field, multiple conflicting pieces of data can be merged.

[0063] For example, when a client requests data record Record2 from master node A, it first searches the conflict data cache based on the key value of the data record. If multiple records are found, forming a conflict list, the conflict is resolved based on the update time and updated fields of each record in the conflict list. The conflicting data in the conflict list is then merged. For example, if there are three conflicting records: Record21, Record22, and Record23, and each record has fields A, B, C, and D; if Record21 was updated at 13:04 with the updated field A', Record22 was updated at 13:14 with the updated field A'", and Record23 was updated at 13:24 with the updated field B', then after merging, the resulting data is: A'", B', C, and D. This merged data is then returned to the client that sent the data access request, and the conflict list is replaced with this new record.

[0064] Example 2

[0065] This embodiment further defines, based on Embodiment 1, that: a preset asynchronous process is added to asynchronously handle data conflicts in the conflict data buffer, specifically:

[0066] It also includes the following steps:

[0067] A preset asynchronous process is triggered to perform conflict resolution on the data in the conflict data cache, the data in the data source storage area is updated according to the conflict-resolved data, and the conflict-resolved data is deleted from the conflict data cache.

[0068] Among them, preset asynchronous processes can be triggered at regular intervals;

[0069] The conflict handling of data in the conflict data cache includes:

[0070] For each piece of data to be processed in the conflict data buffer, perform the following operations in sequence:

[0071] Search the remaining data in the conflict data buffer based on the key value of the data to be processed, and determine whether the data corresponding to the key value is found. If not, the data to be searched is the data after conflict processing. Otherwise, conflict processing is performed based on the data to be processed and all the data with the corresponding key value found.

[0072] In an optional implementation, the data in the conflict data cache can be pre-integrated, and conflict data belonging to the same key value can be stored in the same conflict list. A correspondence between the conflict list and the key value can be established. In this way, the corresponding conflict list can be quickly located based on the key value, which is convenient for handling the corresponding conflicts, adding new conflict data to the conflict list, and deduplication. For example, if two consecutive data synchronizations target the same data in the data source storage area and both are conflict data, when storing the conflict data for the second time, it is not necessary to store the data in the data source storage area into the conflict data cache again, because it has already been added when storing the conflict data for the first time. With the help of the corresponding conflict list, it is easy to identify the duplicate data and quickly perform deduplication. It is also convenient for handling and updating conflict data during access.

[0073] Furthermore, through the above integration operation, when the preset asynchronous process is performing conflict handling, it can quickly determine whether the data has been processed during the access process based on the number of data in each conflict list. If the number is 1, it means that the data has been merged. At this time, it can be updated to the data source storage area first, and then the conflict list with a data count of not 1 can be processed one by one.

[0074] After data conflict resolution is completed, when synchronizing the processed data to the data source storage area:

[0075] In an optional implementation, once a conflict-resolved data is identified, the data in the data source storage area is updated based on the identified conflict-resolved data.

[0076] In another optional implementation, after all the data in the conflict data cache has been conflict-handled, the data in the data source storage area is updated based on the conflict-handled data.

[0077] Specifically, updating the data in the data source storage area based on the conflict-resolved data involves:

[0078] Based on the key value of the conflict-resolved data, the system retrieves the corresponding data from the data source storage area, and then updates the located data with the conflict-resolved data.

[0079] Example 3

[0080] Please refer to Figure 2 A data processing terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to perform the various steps of the data processing method described in Embodiment 1 or Embodiment 2 above.

[0081] In summary, the data processing method and terminal provided by this invention, during data synchronization, prioritizes writing conflicting data to a conflict data cache. When data is accessed, the conflict data cache is accessed first. If the data to be accessed is stored in the conflict data cache, conflict resolution is performed on the data before access, and the conflict data cache is updated based on the resolved data. Simultaneously, an asynchronous process is set up to periodically resolve conflicts in the data in the conflict data cache. This mechanism of conflict data caching, priority access to the conflict data cache, and asynchronous process handling of conflicting data ensures that conflicting data can be accessed directly before any conflicting data is processed. Furthermore, accessing the conflict data cache also allows for simultaneous conflict resolution, achieving synchronous conflict resolution and access. The asynchronous process handles conflicting data... Conflicting data in the cache is processed periodically, achieving centralized and unified processing of conflicting data. This avoids the problems of existing technologies where, when conflicting data exists, only old data can be accessed. To access new data, data synchronization can only be performed after the conflicting data has been processed. Furthermore, conflict resolution is performed one conflict at a time, meaning each conflicting data is processed individually, leading to delays in accessing the latest data and slow conflict processing speed. By adopting the aforementioned conflicting data caching, priority access to the conflicting data cache, and asynchronous process synchronous processing of conflicting data, the latest data can be accessed immediately, and data conflicts are handled during the access process. At the same time, the centralized and unified processing of conflicting data through asynchronous processes greatly improves data synchronization efficiency.

[0082] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A data processing method, characterized by, The method comprises the steps of: S1, receiving synchronization data sent by other master nodes, judging whether the synchronization data is conflict data, if not, storing the synchronization data to a data source storage area, if yes, storing the synchronization data and conflict data corresponding to the synchronization data in the data source storage area to a conflict data cache area; S2, receiving a data access request, judging whether the accessed data is in the conflict data cache area, if yes, reading all corresponding data from the conflict data cache area, performing conflict processing on all the read data, returning the conflict-processed data, and updating all the read data to the conflict-processed data in the conflict data cache area, if not, reading and returning corresponding data from the data source storage area.

2. The data processing method of claim 1, wherein, The synchronization data comprises key values, and the key values of different synchronization data are different; The judgment of whether the synchronization data is conflict data comprises: searching the data source storage area according to the key value of the synchronization data, judging whether there is target data containing the key value, if not, the synchronization data is not conflict data, if yes, judging whether the data update time of the synchronization data is consistent with the data update time of the target data, if not, the synchronization data is conflict data.

3. The data processing method of claim 2, wherein, If the data update time of the synchronization data is consistent with the data update time of the target data, the synchronization data is not conflict data, and the step of storing the synchronization data to the data source storage area is not performed.

4. The data processing method according to claim 2 or 3, characterized in that, The synchronization data further comprises a timestamp, and the timestamp is added to the key value in the form of a suffix; The searching of the data source storage area according to the key value of the synchronization data comprises: searching the data source storage area according to the key value of the synchronization data after removing the timestamp suffix.

5. The data processing method according to claim 2 or 3, characterized in that, The conflict processing on all the read data comprises: merging all the data according to the data update time of all the data, and retaining the latest data.

6. The data processing method of claim 2, wherein, The method further comprises the steps of: triggering a preset asynchronous process, performing conflict processing on the data in the conflict data cache area through the preset asynchronous process, updating the data in the data source storage area according to the conflict-processed data, and deleting the conflict-processed data from the conflict data cache area.

7. The data processing method of claim 6, wherein, The preset asynchronous process is triggered at a fixed time.

8. The data processing method according to claim 6 or 7, characterized in that, The conflict processing on the data in the conflict data cache area comprises: sequentially performing the following operations on each piece of to-be-processed data in the conflict data cache area: searching the remaining data in the conflict data cache area according to the key value of the to-be-processed data, judging whether the data with the corresponding key value is searched, if not, the to-be-processed data is conflict-processed data, otherwise, performing conflict processing on the to-be-processed data and all the data with the corresponding key value searched.

9. The data processing method of claim 8, wherein, Each time a conflict-processed data is determined, the data in the data source storage area is updated according to the determined conflict-processed data; or after all the data in the conflict data cache area is conflict-processed, the data in the data source storage area is updated according to the conflict-processed data.

10. A data processing terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the computer program, realizes each step in the data processing method of any one of claims 1 to 9.

Images