A distributed database data synchronization method

Abstract

The invention discloses a distributed database data synchronization method, which relates to the technical field of data synchronization. A distributed database synchronization platform is set; files that each customer needs to synchronize are uploaded to the distributed database synchronization platform; queue information of files to be synchronized of the distributed database synchronization platform is generated; an execution priority is set for each file in the queue of files to be synchronized; library priority is set for the distributed database at the same time; time slices of different lengths are set for each distributed database according to the library priority size; each file is allocated to a distributed database with different library priorities for synchronization according to the execution priority of each file; a data synchronization exception record table is created; and abnormal data information after data synchronization is completed is recorded. The invention uses a distributed database for data synchronization, so that the resources of each distributed database middleware node are reasonably utilized, and the waiting time for user data synchronization is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a distributed database data synchronization method. Background Art

[0002] With the rapid development of the economy, society has entered the information age. The Internet has become increasingly popular, and a large amount of information and services have been applied, enriching people's lives and making people's work and life more convenient and efficient. At the same time, with the increase in the number of users and the amount of data that users need to synchronize, the increasing amount of synchronized data will bring great pressure to the operation of the database, resulting in problems such as database performance degradation and users waiting too long for data synchronization.

[0003] How to reasonably distribute the data flow passing through the database and improve the operating efficiency and quality of database data synchronization while ensuring the security of database information processing is a problem we need to solve. To this end, a distributed database data synchronization method is now provided. Summary of the invention

[0004] In order to solve the above technical problems, the object of the present invention is to provide a distributed database synchronization method.

[0005] The purpose of the present invention can be achieved by the following technical solution: A distributed database synchronization method comprises the following steps:

[0006] Step S1: Setting up a distributed database synchronization platform; uploading the files that each customer needs to synchronize to the distributed database synchronization platform, obtaining the feature identification code of each file data, and generating the queue information of the files to be synchronized of the distributed database synchronization platform;

[0007] Step S2: Set an execution priority for each file in the queue of files to be synchronized; set a library priority for the distributed database at the same time; and sort each library priority from high to low; set time slices of different lengths for each distributed database according to the library priority; assign each file to a distributed database with a different library priority for synchronization according to the execution priority of each file; when a file fails to complete data synchronization within the time slice of the distributed database with the current library priority, assign its unfinished synchronized data to a distributed database with a lower library priority for synchronization until file synchronization is completed;

[0008] Step S3: Create a data synchronization exception record table; match the feature identification code of the data after synchronization with the consistency of the data feature identification code received in the distributed database; and record the abnormal data information after data synchronization is completed according to the matching result;

[0009] Furthermore, the process of setting up the distributed database synchronization platform includes:

[0010] A plurality of middleware nodes are constructed between the client and the distributed database, and each middleware node is linked to a corresponding distributed database, and each middleware node is linked to each other; a buffer node group is arranged in the middleware node; the buffer node group includes an input buffer node, a hierarchical buffer node, a synchronization buffer node, and a verification buffer node;

[0011] Furthermore, the process of obtaining the feature identification code of each file data includes:

[0012] Enter the user's personal information, and generate a unique personal identification data segment based on the user's personal information, associate the generated unique personal identification data segment with the data information that needs to be synchronized, mark the unique personal identification data segment as a feature identification code, and send the feature identification code to other middleware nodes;

[0013] Furthermore, the process of setting an execution priority for each file in the queue of files to be synchronized includes:

[0014] Obtaining the data statistics time start field and the data statistics time end field of each queue file data in the file queue to be synchronized, and obtaining the estimated execution time according to the data body content size between the data statistics time start field and the data statistics time end field, the hierarchical buffer node obtains the waiting time of each file according to the estimated execution time of each queue file data, the hierarchical buffer node obtains user information according to the feature identification code of the file, and sets the user priority according to the user information;

[0015] Set the execution priority of each file based on its waiting time, estimated execution time, and user priority.

[0016] Furthermore, the process of setting the database priority according to the performance index of each distributed database includes:

[0017] Obtain the data processing speed of the CPU in each database and the data running memory ratio in the current time period, and obtain the data processing performance index of the distributed database based on the data processing speed of the CPU in each database and the data running memory ratio in the current time period;

[0018] And set the database priority according to the data processing performance indicators of the distributed database;

[0019] Furthermore, the process of setting time slices of different lengths for each distributed database according to the database priority size includes:

[0020] Obtain the number of middleware nodes of the distributed database, set the maximum time length of the time slice, sort the distributed databases from high priority to low priority according to the library priority of the distributed database, set the time slice for each distributed database from high priority to low priority, and obtain the time slice length of each distributed database according to the number of middleware nodes of the distributed database and the maximum time length of the time slice;

[0021] The lower the priority of the distributed database, the longer the time slice length; and the highest priority distributed database does not set the time slice;

[0022] Furthermore, when data synchronization of a file is not completed within a time slice in the current priority database, the process of allocating the incompletely synchronized data includes:

[0023] When data synchronization of a file is not completed within the time slice of the current priority distributed database, obtain the remaining data in the file that has not been synchronized, and distribute the remaining data that has not been synchronized to the distributed database of a lower priority for synchronization. If the remaining data is not synchronized within the time slice of the distributed database of a lower priority, continue to distribute the remaining data that has not been synchronized to the distributed database of a lower priority for synchronization until all the data of the file is synchronized.

[0024] Furthermore, the process of synchronizing data within a time slice in a distributed database includes:

[0025] Obtain the data ID and text content of each queue file in the queue of files to be synchronized, and simultaneously obtain the historical data ID and text content of the historical data ID of the distributed database; and compare the data ID of the file with the historical data ID of the distributed database, obtain the data ID in the file that is consistent with the historical data ID in the distributed database, mark it as the data ID to be updated, obtain the field value in the text content of the data ID to be updated that is not equal to the text content in the historical data ID of the distributed database, and update the field value in the distributed database to the field value in the data ID to be updated;

[0026] Then, obtain the data ID that does not exist in the distributed database in the file, mark it as the data ID to be synchronized, and synchronize the data ID to be synchronized to the distributed database;

[0027] Mark the data that has completed the process of updating the field value in the distributed database to the field value in the data ID to be updated and synchronizing the data ID to be synchronized to the data in the distributed database as synchronized data;

[0028] Furthermore, the process of recording data anomaly information includes:

[0029] Creating a data synchronization exception record table, matching the feature identification code of the synchronized data with the feature identification code of the distributed database;

[0030] When the characteristic identification code of the data matches the characteristic identification code of the distributed database, it is recorded as normal synchronization data;

[0031] When the feature identification code of the data does not match the feature identification code of the database, it is recorded as abnormal synchronization data.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows: the traditional single-database data synchronization method can only guarantee the fairness and throughput of data synchronization, but cannot reduce the average waiting time for user data synchronization. The present invention sets priorities for user files and distributed databases, and sets time slices for distributed databases, thereby achieving file synchronization of different users in which short tasks are executed in queues with higher priorities, while the priority of long tasks is lowered, gradually settling from the smallest high-priority time slice to the largest lowest-priority time slice, thereby reducing the average waiting time for users to perform data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of a distributed database data synchronization method of the present invention;

[0034] Figure 2 A schematic diagram of a process of setting an execution priority for each file in a queue of files to be synchronized. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0036] like Figure 1 As shown, a distributed database data synchronization method includes the following steps:

[0037] Step S1: Setting up a distributed database synchronization platform; uploading the files that each customer needs to synchronize to the distributed database synchronization platform, obtaining the feature identification code of each file data, and generating the queue information of the files to be synchronized of the distributed database synchronization platform;

[0038] Step S2: Set an execution priority for each file in the queue of files to be synchronized; set a library priority for the distributed database at the same time; and sort each library priority from high to low; set time slices of different lengths for each distributed database according to the library priority; assign each file to a distributed database with a different library priority for synchronization according to the execution priority of each file; when a file fails to complete data synchronization within the time slice of the distributed database with the current library priority, assign its unfinished synchronized data to a distributed database with a lower library priority for synchronization until file synchronization is completed;

[0039] Step S3: Create a data synchronization exception record table; match the feature identification code of the data after synchronization with the consistency of the data feature identification code received in the distributed database; and record the abnormal data information after data synchronization is completed according to the matching result;

[0040] It should be further explained that, in the specific implementation process, the construction process of the distributed database synchronization platform includes:

[0041] Several middleware nodes are constructed between the client and the distributed database; each node is linked to each other, and each node is linked to a corresponding distributed database; a buffer node group is set in the middleware node; the buffer node group includes an input buffer node, a hierarchical buffer node, a synchronization buffer node, and a verification buffer node;

[0042] It should be further explained that, in the specific implementation process, the process of obtaining the feature identification code of each file data includes:

[0043] The middleware node is provided with an entry buffer node, and the entry buffer node is used to generate an exclusive personal identification data segment according to the user's personal information, associate the generated exclusive personal identification data segment with the data information to be synchronized; and mark the exclusive personal identification data segment as a feature identification code; the middleware node sends the file data to each distributed database; and the user uploads the data information to be synchronized to the classification buffer node of the middleware node;

[0044] It should be further explained that, in the specific implementation process, before the user uploads the data information to be synchronized to the database, the user registers and logs in personal information through the data identification terminal. The specific process includes:

[0045] The input buffer node sets a registration port and a login port, and the user inputs personal information into the registration port; it should be further explained that, in the specific implementation process, the personal information includes name; gender, age; ID number for real-name authentication and registered mobile phone number; the input buffer node verifies the personal information input by the user, and generates a password port after the verification is passed; the user sets his own login password at the password port; the input buffer node saves the user's registered mobile phone number and login password; the user information enters the middleware node through the input buffer node according to the input mobile phone number and login password;

[0046] It should be further explained that, in a specific implementation process, the process of setting an execution priority for each file in the queue of files to be synchronized includes:

[0047] like Figure 2 As shown, the hierarchical buffer node obtains the data statistics time start field and the data statistics time end field of each queue file data in the file queue to be synchronized; and obtains the estimated execution time P according to the data body content size between the data statistics time start field and the data statistics time end field; the hierarchical buffer node obtains the waiting time W of each file according to the estimated execution time P of each queue file data; the hierarchical buffer node obtains the personal identification data segment; and sets the user priority K according to the user information;

[0048] When the user is an SVIP user, set the user priority K to K1.

[0049] When the user is a VIP user, set the user priority K to K2.

[0050] When the user is a common user, the user priority K is set to K3, where K1>K2>K3;

[0051] Then the execution priority = (W / P)*K;

[0052] It should be further explained that, in the specific implementation process, the process of setting the database priority according to the data processing performance index of each distributed database includes:

[0053] According to the data processing speed V of the CPU in each database and the data running memory ratio Q in the current time period, the data processing performance index P of the distributed database is set; where P = V / Q;

[0054] Obtaining a data processing performance index P of each distributed database, wherein the database priority = P;

[0055] It should be further explained that, in the specific implementation process, the process of setting time slices of different lengths for each distributed database according to the database priority size includes:

[0056] Obtain the total number of distributed database nodes N; set the maximum time length H1 of the time slice; sort the distributed databases from high priority to low priority according to the library priority of the distributed database, set a time slice for each distributed database from high priority to low priority, and set a time slice length H for each distributed database according to the library priority of the distributed database; where H = H1 / (N-(i-1)); where i = 1, 2, ..., N-1; i is the number of the distributed databases arranged in order from low to high according to the library priority;

[0057] It should be further explained that, in the specific implementation process, the distributed database with the highest database priority does not set time slices; only the file data of the distributed database is synchronized according to the first-come-first-served principle, and the next file data is processed after the synchronization of the file data is completed;

[0058] It should be further explained that, in the specific implementation process, the distributed database with the highest library priority is marked as an urgent task queue, which is used to synchronize those very important files. It will not be assigned to a distributed database with a lower priority due to the execution time of file synchronization. The file data in the urgent task queue is based on the principle of first come first served. After the time required for the synchronization of one file has passed, the synchronization of the next file will be carried out;

[0059] It should be further explained that, in a specific implementation process, when the data synchronization of a file is not completed within the time slice in the current priority database, the process of allocating the incompletely synchronized data includes:

[0060] When data synchronization of a file is not completed within the time slice of the current priority distributed database, obtain the remaining data in the file that has not been synchronized, and distribute the remaining data that has not been synchronized to the distributed database of a lower priority for synchronization. If the remaining data is not synchronized within the time slice of the distributed database of a lower priority, continue to distribute the remaining data that has not been synchronized to the distributed database of a lower priority for synchronization until all the data of the file is synchronized.

[0061] For example:

[0062] Synchronization time of files uploaded by user A T A The synchronization time of the file uploaded by user B is T B The synchronization time of the file uploaded by user C is T C30 minutes; where the execution priority is A=B=C, and C uploads file data to the distributed data synchronization platform before B, and B uploads file data to the distributed data synchronization platform before A; if the traditional first-come-first-served single database data synchronization method is used; then the waiting time required for user C's file synchronization is 0, the waiting time required for user B's file synchronization is 30 minutes, and the waiting time required for user A's file synchronization is 40 minutes;

[0063] The average waiting time required for users to complete all file synchronization is T avg1 =(0+30+40) / 3=23.33 minutes;

[0064] If the database priorities of the three distributed databases are set to be the first priority, the second priority, and the third priority respectively; the first priority is greater than the second priority, and the second priority is greater than the third priority;

[0065] The time slice length of the distributed database of the first priority is 3 minutes, the time slice length of the distributed database of the second priority is 9 minutes, and the time slice length of the distributed database of the third priority is 20 minutes; and according to the execution priorities of the files corresponding to user A, user B and user C, user A, user B and user C are assigned to the distributed database of the first priority;

[0066] The first priority distributed database sets three time slices for the files corresponding to user C, user B and user A; according to the principle of first come first served, the corresponding file data of user C is synchronized in the first time slice; the corresponding file data of user B is synchronized in the second time slice; and the corresponding file data of user A is synchronized in the third time slice;

[0067] The first-priority distributed database synchronizes the data MC1 of file C within its first time slice. If the synchronization of all the data of file C is not completed within the time slice length of 3 minutes, the remaining data MC2 of file C that has not been synchronized is allocated to the second-priority distributed database for synchronization; the second-priority distributed database sets a first time slice for the remaining data MC2; if the synchronization of the remaining data MC2 of file C is not completed within the first time slice length of 9 minutes of the second-priority distributed database, the remaining data MC3 of file C that has not been synchronized is allocated to the third-priority distributed database for synchronization; the third-priority distributed database sets a first time slice for the remaining data MC3; if the synchronization of the remaining data MC3 of file C is completed within the first time slice length of 20 minutes of the third-priority distributed database, the third-priority distributed database sends the data synchronization completion information of file C to the middleware node linked to it;

[0068] The first-priority distributed database synchronizes the data MB1 of file B within its second time slice; if the synchronization of all the data of file B is not completed within the time slice length of 3 minutes, the remaining data MB2 of file B that has not been synchronized is allocated to the second-priority distributed database for synchronization; the second-priority distributed database sets a second time slice for the remaining data MB2; if the synchronization of the remaining data MB2 of file B is completed within the second time slice length of 9 minutes of the second-priority distributed database, the second-priority distributed database sends the data synchronization completion information of file B to the middleware node connected to it;

[0069] The first-priority distributed database synchronizes the data MA1 of file A in its third time slice; if the synchronization of all the data of file A is not completed within the time slice length of 3 minutes, the first-priority distributed database sends the data synchronization completion information of file A to the middleware node connected to it;

[0070] When data synchronization is performed through the distributed database of the present invention, the waiting time required for the user C to perform data synchronization is 0 minutes; the waiting time required for the user B to perform data synchronization is 12 minutes; and the waiting time required for the user A to perform data synchronization is 6 minutes;

[0071] The average waiting time required for users to complete all file synchronization is T avg2 =(0+12+6) / 3=6 minutes;

[0072] Then T avg1 >T avg2 ;

[0073] The average waiting time of users when performing data synchronization with the distributed database is much shorter than the average waiting time of users when performing data synchronization with a single database; the efficiency of data synchronization for users is improved; and complaints caused by users having to wait too long for data synchronization are reduced;

[0074] It should be further explained that, in the specific implementation process, the process of synchronizing files uploaded to the distributed data synchronization platform includes:

[0075] Obtain the data ID and text content of each queue file in the queue of files to be synchronized, and simultaneously obtain the historical data ID and text content of the historical data ID of the distributed database; and compare the data ID of the file with the historical data ID of the distributed database, obtain the data ID in the file that is consistent with the historical data ID in the distributed database, mark it as the data ID to be updated, obtain the field value in the text content of the data ID to be updated that is not equal to the text content in the historical data ID of the distributed database, and update the field value in the distributed database to the field value in the data ID to be updated;

[0076] Then, obtain the data ID that does not exist in the distributed database in the file, mark it as the data ID to be synchronized, and synchronize the data ID to be synchronized to the distributed database;

[0077] Mark the data that has completed the process of updating the field value in the distributed database to the field value in the data ID to be updated and synchronizing the data ID to be synchronized to the data in the distributed database as synchronized data;

[0078] In the process of the middleware node processing the data to be synchronized, the method adopts updating the data first and then inserting the data. Because the data is updated first, the amount of data for the middleware node to judge whether the field value has changed is relatively small, which can reduce some unnecessary data workload of the middleware node;

[0079] It should be further explained that, in the specific implementation process, the process of recording data anomaly information includes:

[0080] The verification buffer node creates a data synchronization exception record table and reads the feature identification code of the synchronized data; matches the feature identification code of the synchronized data with the feature identification code of the distributed database;

[0081] When the feature identification code of the synchronized data matches the feature identification code of the distributed database, it is recorded as normal synchronized data, indicating that the data received by the distributed database is complete;

[0082] When the feature identification code of the synchronized data does not match the feature identification code of the database, it is recorded as abnormal synchronization data; the received data is returned along the original path, and all data residues and traces in the middleware node are obtained; and the abnormal information that occurs during the data synchronization process is recorded; then the error information is parsed to solve the cause of the data anomaly.

[0083] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A distributed database data synchronization method, characterized in that: The following steps are involved: Step S1: Setting up a distributed database synchronization platform; uploading the files that each customer needs to synchronize to the distributed database synchronization platform, obtaining the feature identification code of each file data, and generating the queue information of the files to be synchronized of the distributed database synchronization platform; Step S2: Set an execution priority for each file in the queue of files to be synchronized; set a library priority for the distributed database at the same time; and sort each library priority from high to low; set time slices of different lengths for each distributed database according to the library priority; assign each file to a distributed database with a different library priority for synchronization according to the execution priority of each file; when a file fails to complete data synchronization within the time slice of the distributed database with the current library priority, assign its unfinished synchronized data to a distributed database with a lower library priority for synchronization until file synchronization is completed; Step S3: Create a data synchronization exception record table; match the feature identification code of the data after synchronization with the consistency of the data feature identification code received in the distributed database; and record the abnormal data information after data synchronization is completed according to the matching result; A plurality of middleware nodes are constructed between the client and the distributed database, each middleware node is linked to a corresponding distributed database, and each middleware node is linked to each other, and a buffer node group is set in the middleware node, and the buffer node group includes an input buffer node, a grading buffer node, a synchronization buffer node, and a verification buffer node; The process of setting an execution priority for each file in the queue of files to be synchronized includes: The hierarchical buffer node obtains user information according to the feature identification code of the file and sets the user priority according to the user information; The execution priority of each file is set according to the waiting time, estimated execution time and user priority of each file; the hierarchical buffer node obtains the data statistics time start field and the data statistics time end field of each queue file data in the file queue to be synchronized; and obtains the estimated execution time P according to the data body content size between the data statistics time start field and the data statistics time end field; the hierarchical buffer node obtains the waiting time W of each file according to the estimated execution time P of each queue file data; the hierarchical buffer node obtains the personal identification data segment; and sets the user priority K according to the user information; Then the execution priority = (W / P)*K; The process of setting database priorities based on the performance indicators of each distributed database includes: Obtain the data processing speed V of the CPU in each database and the data running memory ratio Q in the current time period, and obtain the data processing performance index of the distributed database = V / Q according to the data processing speed V of the CPU in each database and the data running memory ratio in the current time period; And set the database priority according to the data processing performance index of the distributed database. The database priority = data processing performance index, that is, V / Q; The process of setting time slices of different lengths for each distributed database according to the database priority size includes: Obtain the number of middleware nodes of the distributed database, set the maximum time length of the time slice, sort the distributed databases from high priority to low priority according to the priority of the distributed database, set the time slice for each distributed database from high priority to low priority, and obtain the time slice length of each distributed database according to the number of middleware nodes of the distributed database and the maximum time length of the time slice; The lower the priority of the distributed database, the longer the time slice length; and the distributed database with the highest priority does not set a time slice.

2. A distributed database data synchronization method according to claim 1, characterized in that: The process of obtaining the characteristic identification code of each file data includes: Enter the user's personal information, and generate an exclusive personal identification data segment based on the user's personal information, associate the generated exclusive personal identification data segment with the data information that needs to be synchronized, mark the exclusive personal identification data segment as a feature identification code, and send the feature identification code to other middleware nodes.

3. A distributed database data synchronization method according to claim 1, characterized in that: When the data synchronization of a file is not completed within the time slice of the current priority database, the process of allocating the incompletely synchronized data includes: When data synchronization of a file is not completed within the time slice of the current priority distributed database, obtain the remaining data in the file that has not been synchronized, and distribute the remaining data that has not been synchronized to a distributed database with a lower priority for synchronization. If data synchronization of the remaining data is not completed within the time slice of a distributed database with a lower priority, continue to distribute the remaining data that has not been synchronized to a distributed database with a lower priority for synchronization until all data of the file are synchronized.

4. A distributed database data synchronization method according to claim 3, characterized in that: The process of synchronizing data within a time slice in a distributed database includes: Obtain the data ID and the text content of the data ID of each queue file in the queue of files to be synchronized, and obtain the historical data ID and the text content of the historical data ID of the distributed database; and compare the data ID of the file with the historical data ID of the distributed database, obtain the data ID in the file that is consistent with the historical data ID in the distributed database, mark it as the data ID to be updated, obtain the field value in the text content of the data ID to be updated that is not equal to the text content in the historical data ID of the distributed database, and update the field value in the distributed database to the field value in the data ID to be updated; Then, obtain the data ID that does not exist in the distributed database in the file, mark it as the data ID to be synchronized, and synchronize the data ID to be synchronized to the distributed database; The data that completes the process of updating the field value in the distributed database to the field value in the data ID to be updated and synchronizing the data ID to be synchronized to the data in the distributed database is marked as synchronized data.

5. A distributed database data synchronization method according to claim 4, characterized in that: The process of recording data anomaly information includes: A data synchronization exception record table is created, and the feature identification code of the synchronized data is matched with the feature identification code of the distributed database.

6. A distributed database data synchronization method according to claim 5, characterized in that: Further including: When the feature identification code of the data matches the feature identification code of the distributed database, it is recorded as normal synchronization data; when the feature identification code of the data does not match the feature identification code of the database, it is recorded as abnormal synchronization data.

Images