A data synchronization method, device, apparatus, and storage medium

Abstract

This application discloses a data synchronization method, apparatus, device, and storage medium. The method includes: determining a first distinguishing field combination value corresponding to first data in a source data table; determining a first hash value based on the first distinguishing field combination value; and synchronizing the first data to a target data table if the first hash value is not present among multiple second hash values. As can be seen, this application calculates the corresponding hash value using fields present in the source data table and combines this with multiple second hash values ​​corresponding to the initial data table to determine whether the first data is newly added data in the source data table, and performs corresponding data synchronization. This avoids the need for full data synchronization every time incremental data is synchronized, especially when the database itself does not support update logs and the data table has no incremental fields available, thereby improving data synchronization efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data synchronization method, apparatus, device, and storage medium. Background Technology

[0002] There are two main methods for incremental data synchronization. The first method synchronizes data based on the database's own log records. For example, MySQL synchronizes newly added and modified data to the target database by enabling binlog logging and setting it to slave mode. The second method synchronizes based on incremental fields in the data table, such as timestamps or auto-incrementing primary keys. Each synchronization records the maximum synchronized time or maximum primary key value. The next synchronization uses a synchronization statement with corresponding filtering conditions to synchronize newly added and modified data to the target database. However, if a database does not have logging enabled and the data table does not have an incremental field, it becomes difficult to synchronize incremental data to the target database.

[0003] Currently, when the database does not support update logs and the data table has no incremental fields available, incremental data synchronization is achieved by first deleting data from the target database and then synchronizing all data from the source database to the target database. This method uses full data synchronization to achieve incremental data synchronization, resulting in a long time consumption for each incremental data synchronization step and thus low data synchronization efficiency. Summary of the Invention

[0004] In view of this, embodiments of this application disclose a data synchronization method, apparatus, device, and storage medium, which avoids the need for full data synchronization every time incremental data is synchronized when the database itself does not support update logs and the data table has no incremental fields available, thereby improving the efficiency of data synchronization.

[0005] The technical solutions provided in this application are as follows:

[0006] In a first aspect, embodiments of this application provide a data synchronization method, the method comprising:

[0007] Determine the first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is the value corresponding to the distinguishing field combination of the first data, and the distinguishing field combination is obtained by concatenating the fields in the source data table;

[0008] The first hash value is determined based on the first combination of distinguishing fields;

[0009] If the first hash value is not found among the multiple second hash values, the first data is synchronized to the target data table; the multiple second hash values ​​are obtained based on the initial data table, and all data in the initial data table has been synchronized to the target data table.

[0010] In one possible implementation, the method further includes:

[0011] If the first hash value exists among the plurality of second hash values, and the first distinguishing field combination value does not exist in the deferred memory, the deferred memory is updated using the first distinguishing field combination value;

[0012] If the first hash value exists among the plurality of second hash values, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is less than or equal to the delay threshold, the number of delays is incremented by one;

[0013] If the first hash value exists among the plurality of second hash values, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is greater than the delay threshold, the target data table is updated using the first data.

[0014] In one possible implementation, updating the target data table using the first data includes:

[0015] Delete the second data in the target data table that matches the first combination of the distinguishing fields;

[0016] Synchronize the first data to the target data table.

[0017] In one possible implementation, the method further includes:

[0018] When the set time is reached, the target data table is scanned to determine the third data in the target data table whose update duration is greater than the time threshold;

[0019] If the second distinguishing field combination value corresponding to the third data does not exist in the source data table, the third data is deleted.

[0020] In one possible implementation, the method further includes:

[0021] Determine the total number of data entries for the fourth data in the initial data table;

[0022] The values ​​corresponding to each field in the initial data table are deduplicated to obtain the total number of deduplicated values ​​for each field.

[0023] Based on the total number of data entries and the total number of duplicate entries after deduplication, determine the distinguishability of each field;

[0024] Fields with a discrimination score greater than the discrimination threshold are identified as discrimination fields;

[0025] Multiple distinguishing fields are concatenated to obtain the distinguishing field combination.

[0026] In one possible implementation, concatenating multiple distinguishing fields to obtain a distinguishing field combination includes:

[0027] Based on the priority of the distinguishing fields, multiple target distinguishing fields are determined from the multiple distinguishing fields;

[0028] Based on the priority and discriminative power of the distinguishing fields, the multiple target distinguishing fields are sorted to obtain a sorting result;

[0029] Based on the sorting result, multiple target distinguishing fields are concatenated to obtain the distinguishing field combination.

[0030] In one possible implementation, the method further includes:

[0031] Determine the combination value of the third distinguishing field corresponding to the fourth data in the initial data table;

[0032] The second hash value is determined based on the combination value of the third distinguishing field;

[0033] The Bloom filter is set according to a plurality of the second hash values;

[0034] Accordingly, if the first hash value is not found among multiple second hash values, the first data is synchronized to the target data table, including:

[0035] If the first hash value is not present in the Bloom filter, the first data is synchronized to the target data table.

[0036] Secondly, embodiments of this application provide a data synchronization device, the device comprising:

[0037] The determining module is used to determine the first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is the value corresponding to the distinguishing field combination, and the distinguishing field combination is obtained by concatenating the fields in the source data table;

[0038] The determining module is further configured to determine a first hash value based on the value of the first distinguishing field combination;

[0039] The synchronization module is used to synchronize the first data to the target data table if the first hash value is not found among the multiple second hash values; the multiple second hash values ​​are obtained based on the initial data table, and all data in the initial data table has been synchronized to the target data table.

[0040] Thirdly, embodiments of this application provide a data synchronization device, including:

[0041] Memory, used to store instructions;

[0042] A processor for executing the instructions in the memory to perform the data synchronization method described in any of the first aspects above.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the data synchronization method described in any of the first aspects above.

[0044] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the data synchronization method described in any of the first aspects above.

[0045] Based on the above technical solution, this application has the following beneficial effects:

[0046] This application discloses a data synchronization method, apparatus, device, and storage medium. The method includes: determining a first distinguishing field combination value corresponding to first data in a source data table; determining a first hash value based on the first distinguishing field combination value; and synchronizing the first data to a target data table if the first hash value is not present among multiple second hash values. As can be seen, this application calculates the corresponding hash value using fields present in the source data table and combines this with multiple second hash values ​​corresponding to the initial data table to determine whether the first data is newly added data in the source data table, and performs corresponding data synchronization. This avoids the need for full data synchronization every time incremental data is synchronized, especially when the database itself does not support update logs and the data table has no incremental fields available, thereby improving data synchronization efficiency. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0048] Figure 1 This is a flowchart of a data synchronization method disclosed in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating another data synchronization method disclosed in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the source data table, target data table, and deferred memory after full data synchronization, as disclosed in an embodiment of this application.

[0051] Figure 4 This is a schematic diagram of the source data table, target data table, and deferred memory after the first data synchronization as disclosed in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the source data table, target data table, and deferred memory after a second data synchronization as disclosed in an embodiment of this application.

[0053] Figure 6 This is a schematic diagram of the source data table, target data table, and deferred memory after a third data synchronization as disclosed in an embodiment of this application.

[0054] Figure 7 This is a schematic diagram of a source data table and a target data table after performing a scanning task, as disclosed in an embodiment of this application.

[0055] Figure 8 This is a schematic diagram of the structure of a data synchronization device disclosed in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] See Figure 1 The flowchart of a data synchronization method disclosed in this application includes:

[0058] S101. Determine the first distinguishing field combination value corresponding to the first data in the source data table.

[0059] The first distinguishing field combination value is the value corresponding to the distinguishing field combination of the first data, which is obtained by concatenating the fields in the source data table.

[0060] Please refer to Table 1 below, which is a source data table disclosed in an embodiment of this application. The first data can be "185 Liu Yinan 27", "186 Chen Ernan 28", "187 Zhang Sannan 40", "188 Li Sinu 49", "189 Wang Wunu 28", "190 Zhao Liunu 61", etc. Assuming the distinguishing field combination is pattern_id_name, then the first distinguishing field combination value corresponding to the first data "185 Liu Yinan 27" is "185_Liu Yi". It should be understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0061] Table 1 Source Data Table

[0062] patient_id name sex age 185 Liu Yi male 27 186 Chen Er male 28 187 Zhang San male 40 188 Li Si female 49 189 Wang Wu female 28 190 Zhao Liu female 61 …… …… …… ……

[0063] S102. Determine the first hash value based on the combination value of the first distinguishing field.

[0064] In this embodiment, a hash algorithm can be used to calculate the hash value corresponding to the first distinguishing field combination value to obtain the first hash value. The hash algorithm can include: Message Digest (MD) series, Secure Hash Algorithm (SHA) series, and SM3 algorithm, etc.

[0065] S103. If the first hash value is not found among the multiple second hash values, synchronize the first data to the target data table.

[0066] Among them, multiple second hash values ​​are obtained from the initial data table, and the full data of the initial data table has been synchronized to the target data table.

[0067] It is understood that the initial data table is the initial source data table. In this embodiment, before executing S101, all data in the initial data table is synchronized to the target data table. The calculation method for the second hash value is similar to that for the first hash value. The source data table is stored in the source database, and the target data table is stored in the target database.

[0068] It should be noted that if the first hash value is not found among multiple second hash values, it indicates that the first data is newly added or has been modified in the source data table. In this case, the first data can be directly synchronized to the target data table, i.e., directly inserted into the target data table. The time spent determining whether a first hash value corresponding to all the first data exists among multiple second hash values ​​is far less than the time spent inserting all the first data into the target data table.

[0069] After this data synchronization is completed, embodiments of this application can add a first hash value that is not present in any of the multiple second hash values ​​to the multiple second hash values. For example, if the multiple second hash values ​​include 0001, 0010, 0011, and 0100, and the first hash value is 1000, then after this data synchronization, the multiple second hash values ​​will include 0001, 0010, 0011, 0100, and 1000. It should be understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application. It should be noted that completing this data synchronization means that multiple first data entries in the source data table have been judged and the corresponding operations have been performed.

[0070] As can be seen, in this embodiment, the corresponding hash value is calculated by the fields existing in the source data table, and combined with multiple second hash values ​​corresponding to the initial data table, to determine whether the first data is data that has been added or modified in the source data table, and to perform corresponding data synchronization. In this way, when the database itself does not support update logs and there are no incremental fields available in the data table, the full data synchronization is avoided every time incremental data is synchronized, thereby improving the efficiency of data synchronization.

[0071] In one possible implementation, the data synchronization method provided in this application embodiment further includes:

[0072] a1. If a first hash value exists among multiple second hash values, and the first distinguishing field combination value does not exist in the deferred memory, update the deferred memory using the first distinguishing field combination value.

[0073] It should be noted that if a first hash value exists among multiple second hash values, it indicates that the first data in the source data table has most likely not been modified. However, it is also possible that the values ​​corresponding to fields not in the distinguishing field combination of the first data have been modified, or that the first data is newly added data, but the hash value corresponding to the distinguishing field combination value just happens to exist among multiple second hash values. To address the situation where a first hash value exists among multiple second hash values, this application embodiment adopts a lazy synchronization method for data synchronization, combining delayed memory, delayed number of delays, and delayed thresholds to perform corresponding operations.

[0074] In this embodiment, updating the deferred memory using the first distinguishing field combination value can be achieved by adding the first distinguishing field combination value to the deferred memory and setting the deferred number corresponding to the first distinguishing field combination value to "1". The deferred memory in this embodiment can be a memory middleware.

[0075] a2. If a first hash value exists among multiple second hash values, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is less than or equal to the delay threshold, increment the number of delays by one.

[0076] The delay threshold in this embodiment can be set according to the sensitivity and timeliness of the first data in the source data table. The delay threshold can be set to 2, 3, 4, etc., and this embodiment does not limit it.

[0077] It should be noted that during a data synchronization, for the same combination of distinguishing fields, only one delay count needs to be added to the delay memory.

[0078] a3. If a first hash value exists among multiple second hash values, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is greater than the delay threshold, update the target data table using the first data.

[0079] In this embodiment of the application, if a first hash value exists among multiple second hash values, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is greater than the delay threshold, the number of delays corresponding to the first distinguishing field combination value can be directly reset in the delay memory, or the number of delays corresponding to the first distinguishing field combination value can be reset after this data synchronization is completed. Resetting the number of delays means resetting the number of delays to "0".

[0080] It is understandable that the source data table will generally include multiple first data entries. Each first data entry corresponds to the execution of S103, or the execution of a1, or the execution of a2, or the execution of a3. After the execution of S103, or the execution of a1, or the execution of a2, or the execution of a3, the judgment of the next first data entry will continue, that is, the execution of S101 will be returned.

[0081] In this embodiment, "updating the target data table using the first data" in a3 may include: deleting second data in the target data table that matches the first distinguishing field combination value; and synchronizing the first data to the target data table. Thus, if the value corresponding to a field in the first data that is not in the distinguishing field combination has been modified, or if the first data is new data but the hash value corresponding to the distinguishing field combination value happens to exist in multiple second hash values, it can be inserted into the target data table after the extension period expires, achieving delayed synchronization. Furthermore, the second data that matches the first distinguishing field combination value is deleted from the target data table, avoiding data duplication in the target data table.

[0082] In a possible implementation, in the embodiments of the present application, there may be a first hash value among multiple second hash values, and when the number of extension times corresponding to the first combination value of distinguishing fields in the extended memory is greater than the extension threshold, multiple second data that can be matched by the first combination value of distinguishing fields in the target data table are all deleted, multiple first data that can be matched by the first combination value of distinguishing fields in the source data table are inserted into the target data table, and the number of extension times is reset to "0". For example: The first combination value of distinguishing fields is "185_Liu Yi". In this data synchronization, the source data table includes "185 Liu Yi male 27", "185 Liu Yi female 27", "185 Liu Yi male 40". "185 Liu Yi male 27" is the modified first data, "185 Liu Yi female 27" is the newly added first data, and "185 Liu Yi male 40" is the unmodified first data. The target data table includes "185 Liu Yi male 26", "185 Liu Yi male 40". After judging "185 Liu Yi male 27", it is determined that the number of extension times corresponding to "185_Liu Yi" is greater than the extension threshold, then multiple second data "185 Liu Yi male 26", "185 Liu Yi male 40" in the target data table are all deleted, and multiple first data "185 Liu Yi male 27", "185 Liu Yi female 27", "185 Liu Yi male 40" in the source data are inserted into the target data table. It can be understood that the above is only an exemplary illustration and should not be construed as a limitation of the embodiments of the present application.

[0083] In a possible implementation, it is necessary to judge and process the first data in the source data table one by one. In the embodiments of the present application, when there is a first hash value among multiple second hash values and the number of extension times corresponding to the first combination value of distinguishing fields in the extended memory is greater than the extension threshold, it is first judged whether it is necessary to delete the data in the target data table. If there is no indication in the extended memory that the second data that can be matched by the first combination value of distinguishing fields in the target data table has been deleted, and the indication is like: delete_update_185_Liu Yi = true, etc., it is necessary to delete all the second data that can be matched by the first combination value of the partition fields in the target data table, insert this first data into the target data table, and set the indication in the extended memory that the second data that can be matched by the first combination value of distinguishing fields in the target data table has been deleted. If there is an indication in the extended memory that the second data that can be matched by the first combination value of distinguishing fields in the target data table has been deleted, it means that it is no longer necessary to delete the data in the target data table, and the corresponding second data has been deleted before this first data is inserted into the target data table. If it is the last first data to be processed, after processing, it is necessary to delete all the corresponding indications in the extended memory and reset all the extension times that reach the extension threshold to "0".

[0084] As can be seen, in this embodiment of the application, when a first hash value exists among multiple second hash values, a lazy synchronization method is adopted for data synchronization. The corresponding operations are performed by combining the delay memory, the number of delays, and the delay threshold. In this way, the data synchronization efficiency is improved while ensuring that the data in the source data table can be accurately synchronized to the target data table.

[0085] In one possible implementation, the data synchronization method provided in this application embodiment further includes:

[0086] b1. When the set time is reached, scan the target data table and identify the third data in the target data table whose update duration is greater than the time threshold.

[0087] It should be noted that in this embodiment, if any field in the first distinguishing field combination of the first data is modified, and the first hash value corresponding to the modified first distinguishing field combination does not exist in multiple second hash values, the first data will be directly inserted into the target data table. This will result in duplicate data in the target data table, leading to dirty data. To address the problem of dirty data, this embodiment scans the target data table and deletes the corresponding data based on the update duration and time threshold.

[0088] In this embodiment of the application, the time and time threshold can be set according to the sensitivity and timeliness of the first data in the source data table. For example, the time can be 24:00 every day, or 12:00 and 24:00 every day, etc., and the time threshold can be 24 hours, or 48 hours, etc. This embodiment of the application does not limit this.

[0089] In this embodiment of the application, an update time can be set in the target data table, and the update duration can be calculated by using the current time and the update time in the target data table.

[0090] b2. If the third data does not exist in the source data table, delete the third data.

[0091] For example, the third data is "185Liu Yinan 27" and "187Liu Yinan 27". If the second distinguishing field combination value "187_Liu Yi" corresponding to the third data does not exist in the source data table, "187Liu Yinan 27" will be deleted from the target data table. It should be understood that the above is only an exemplary illustration and should not be construed as a limitation on the embodiments of this application.

[0092] As can be seen, in this embodiment of the application, when there is dirty data in the target data table, the corresponding data is deleted by scanning the target data table and combining the update duration and time threshold. In this way, the data synchronization efficiency is improved while ensuring that the data in the source data table can be accurately synchronized to the target data table.

[0093] See Figure 2 A flowchart of another data synchronization method disclosed in the application embodiment, the method including:

[0094] S201. Determine the total number of data rows for the fourth data in the initial data table.

[0095] Please refer to Table 2 below, which is an initial data table disclosed in an embodiment of this application. The fourth data can be "185 Liu Yinan 26", "186 Chen Ernan 28", "187 Zhang Sannan 40", "188 Li Sinan 49", "189 Wang Wunu 28", and "190 Zhao Liunu 61", for a total of 6 data entries. It is understood that the above is merely illustrative and should not be construed as a limitation on the embodiments of this application.

[0096] Table 2 Initial Data Table

[0097] patient_id name sex age 185 Liu Yi male 26 186 Chen Er male 28 187 Zhang San male 40 188 Li Si female 49 189 Wang Wu female 28 190 Zhao Liu female 61

[0098] S202. Remove duplicate values ​​from each field in the initial data table to obtain the total number of duplicate values ​​for each field.

[0099] Please refer to Table 2 above. After removing duplicate values ​​for pattern_id, name, sex, and age, the total number of duplicates for pattern_id is 6, for name it is 6, for sex it is 2, and for age it is 5. It should be understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0100] S203. Determine the discrimination index of each field based on the total number of data entries and the total number of duplicate entries after deduplication.

[0101] The lower the discrimination, the greater the probability of duplicate values ​​for that field; the higher the discrimination, the less likely the value for that field is to be duplicated. Therefore, it is advisable to use fields with higher discrimination to distinguish data in the data table.

[0102] Please refer to Table 2 above. The discrimination index corresponding to pattern_id is 6 / 6 = 1, the discrimination index corresponding to name is 6 / 6 = 1, the total number of unique entries after deduplication for sex is 2 / 6 ≈ 0.333, and the total number of unique entries after deduplication for age is 5 / 6 ≈ 0.833. It should be understood that the above is only an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0103] S204. The fields corresponding to the discrimination scores that are greater than the discrimination threshold are identified as discrimination fields.

[0104] In this embodiment, the distinction threshold can be set according to the actual situation, such as a distinction threshold of 0.8, 0.9, etc., but this embodiment does not impose any restrictions on it.

[0105] Please refer to Table 2 above. Assuming a discrimination threshold of 0.8, patient_id, name, and age are determined as the discrimination fields. It is understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0106] S205. Concatenate multiple distinguishing fields to obtain a distinguishing field combination.

[0107] Please refer to Table 2 above. By concatenating patient_id, name, and age, the distinguishing field combination is obtained as patient_id_name_age. It should be understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application. Thus, in this embodiment, by concatenating multiple fields with high distinguishability to obtain a distinguishing field combination, it is easier to subsequently distinguish data in the data table based on this distinguishing field combination, thereby better achieving data synchronization.

[0108] In one possible implementation, S205 in this application may include:

[0109] c1. Determine multiple target distinguishing fields from multiple distinguishing fields based on the priority of the distinguishing fields.

[0110] Please refer to Table 2 above. In this embodiment, priorities can be set for patent_id, name, and age according to the meaning of the distinguishing fields. The priorities can be set to any value between 80 and 100, or set to not be used. This embodiment does not impose any restrictions on this. During actual data synchronization, although the current age has a distinguishability greater than the distinguishing threshold, it is certain that the distinguishability of age will decrease during subsequent incremental data synchronization. Therefore, the priority of age can be set to not be used, thus determining patent_id and name as the target distinguishing fields. It should be understood that the above is only an exemplary description and should not be construed as a limitation on the embodiments of this application.

[0111] In one possible implementation, the priority of a distinguishing field can be set based on its meaning and / or data lineage. The data lineage records which other fields and / or subject libraries reference a field; the more times a field is referenced, the more important it is, and the higher its priority can be assigned. In this embodiment, the number of times a distinguishing field is referenced can be determined through data lineage, and then combined with the meaning of the distinguishing field to assign a comprehensive priority.

[0112] c2. Sort multiple target distinguishing fields according to their priority and distinguishability to obtain the sorting results.

[0113] In this embodiment, corresponding weights can be set for the two influencing factors, priority and discrimination, such as setting both priority weight and discrimination weight to 50%. It is understood that the above weight settings are only illustrative examples, and the weight ratios can be adjusted according to the actual situation of the data table in this embodiment.

[0114] The formula for calculating the ranking weight can be as follows:

[0115] Ranking weight = Priority weight * Priority + Discrimination weight * Discrimination * 100.

[0116] Please refer to Table 2 above. Assume the multiple target distinguishing fields are pattern_id and name, with pattern_id having a priority of 95 and name having a priority of 85. Both the priority weight and the distinguishability weight are 0.5. The sorting weight corresponding to pattern_id is 0.5*95 + 0.5*1*100 = 97.5, and the sorting weight corresponding to name is 0.5*85 + 0.5*1*100 = 92.5. Therefore, the sorting result is pattern_id, name. It should be understood that the above is merely an illustrative example and should not be construed as a limitation on the embodiments of this application.

[0117] c3. Based on the sorting results, concatenate multiple target distinguishing fields to obtain a distinguishing field combination.

[0118] In this embodiment, by combining priority to determine the combination of distinguishing fields, inappropriate distinguishing fields can be further filtered out. This reduces the computational load of subsequent data synchronization while ensuring that the combination of distinguishing fields can better distinguish data in the data table. Moreover, the target distinguishing fields are sorted, providing a feasible way to obtain the combination of distinguishing fields.

[0119] S206. Determine the combined value of the third distinguishing field corresponding to the fourth data.

[0120] Please refer to Table 2 above. The third distinguishing field combination value corresponding to the fourth data "185 Liu Yinan 26" is "185_Liu Yi", the third distinguishing field combination value corresponding to the fourth data "186 Chen Ernan 28" is "186_Chen Er", the third distinguishing field combination value corresponding to the fourth data "187 Zhang Sannan 40" is "187_Zhang San", the third distinguishing field combination value corresponding to the fourth data "188 Li Sinan 49" is "188_Li Si", the third distinguishing field combination value corresponding to the fourth data "189 Wang Wunv 28" is "189_Wang Wu", and the third distinguishing field combination value corresponding to the fourth data "190 Zhao Liunv 61" is "190_Zhao Liu". It is understood that the above is only an exemplary description and should not be construed as a limitation on the embodiments of this application.

[0121] S207. Determine the second hash value based on the combination value of the third distinguishing field.

[0122] S208. Set up a Bloom filter based on multiple second hash values.

[0123] In this embodiment, multiple second hash values ​​can be stored in a Bloom filter. The Bloom filter is then used to determine whether a specific hash value exists within it. During subsequent incremental data synchronization, the corresponding hash value is adaptively added to the Bloom filter, meaning the multiple second hash values ​​stored in the Bloom filter will change. Thus, by setting up a Bloom filter, it is possible to quickly determine whether a first hash value already exists.

[0124] It should be noted that a Bloom filter is a long binary vector and a series of random mapping functions, which can be used to retrieve whether an element is in a set. Bloom filters offer significantly better space efficiency and query time than general algorithms. It is understood that other algorithms or filters can also be used for hash value storage and judgment in the embodiments of this application, and the comparison of the embodiments in this application is not limited.

[0125] S209. Synchronize all data from the initial data table to the target data table.

[0126] It should be noted that synchronizing all data from the initial data table to the target data table means synchronizing all data from the initial data table to the target data table, thus initializing the target data table.

[0127] S210. Determine the first distinguishing field combination value corresponding to the first data in the source data table.

[0128] It is understood that S210 in this embodiment is the same as S101 in the above embodiment, so it will not be described again.

[0129] S211. Determine the first hash value based on the combination value of the first distinguishing field.

[0130] It is understood that S211 in this embodiment is the same as S102 in the above embodiment, so it will not be described again.

[0131] S212. If the first hash value does not exist in the Bloom filter, synchronize the first data to the target data table.

[0132] It is understood that S212 in this embodiment is similar to S103 in the above embodiment, and therefore will not be described again.

[0133] S213. If the first hash value exists in the Bloom filter and the first distinguishing field combination value does not exist in the deferred memory, update the deferred memory using the first distinguishing field combination value.

[0134] It is understood that S213 in this embodiment is similar to a1 in the above embodiment, and therefore will not be described again.

[0135] S214. If a first hash value exists in the Bloom filter, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is less than or equal to the delay threshold, increment the number of delays by one.

[0136] It is understood that S214 in this embodiment is similar to a2 in the above embodiment, and therefore will not be described again.

[0137] S215. If a first hash value exists in the Bloom filter, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is greater than the delay threshold, update the target data table using the first data.

[0138] It is understood that S215 of this application embodiment is similar to a3 in the above embodiment, and therefore will not be described again.

[0139] As can be seen, in this embodiment of the application, by setting a distinguishing field combination and setting a Bloom filter, the corresponding hash value can be calculated by using the fields existing in the source data table during subsequent data synchronization. The Bloom filter is then used to determine whether the first data is newly added or modified data in the source data table. Combined with the number of delays and the delay threshold, the corresponding operation is executed. In this way, when the database itself does not support update logs and there are no incremental fields available in the data table, the full data synchronization is avoided every time incremental data is synchronized, thereby improving the efficiency of data synchronization.

[0140] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential for this application.

[0141] Please refer to Figures 3 to 7 , Figures 3 to 7 The data in the source data table and the data in the target data table are medical data. The data synchronization method provided by the embodiments of this application will be exemplarily described below in conjunction with Figures 3 to 7 this.

[0142] 1), After synchronizing all the data in the source data table to the target data table, the source data table, the target data table, and the deferred memory are as Figure 3 shown. The source data table is consistent with the target data table, and the values corresponding to the distinguished fields patient_id and name in the deferred memory are both empty. It should be noted that Figure 3 the source data table in

[0143] is the initial data table mentioned above Figure 4 2), After the first data synchronization, the source data table, the target data table, and the deferred memory are as

[0144] shown. Among them, the distinguished field patient_id of the data "185 Liu Yinan 26" in the source data table has been modified, and the non-distinguished field sex of the data "186 Chen Ernan 28" in the source data table has been modified. The first data synchronization was performed at 2024-3-2-07:00. Since there is no hash value corresponding to "120 Liu Yi" in the Bloom filter, the modified data "120 Liu Yinan 26" is directly inserted into the target data table, and the data "186 Chen Er 1", "187 Zhang San 1", and "188 Li Si 1" are added to the deferred memory. At this time, there is a piece of data "186 Chen Env 28" in the target data table that has not been synchronized and needs to be deferred for synchronization; there is a piece of data "185 Liu Yinan 26" that has not been deleted.

[0144] 3), After the second data synchronization, the source data table, the target data table, and the deferred memory are as Figure 5As shown. Among them, new data "188 Li Si, male, 61" and "189 Wang Wu, female, 28" are added to the source data table. The second data synchronization is performed at 2024-3-2-09:00. Since the hash value corresponding to "188 Li Si" exists in the Bloom filter, the new data "188 Li Si, male, 61" is not inserted into the target data table. Since the hash value corresponding to "189 Wang Wu" does not exist in the Bloom filter, the new data "189 Wang Wu, female, 28" is inserted into the target data table. The data "120 Liu Yi, 1" is added to the deferred memory, and the deferred times corresponding to "186 Chen Er", "187 Zhang San", and "188 Li Si" are updated to 2.

[0145] 4), After the third data synchronization, the source data table, the target data table, and the deferred memory are as Figure 6 shown. Among them, new data "190 Zhao Liu, female, 60" is added to the source data table. The third data synchronization is performed at 2024-3-2-18:00. Since the hash value corresponding to "190 Zhao Liu" does not exist in the Bloom filter, the new data "190 Zhao Liu, female, 60" is inserted into the target data table. Since the deferred times corresponding to "186 Chen Er", "187 Zhang San", and "188 Li Si" have reached the deferred threshold of 2, the data "186 Chen Er, male, 28", "187 Zhang San, male, 40", and "188 Li Si, female, 40" in the target data table are deleted, and the data "186 Chen Er, female, 28", "187 Zhang San, male, 40", "188 Li Si, female, 40", and "188 Li Si, male, 61" in the source data table are inserted into the target data table. The data "189 Wang Wu, 1" is added to the deferred memory, the deferred times corresponding to "186 Chen Er", "187 Zhang San", and "188 Li Si" are reset to 0, and the deferred times corresponding to "120 Liu Yi" are updated to 2.

[0146] 5), At 2024-3-2-24:00, a scanning task is executed. After scanning the target data table, the source data table and the target data table are as Figure 7 shown. Among them, the update duration of the data "185 Liu Yi, male, 26" in the target data table is 40 hours, which is greater than the time threshold of 24 hours. The source data table is queried, and it is determined that "185 Liu Yi" does not exist in the source data table. The data "185 Liu Yi, male, 26" is deleted from the target data table. At this time, the data in the source data table is consistent with the data in the target data table.

[0147] It can be understood that the above combination Figures 3 to 7 is only an exemplary illustration and should not be construed as a limitation on the embodiments of the present application.

[0148] See Figure 8 , The structural schematic diagram of a data synchronization device disclosed in an embodiment of the present application. The device includes:

[0149] The determining module 801 is used to determine the first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is the value corresponding to the distinguishing field combination, and the distinguishing field combination is obtained by concatenating the fields in the source data table;

[0150] The determining module 801 is further configured to determine a first hash value based on the value of the first distinguishing field combination;

[0151] The synchronization module 802 is used to synchronize the first data to the target data table if the first hash value is not found among the plurality of second hash values; the plurality of second hash values ​​are obtained based on the initial data table, and all data in the initial data table has been synchronized to the target data table.

[0152] As can be seen, in this embodiment, the corresponding hash value is calculated by the fields existing in the source data table, and combined with multiple second hash values ​​corresponding to the initial data table, to determine whether the first data is data that has been added or modified in the source data table, and to perform corresponding data synchronization. In this way, when the database itself does not support update logs and there are no incremental fields available in the data table, the full data synchronization is avoided every time incremental data is synchronized, thereby improving the efficiency of data synchronization.

[0153] In one possible implementation, the data synchronization device provided in this application embodiment further includes:

[0154] The update module is used to update the delayed memory using the first distinguishing field combination value if the first hash value exists among the plurality of second hash values ​​and the first distinguishing field combination value does not exist in the delayed memory.

[0155] The counting module is used to increment the delay count by one if the first hash value exists among the plurality of second hash values ​​and the delay count corresponding to the first distinguishing field combination value in the delay memory is less than or equal to the delay threshold.

[0156] The synchronization module is further configured to update the target data table using the first data if the first hash value exists among the plurality of second hash values ​​and the number of delays corresponding to the first distinguishing field combination value in the delay memory is greater than the delay threshold.

[0157] In one possible implementation, the synchronization module provided in this application embodiment includes:

[0158] The deletion unit is used to delete the second data in the target data table that can be matched by the first combination of distinguishing fields;

[0159] A synchronization unit is used to synchronize the first data to the target data table.

[0160] In one possible implementation, the data synchronization device provided in this application embodiment further includes:

[0161] The scanning module is used to scan the target data table when a set time is reached, and to determine the third data in the target data table whose update duration is greater than the time threshold;

[0162] The deletion module is used to delete the third data if the second distinguishing field combination value corresponding to the third data does not exist in the source data table.

[0163] In one possible implementation, the determining module in the data synchronization device provided in this application embodiment is further configured to determine the total number of data entries of the fourth data in the initial data table;

[0164] The data synchronization device also includes:

[0165] The deduplication module is used to remove duplicate values ​​from each field in the initial data table to obtain the total deduplicated value for each field.

[0166] The determination module is also used to determine the distinguishability of each field based on the total number of data entries and the total number of duplicates after deduplication;

[0167] The determination module is also used to determine the fields corresponding to discrimination scores greater than the discrimination threshold as discrimination fields;

[0168] The data synchronization device also includes:

[0169] The splicing module is used to splice multiple distinguishing fields to obtain the distinguishing field combination.

[0170] In one possible implementation, the splicing module in the data synchronization device provided in this application embodiment includes:

[0171] A determining unit is configured to determine multiple target distinguishing fields from multiple distinguishing fields based on the priority of the distinguishing fields;

[0172] A sorting unit is used to sort multiple target distinguishing fields according to their priority and distinguishability to obtain a sorting result;

[0173] The splicing unit is used to splice multiple target distinguishing fields according to the sorting result to obtain the distinguishing field combination.

[0174] In one possible implementation, the determining module in the data synchronization device provided in this application embodiment is further used to determine the third distinguishing field combination value corresponding to the fourth data in the initial data table;

[0175] The determining module is also configured to determine the second hash value based on the combined value of the third distinguishing field;

[0176] The data synchronization device also includes:

[0177] A setting unit is configured to set the Bloom filter based on a plurality of second hash values;

[0178] The synchronization module is specifically used to synchronize the first data to the target data table if the first hash value does not exist in the Bloom filter.

[0179] Furthermore, embodiments of this application also provide a data synchronization device, including:

[0180] Memory, used to store instructions;

[0181] A processor is used to execute instructions in memory to perform any of the above-described data synchronization methods.

[0182] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform any of the above-described implementations of the data synchronization method.

[0183] Furthermore, this application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute any of the above-described data synchronization methods.

[0184] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0185] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0186] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0188] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization method, characterized in that, The method includes: Determine the first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is the value corresponding to the distinguishing field combination of the first data, and the distinguishing field combination is obtained by concatenating the fields in the source data table; The first hash value is determined based on the first combination of distinguishing fields; If the first hash value is not found among the multiple second hash values, the first data is synchronized to the target data table; the multiple second hash values ​​are obtained based on the initial data table, and all data in the initial data table has been synchronized to the target data table; If the first hash value exists among the plurality of second hash values, and the first distinguishing field combination value does not exist in the deferred memory, the deferred memory is updated using the first distinguishing field combination value; If the first hash value exists among the plurality of second hash values, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is less than or equal to the delay threshold, the number of delays is incremented by one; If the first hash value exists among the plurality of second hash values, and the number of delays corresponding to the first distinguishing field combination value in the delay memory is greater than the delay threshold, delete the second data that can be matched by the first distinguishing field combination value in the target data table, and synchronize the first data to the target data table; When the set time is reached, the target data table is scanned to determine the third data in the target data table whose update duration is greater than the time threshold; If the second distinguishing field combination value corresponding to the third data does not exist in the source data table, the third data is deleted.

2. The method according to claim 1, characterized in that, The method further includes: Determine the total number of data entries for the fourth data in the initial data table; The values ​​corresponding to each field in the initial data table are deduplicated to obtain the total number of deduplicated values ​​for each field. Based on the total number of data entries and the total number of duplicate entries after deduplication, determine the distinguishability of each field; Fields with a discrimination score greater than the discrimination threshold are identified as discrimination fields; Multiple distinguishing fields are concatenated to obtain the distinguishing field combination.

3. The method according to claim 2, characterized in that, The concatenation of multiple distinguishing fields to obtain a distinguishing field combination includes: Based on the priority of the distinguishing fields, multiple target distinguishing fields are determined from the multiple distinguishing fields; Based on the priority and discriminative power of the distinguishing fields, the multiple target distinguishing fields are sorted to obtain a sorting result; Based on the sorting result, multiple target distinguishing fields are concatenated to obtain the distinguishing field combination.

4. The method according to claim 1, characterized in that, The method further includes: Determine the combination value of the third distinguishing field corresponding to the fourth data in the initial data table; The second hash value is determined based on the combination value of the third distinguishing field; Set a Bloom filter based on multiple second hash values; Accordingly, if the first hash value is not found among multiple second hash values, the first data is synchronized to the target data table, including: If the first hash value does not exist in the Bloom filter, the first data is synchronized to the target data table.

5. A data synchronization device, characterized in that, The device includes: The determining module is used to determine the first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is the value corresponding to the distinguishing field combination, and the distinguishing field combination is obtained by concatenating the fields in the source data table; The determining module is further configured to determine a first hash value based on the value of the first distinguishing field combination; A synchronization module is used to synchronize the first data to a target data table if the first hash value is not found among a plurality of second hash values; the plurality of second hash values ​​are obtained based on an initial data table, and all data in the initial data table has been synchronized to the target data table; The update module is used to update the delayed memory using the first distinguishing field combination value if the first hash value exists among the plurality of second hash values ​​and the first distinguishing field combination value does not exist in the delayed memory. The counting module is used to increment the delay count by one if the first hash value exists among the plurality of second hash values ​​and the delay count corresponding to the first distinguishing field combination value in the delay memory is less than or equal to the delay threshold. The synchronization module is further configured to, if the first hash value exists among the plurality of second hash values ​​and the number of delays corresponding to the first distinguishing field combination value in the delay memory is greater than the delay threshold, delete the second data that can be matched by the first distinguishing field combination value in the target data table and synchronize the first data to the target data table; The scanning module is used to scan the target data table when a set time is reached, and to determine the third data in the target data table whose update duration is greater than the time threshold; The deletion module is used to delete the third data if the second distinguishing field combination value corresponding to the third data does not exist in the source data table.

6. A data synchronization device, characterized in that, include: Memory, used to store instructions; A processor for executing the instructions in the memory to perform the data synchronization method according to any one of claims 1 to 4.

7. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the data synchronization method according to any one of claims 1 to 4.

Images