Data synchronization method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data synchronization method and device, equipment and a storage medium. The method comprises the following steps: determining a first distinguishing field combination value corresponding to first data in a source data table; determining a first hash value according to the first distinguishing field combination value; and if the first hash value does not exist in the plurality of second hash values, synchronizing the first data to the target data table. It can be seen that in the embodiment of the invention, the corresponding hash values are calculated through the fields existing in the source data table, whether the first data are newly added data in the source data table or not is determined in combination with the multiple second hash values corresponding to the initial data table, and corresponding data synchronization is carried out, so that log updating is not supported when the database itself does not support log updating; and under the condition that no incremental field exists in the data table and the data table can be used, the situation that full-amount data synchronization needs to be carried out every time incremental data synchronization is carried out is avoided, so that the data synchronization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data synchronization method, apparatus, device and storage medium. Background Art

[0002] There are two main methods for incremental data synchronization. The first method is to synchronize data based on the log records of the database itself, such as MySQL, which can synchronize the newly added and changed data to the target database by opening the binlog log and setting the slave mode; the second method is to synchronize based on the incremental fields in the data table, such as based on timestamps or auto-increment primary keys, etc. Each time synchronization is performed, the maximum time or maximum primary key value that has been synchronized is recorded. The next time synchronization is performed, the newly added and changed data is synchronized to the target database using the synchronization statement plus the corresponding filtering conditions. However, if a database does not have a log turned on and there is no incremental field in the data table, it is difficult to synchronize the incremental data to the target database.

[0003] Currently, when the database does not support update logs and the data table has no incremental fields to use, the target database data is deleted first, and then the full data of the source database is synchronized to the target database. This is to achieve incremental data synchronization through full data synchronization, resulting in a long time for each incremental data synchronization, resulting in low data synchronization efficiency. Summary of the invention

[0004] In view of this, the embodiments of the present application disclose a data synchronization method, apparatus, device and storage medium, which can avoid full data synchronization every time incremental data is synchronized when the database itself does not support update logs and the data table has no incremental fields that can be used, thereby improving the efficiency of data synchronization.

[0005] The technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a data synchronization method, the method comprising:

[0007] Determine a first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is a value corresponding to the distinguishing field combination of the first data, and the distinguishing field combination is obtained by concatenating the fields in the source data table;

[0008] Determine a first Hash value according to the first distinguishing field combination value;

[0009] If the first hash value does not exist in the multiple second hash values, the first data is synchronized to the target data table; the multiple second hash values ​​are obtained according to the initial data table, and the full amount of data in the initial data table has been synchronized to the target data table.

[0010] In a possible implementation, the method further includes:

[0011] If the first Hash value exists in the plurality of second Hash values ​​and the first distinguishing field combination value does not exist in the deferred memory, updating the deferred memory with the first distinguishing field combination value;

[0012] If the first Hash value exists in the plurality of second Hash values, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is less than or equal to the extension threshold, the number of extensions is increased by one;

[0013] If the first hash value exists in the plurality of second hash values, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is greater than the extension threshold, the target data table is updated using the first data.

[0014] In a possible implementation manner, the updating the target data table by using the first data includes:

[0015] Deleting the second data in the target data table that can be matched by the first distinguishing field combination value;

[0016] The first data is synchronized to the target data table.

[0017] In a possible implementation, the method further includes:

[0018] When the set time is reached, scanning the target data table to determine third data in the target data table whose update time is greater than the time threshold;

[0019] If the second distinguishing field combination value corresponding to the third data does not exist in the source data table, the third data is deleted.

[0020] In a possible implementation, the method further includes:

[0021] Determine the total number of data items of the fourth data in the initial data table;

[0022] Deduplicating the values ​​corresponding to each field in the initial data table to obtain the deduplicated total amount corresponding to each field;

[0023] Determine the discrimination degree corresponding to each field according to the total number of data items and the total amount after deduplication;

[0024] The field corresponding to the degree of distinction greater than the distinction threshold is determined as the distinction field;

[0025] A plurality of the distinguishing fields are concatenated to obtain the distinguishing field combination.

[0026] In a possible implementation, the concatenating the multiple distinguishing fields to obtain a distinguishing field combination includes:

[0027] Determining a plurality of target distinguishing fields from a plurality of the distinguishing fields according to the priorities of the distinguishing fields;

[0028] Sorting the plurality of target distinguishing fields according to the priority and the degree of distinction of the distinguishing fields to obtain a sorting result;

[0029] According to the sorting result, a plurality of the target distinguishing fields are concatenated to obtain the distinguishing field combination.

[0030] In a possible implementation, the method further includes:

[0031] Determine a third distinguishing field combination value corresponding to the fourth data in the initial data table;

[0032] Determine a second Hash value according to the third distinguishing field combination value;

[0033] Setting the Bloom filter according to the plurality of the second hash values;

[0034] Correspondingly, if the first hash value does not exist in the plurality of second hash values, synchronizing the first data to the target data table includes:

[0035] If the first hash value does not exist in the Bloom filter, the first data is synchronized to the target data table.

[0036] In a second aspect, an embodiment of the present application provides a data synchronization device, the device comprising:

[0037] A determination module, used to determine a first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is a value corresponding to the distinguishing field combination, and the distinguishing field combination is obtained by concatenating the fields in the source data table;

[0038] The determination module is further used to determine a first Hash value according to the value of the first distinguishing field combination;

[0039] A synchronization module is used to synchronize the first data to a target data table if the first hash value does not exist in multiple second hash values; the multiple second hash values ​​are obtained based on an initial data table, and all data in the initial data table has been synchronized to the target data table.

[0040] In a third aspect, an embodiment of the present application provides a data synchronization device, including:

[0041] A memory for storing instructions;

[0042] A processor is used to execute the instructions in the memory to perform the data synchronization method described in any one of the first aspects above.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the data synchronization method described in any one of the first aspects above.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the data synchronization method described in any one of the first aspects above.

[0045] Based on the above technical solution, this application has the following beneficial effects:

[0046] The embodiment of the present application discloses a data synchronization method, device, equipment and storage medium. Among them, the method includes: determining the first distinguishing field combination value corresponding to the first data in the source data table; determining the first hash value according to the first distinguishing field combination value; if the first hash value does not exist in the multiple second hash values, synchronizing the first data to the target data table. It can be seen that in the embodiment of the present application, the corresponding hash value is calculated by the field existing in the source data table, and combined with the multiple second hash values ​​corresponding to the initial data table, it is determined whether the first data is the newly added data in the source data table, and the corresponding data synchronization is performed. In this way, when the database itself does not support update logs and the data table has no incremental fields to use, it is avoided that the full data synchronization is required every time the incremental data is synchronized, thereby improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying any creative work.

[0048] Figure 1 A flowchart of a data synchronization method disclosed in an embodiment of the present application;

[0049] Figure 2 A flowchart of another data synchronization method disclosed in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a source data table, a target data table, and a deferred memory after full data synchronization disclosed in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a source data table, a target data table, and a deferred memory after a first data synchronization disclosed in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a source data table, a target data table and a deferred memory after a second data synchronization disclosed in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of a source data table, a target data table and a deferred memory after a third data synchronization disclosed in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of a source data table and a target data table after executing a scanning task disclosed in an embodiment of the present application;

[0055] Figure 8 A schematic diagram of the structure of a data synchronization device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] See also Figure 1 , a flow chart of a data synchronization method disclosed in an embodiment of the present application, the method comprising:

[0058] S101: Determine a first distinguishing field combination value corresponding to first data in a source data table.

[0059] The first distinguishing field combination value is a value corresponding to the distinguishing field combination of the first data, and the distinguishing field combination is obtained by concatenating fields in the source data table.

[0060] Please refer to Table 1 below, which is a source data table disclosed in the embodiment of the present application. Among them, the first data can be "185 Liu Yi male 27", "186 Chen Er male 28", "187 Zhang San male 40", "188 Li Si female 49", "189 Wang Wu female 28", "190 Zhao Liu female 61", etc. Assuming that the distinguishing field combination is patient_id_name, the first distinguishing field combination value corresponding to the first data "185 Liu Yi male 27" is "185_Liu Yi". It can be understood that the above is only an exemplary description and should not be understood as a limitation to the embodiments of the present application.

[0061] Table 1 Source data table

[0062] patient_id name sex age 185 Liu Yi male 27 186 Chen Er male 28 187 Zhang San male 40 188 Li Si female 49 189 Wang Wu female 28 190 Zhao Liu female 61 …… …… …… ……

[0063] S102: Determine a first hash value according to a first distinguishing field combination value.

[0064] In the embodiment of the present application, a hash algorithm can be used to calculate the hash value corresponding to the first distinguishing field combination value to obtain the first hash value. The hash algorithm may include: Message Digest (MD) series, Secure Hash Algor ithm (SHA) series, SM3 algorithm, etc.

[0065] S103: If the first hash value does not exist in the plurality of second hash values, synchronize the first data to the target data table.

[0066] Among them, multiple second hash values ​​are obtained according to the initial data table, and the full amount of data in the initial data table has been synchronized to the target data table.

[0067] It is understandable that the initial data table is the initial source data table, and the embodiment of the present application synchronizes the full amount of data in the initial data table to the target data table before executing S101. The calculation method of the second hash value is similar to the calculation method of the first hash value. Among them, the source data table is stored in the source database, and the target data table is stored in the target database.

[0068] It should be noted that if the first hash value does not exist in the multiple second hash values, it means that the first data is newly added data or the first data has been modified in the source data table, and the first data can be directly synchronized to the target data table, that is, the first data can be directly inserted into the target data table. The time spent on determining whether the first hash values ​​corresponding to all the first data exist in the multiple second hash values ​​is much less than the time spent on inserting all the first data into the target data table.

[0069] After executing this data synchronization, the embodiment of the present application can add the first hash value that does not exist in the multiple second hash values ​​to the multiple second hash values. For example: the multiple second hash values ​​include: 0001, 0010, 0011, 0100, and the first hash value is 1000. After executing this data synchronization, the multiple second hash values ​​include: 0001, 0010, 0011, 0100, 1000. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiment of the present application. It should be noted that the execution of this data synchronization means that the multiple first data in the source data table have been judged and the corresponding operations have been performed.

[0070] It can be seen that in the embodiment of the present application, the corresponding hash value is calculated through the fields existing in the source data table, and combined with the multiple second hash values ​​corresponding to the initial data table to determine whether the first data is newly added or modified data in the source data table, and the corresponding data synchronization is performed. In this way, when the database itself does not support update logs and the data table has no incremental fields to use, it is possible to avoid full data synchronization every time incremental data is synchronized, thereby improving the efficiency of data synchronization.

[0071] In a possible implementation, the data synchronization method provided in the embodiment of the present application further includes:

[0072] a1. If the first hash value exists in the plurality of second hash values ​​and the first distinguishing field combination value does not exist in the deferred memory, the deferred memory is updated using the first distinguishing field combination value.

[0073] It should be noted that if the first hash value exists in multiple second hash values, it means that the first data has not been modified in the source data table, but it is also possible that the value corresponding to the field in the first data that is not in the distinguishing field combination has been modified, or it is also possible that the first data is newly added data, but the hash value corresponding to the distinguishing field combination value happens to exist in multiple second hash values. In the case where the first hash value exists in multiple second hash values, the embodiment of the present application adopts lazy synchronization to synchronize data, and performs corresponding operations in combination with the deferred memory, the number of deferred times and the deferred threshold.

[0074] In the embodiment of the present application, the first distinguishing field combination value is used to update the deferred memory, which may be to add the first distinguishing field combination value to the deferred memory and set the number of deferred times corresponding to the first distinguishing field combination value to "1". The deferred memory in the embodiment of the present application may be a memory middleware.

[0075] a2. If the first hash value exists among the multiple second hash values, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is less than or equal to the extension threshold, the number of extensions is increased by one.

[0076] The extension threshold in the embodiment of the present application can be set according to the sensitivity and timeliness of the first data in the source data table. The extension threshold can be set to 2, 3, 4, etc., and the embodiment of the present application does not impose any limitation on this.

[0077] It should be noted that, in one data synchronization, for the same distinguishing field combination, only one corresponding extension number needs to be added in the extension memory.

[0078] a3. If the first hash value exists among the plurality of second hash values, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is greater than the extension threshold, the target data table is updated using the first data.

[0079] In an embodiment of the present application, if the first hash value exists among multiple second hash values, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is greater than the extension threshold, the number of extensions corresponding to the first distinguishing field combination value can also be directly reset in the extension memory, or the number of extensions corresponding to the first distinguishing field combination value can be reset after the data synchronization is completed. Resetting the extension number means resetting the extension number to "0".

[0080] It is understandable that a general source data table will include multiple first data, and one first data corresponds to the execution of S103, or corresponds to the execution of a1, or corresponds to the execution of a2, or corresponds to the execution of a3. After executing S103, or executing a1, or executing a2, or executing a3, the judgment of the next first data will continue, that is, return to execute S101.

[0081] In the embodiment of the present application, "updating the target data table with the first data" in a3 may include: deleting the second data that can be matched by the first distinguishing field combination value in the target data table; synchronizing the first data to the target data table. In this way, in the case where the value corresponding to the field that is not in the distinguishing field combination in the first data has been modified, or the first data is newly added data, but the hash value corresponding to the distinguishing field combination value happens to exist in multiple second hash values, it can be inserted into the target data table after the extension period expires to achieve extended synchronization, and the second data that can be matched by the first distinguishing field combination value is correspondingly deleted in the target data table, avoiding data duplication in the target data table.

[0082] In a possible implementation, in the embodiments of the present application, when there is a first hash value among multiple second hash values, and the number of deferral times corresponding to the first combination value of distinguishing fields in the deferred memory is greater than the deferral threshold, multiple second data that can be matched by the first combination value of distinguishing fields in the target data table are all deleted, multiple first data that can be matched by the first combination value of distinguishing fields in the source data table are all inserted into the target data table, and the number of deferral times is reset to "0". For example: the first combination value of distinguishing fields is "185_Liu Yi". In this data synchronization, the source data table includes "185 Liu Yi male 27", "185 Liu Yi female 27", "185 Liu Yi male 40". "185 Liu Yi male 27" is the modified first data, "185 Liu Yi female 27" is the newly added first data, and "185 Liu Yi male 40" is the unmodified first data. The target data table includes "185 Liu Yi male 26", "185 Liu Yi male 40". After judging "185 Liu Yi male 27", it is determined that the number of deferral times corresponding to "185_Liu Yi" is greater than the deferral threshold, then multiple second data "185 Liu Yi male 26", "185 Liu Yi male 40" in the target data table are all deleted, and multiple first data "185 Liu Yi male 27", "185 Liu Yi female 27", "185 Liu Yi male 40" in the source data are all inserted into the target data table. It can be understood that the above is only an exemplary illustration and should not be construed as a limitation of the embodiments of the present application.

[0083] In a possible implementation, it is necessary to judge and process the first data in the source data table one by one. In the embodiments of the present application, when there is a first hash value among multiple second hash values, and the number of deferral times corresponding to the first combination value of distinguishing fields in the deferred memory is greater than the deferral threshold, it is first judged whether it is necessary to delete the data in the target data table. If there is no indication in the deferred memory that the second data that can be matched by the first combination value of distinguishing fields in the target data table has been deleted, such as indication: delete_update_185_Liu Yi = true, etc., it is necessary to delete all the second data that can be matched by the first combination value of the partition fields in the target data table, insert this first data into the target data table, and set the indication in the deferred memory that the second data that can be matched by the first combination value of distinguishing fields in the target data table has been deleted. If there is an indication in the deferred memory that the second data that can be matched by the first combination value of distinguishing fields in the target data table has been deleted, it means that it is no longer necessary to delete the data in the target data table. The corresponding second data has been deleted before this first data is inserted into the target data table. If it is the last first data to be processed, after processing, it is necessary to delete all the corresponding indications in the deferred memory and reset all the number of deferral times that reach the deferral threshold to "0".

[0084] It can be seen that in the embodiment of the present application, for the situation where there is a first hash value among multiple second hash values, an inert synchronization method is adopted to perform data synchronization, and corresponding operations are performed in combination with the deferred memory, the number of deferred times and the deferred threshold. This improves the data synchronization efficiency while ensuring that the data in the source data table can be accurately synchronized to the target data table.

[0085] In a possible implementation, the data synchronization method provided in the embodiment of the present application further includes:

[0086] b1. When the set time is reached, the target data table is scanned to determine the third data in the target data table whose update time is greater than the time threshold.

[0087] It should be noted that, in the embodiment of the present application, if any field in the first distinguishing field combination of the first data is modified, and the first hash value corresponding to the modified first distinguishing field combination does not exist in the multiple second hash values, the first data will be directly inserted into the target data table, which will cause data duplication in the target data table and dirty data. To address the problem of dirty data, the embodiment of the present application deletes the corresponding data by scanning the target data table and combining the update duration and time threshold.

[0088] In the embodiment of the present application, the set time and time threshold can be set according to the sensitivity and timeliness of the first data in the source data table. For example, the set time can be 24:00 every day, or 12:00 and 24:00 every day, and the time threshold can be 24 hours, or 48 hours, and the embodiment of the present application does not limit this.

[0089] In the embodiment of the present application, the update time can be set in the target data table, and the update duration can be calculated by the current time and the update time in the target data table.

[0090] b2. If the second distinguishing field combination value corresponding to the third data does not exist in the source data table, delete the third data.

[0091] For example, the third data is "185 Liu Yi male 27", "187 Liu Yi female 27". If the second distinguishing field combination value "187_Liu Yi" corresponding to the third data does not exist in the source data table, "187 Liu Yi female 27" is deleted in the target data table. It can be understood that the above is only an exemplary description and should not be construed as a limitation on the embodiments of the present application.

[0092] It can be seen that in the embodiment of the present application, in the case where there is dirty data in the target data table, the target data table is scanned, and the corresponding data is deleted in combination with the update duration and time threshold. This improves the data synchronization efficiency while ensuring that the data in the source data table can be accurately synchronized to the target data table.

[0093] See also Figure 2 , a flow chart of another data synchronization method disclosed in an embodiment of the application, the method comprising:

[0094] S201: Determine the total number of data items of the fourth data in the initial data table.

[0095] Please refer to Table 2 below, which is an initial data table disclosed in the embodiment of the present application. Among them, the fourth data can be "185 Liu Yi male 26", "186 Chen Er male 28", "187 Zhang San male 40", "188 Li Si male 49", "189 Wang Wu female 28", "190 Zhao Liu female 61", and the total number of data items is 6. It can be understood that the above is only an exemplary description and should not be understood as a limitation to the embodiments of the present application.

[0096] Table 2 Initial data table

[0097] patient_id name sex age 185 Liu Yi male 26 186 Chen Er male 28 187 Zhang San male 40 188 Li Si female 49 189 Wang Wu female 28 190 Zhao Liu female 61

[0098] S202: De-duplicate the values ​​corresponding to each field in the initial data table to obtain the de-duplicate total amount corresponding to each field.

[0099] Please continue to refer to the above Table 2. After removing the values ​​corresponding to patient_id, name, sex, and age, the total amount after removing the duplicates corresponding to patient_id is 6, the total amount after removing the duplicates corresponding to name is 6, the total amount after removing the duplicates corresponding to sex is 2, and the total amount after removing the duplicates corresponding to age is 5. It can be understood that the above is only an exemplary description and should not be construed as a limitation on the embodiments of the present application.

[0100] S203: Determine the discrimination degree corresponding to each field according to the total number of data items and the total amount after deduplication.

[0101] Among them, the smaller the discrimination, the greater the possibility that the value corresponding to the field is repeated; the larger the discrimination, the smaller the possibility that the value corresponding to the field is repeated. You can consider using a field with a larger discrimination to distinguish the data in the data table.

[0102] Please continue to refer to the above Table 2, the discrimination degree corresponding to patient_id is 6 / 6=1, the discrimination degree corresponding to name is 6 / 6=1, the total amount after deduplication corresponding to sex is 2 / 6≈0.333, and the total amount after deduplication corresponding to age is 5 / 6≈0.833. It can be understood that the above is only an exemplary description and should not be construed as limiting the embodiments of the present application.

[0103] S204: Determine the field corresponding to the degree of distinction greater than the distinction threshold as a distinguishing field.

[0104] In the embodiment of the present application, the distinction threshold can be set according to actual conditions, such as the distinction threshold is 0.8, 0.9, etc., and there is no limitation on this in the embodiment of the present application.

[0105] Please continue to refer to the above Table 2, assuming that the distinction threshold is 0.8, and determine patient_id, name, and age as distinguishing fields. It can be understood that the above is only an exemplary description and should not be construed as limiting the embodiments of the present application.

[0106] S205: Concatenate multiple distinguishing fields to obtain a distinguishing field combination.

[0107] Please continue to refer to Table 2 above, and concatenate patient_id, name, and age to obtain a distinguishing field combination of patient_id_name_age. It is understandable that the above is only an exemplary description and should not be construed as a limitation on the embodiments of the present application. In this way, in the embodiments of the present application, a distinguishing field combination is obtained by concatenating multiple fields with a large degree of distinction, which facilitates the subsequent distinction of data in the data table based on the distinguishing field combination, thereby better realizing data synchronization.

[0108] In a possible implementation method, S205 in the implementation of this application may include:

[0109] c1. According to the priorities of the distinguishing fields, multiple target distinguishing fields are determined from the multiple distinguishing fields.

[0110] Please continue to refer to Table 2 above. In the embodiment of the present application, the corresponding priority can be set for patient_id, name, and age according to the meaning of the distinguishing fields. The priority can be set to any value between 80 and 100, or set to not be used. The embodiment of the present application does not limit this. During actual data synchronization, although the current discrimination of age is greater than the discrimination threshold, during subsequent incremental data synchronization, it can be determined that the discrimination of age will become smaller and smaller. Therefore, the priority of age can be set to not be used, and patient_id and name are determined as the target distinguishing fields. It can be understood that the above is only an exemplary description and should not be understood as a limitation on the embodiments of the present application.

[0111] In a possible implementation, in an embodiment of the present application, the priority of the distinguishing field can also be set according to the meaning of the distinguishing field and / or the data lineage relationship. Among them, the data lineage relationship records which other fields the field is referenced by, and / or which subject libraries it is referenced by. The more times it is referenced, the more important the field is, and the higher the corresponding priority can be set. In an embodiment of the present application, the number of times the distinguishing field is referenced can be determined by the data lineage relationship, and then a priority can be comprehensively given in combination with the meaning of the distinguishing field.

[0112] c2. Sort multiple target distinguishing fields according to the priority and distinction of the distinguishing fields to obtain a sorting result.

[0113] In the embodiment of the present application, corresponding weights can be set for the two influencing factors of priority and discrimination, such as setting the priority weight and the discrimination weight to 50%. It can be understood that the above weight setting is only an exemplary description, and the weight ratio can be adjusted according to the actual situation of the data table in the embodiment of the present application.

[0114] Among them, the calculation formula of the sorting weight can be as follows:

[0115] Sorting weight = priority weight * priority + discrimination weight * discrimination * 100.

[0116] Please continue to refer to Table 2 above. Assume that the multiple target distinguishing fields are patient_id and name, the priority of patient_id is 95, the priority of name is 85, and the priority weight and the discrimination weight are both 0.5. The sorting weight corresponding to patient_id is 0.5*95+0.5*1*100=97.5, and the sorting weight corresponding to name is 0.5*85+0.5*1*100=92.5, and the sorting result is patient_id, name. It can be understood that the above is only an exemplary description and should not be construed as a limitation on the embodiments of the present application.

[0117] c3. According to the sorting results, multiple target distinguishing fields are concatenated to obtain a distinguishing field combination.

[0118] In the embodiment of the present application, by combining priorities to determine the distinguishing field combination, inappropriate distinguishing fields can be further filtered out, thereby reducing the amount of calculation for subsequent data synchronization while ensuring that the distinguishing field combination can better distinguish the data in the data table. Moreover, the target distinguishing fields are sorted, providing a feasible way to obtain the distinguishing field combination.

[0119] S206: Determine a third distinguishing field combination value corresponding to the fourth data.

[0120] Please refer to the above Table 2, the third distinguishing field combination value corresponding to the fourth data "185 Liu Yi Male 26" is "185_Liu Yi", the third distinguishing field combination value corresponding to the fourth data "186 Chen Er Male 28" is "186_Chen Er", the third distinguishing field combination value corresponding to the fourth data "187 Zhang San Male 40" is "187_Zhang San", the third distinguishing field combination value corresponding to the fourth data "188 Li Si Male 49" is "188_Li Si", the third distinguishing field combination value corresponding to the fourth data "189 Wang Wu Female 28" is "189_Wang Wu", and the third distinguishing field combination value corresponding to the fourth data "190 Zhao Liu Female 61" is "190_Zhao Liu". It can be understood that the above is only an exemplary description and should not be understood as a limitation to the embodiments of the present application.

[0121] S207: Determine a second Hash value according to the third distinguishing field combination value.

[0122] S208. Set a Bloom filter according to the multiple second hash values.

[0123] In the embodiment of the present application, multiple second hash values ​​can be saved in the Bloom filter, and the Bloom filter is used to determine whether a certain hash value exists in the Bloom filter. In the subsequent incremental data synchronization, the corresponding hash value will be adaptively added to the Bloom filter, that is, the multiple second hash values ​​saved in the Bloom filter will change. In this way, by setting the Bloom filter, it is possible to quickly determine whether the first hash value already exists in the subsequent process.

[0124] It should be noted that a Bloom filter is a very long binary vector and a series of random mapping functions, which can be used to retrieve whether an element is in a set. The space efficiency and query time of a Bloom filter are much better than those of general algorithms. It is understandable that the embodiments of the present application may also use other algorithms or filters to store and judge hash values, and the embodiments of the present application are not limited in comparison.

[0125] S209: Synchronize all the data in the initial data table to the target data table.

[0126] It should be noted that synchronizing the full amount of data in the initial data table to the target data table means synchronizing all the data in the initial data table to the target data table to initialize the target data table.

[0127] S210: Determine a first distinguishing field combination value corresponding to first data in a source data table.

[0128] It can be understood that S210 of the embodiment of the present application is the same as S101 in the above embodiment, so it is not described again.

[0129] S211. Determine a first hash value according to a first distinguishing field combination value.

[0130] It can be understood that S211 of the embodiment of the present application is the same as S102 in the above embodiment, so it is not repeated here.

[0131] S212: If the first hash value does not exist in the Bloom filter, synchronize the first data to the target data table.

[0132] It can be understood that S212 of the embodiment of the present application is similar to S103 in the above embodiment, so it is not repeated here.

[0133] S213: If the first hash value exists in the Bloom filter and the first distinguishing field combination value does not exist in the deferred memory, update the deferred memory using the first distinguishing field combination value.

[0134] It can be understood that S213 of the embodiment of the present application is similar to a1 in the above embodiment, so it will not be repeated here.

[0135] S214: If the first hash value exists in the Bloom filter, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is less than or equal to the extension threshold, increase the number of extensions by one.

[0136] It can be understood that S214 of the embodiment of the present application is similar to a2 in the above embodiment, so it will not be repeated here.

[0137] S215: If the first hash value exists in the Bloom filter, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is greater than the extension threshold, update the target data table using the first data.

[0138] It can be understood that S215 of the embodiment of the present application is similar to a3 in the above embodiment, so it will not be repeated here.

[0139] It can be seen that in the embodiment of the present application, by setting a distinguishing field combination and setting a Bloom filter, when subsequent data synchronization is performed, the corresponding hash value can be calculated through the fields existing in the source data table, and the Bloom filter is used to determine whether the first data is newly added or modified data in the source data table, and then combined with the number of extensions and the extension threshold judgment, the corresponding operation is performed. In this way, when the database itself does not support update logs and the data table has no incremental fields to use, it is possible to avoid full data synchronization every time incremental data is synchronized, thereby improving the efficiency of data synchronization.

[0140] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0141] Please refer to Figures 3 to 7 , Figures 3 to 7 The data in the source data table and the data in the target data table are medical data. The following will give an exemplary description of the data synchronization method provided by the embodiments of this application in combination with Figures 3 to 7 this.

[0142] 1), After synchronizing all the data in the source data table to the target data table, the source data table, the target data table, and the deferred memory are as Figure 3 shown. The source data table is consistent with the target data table, and the values corresponding to the distinguishing fields patient_id and name in the deferred memory are both empty. It should be noted that Figure 3 the source data table in

[0143] is the above-mentioned initial data table Figure 4 2), After the first data synchronization, the source data table, the target data table, and the deferred memory are as

[0144] shown. Among them, the distinguishing field patient_id of the data "185 Liu Yinan 26" in the source data table has been modified, and the non-distinguishing field sex of the data "186 Chen Ernan 28" in the source data table has been modified. The first data synchronization is performed at 2024-3-2-07:00. Since the hash value corresponding to "120 Liu Yi" does not exist in the Bloom filter, the modified data "120 Liu Yinan 26" is directly inserted into the target data table, and the data "186 Chen Er 1", "187 Zhang San 1", and "188 Li Si 1" are added to the deferred memory. At this time, there is a piece of data "186 Chen Env 28" in the target data table that has not been synchronized and needs to be deferred for synchronization; there is a piece of data "185 Liu Yinan 26" that has not been deleted.

[0144] 3), After the second data synchronization, the source data table, the target data table, and the deferred memory are as Figure 5As shown. In the source data table, the new data "188 Li Si Male 61" and "189 Wang Wu Female 28" are added. The second data synchronization is performed at 2024-3-2-09:00. Since the hash value corresponding to "188 Li Si" exists in the Bloom filter, the new data "188 Li Si Male 61" is not inserted into the target data table. Since the hash value corresponding to "189 Wang Wu" does not exist in the Bloom filter, the new data "189 Wang Wu Female 28" is inserted into the target data table, and the data "120 Liu Yi 1" is added to the deferred memory, and the deferred times corresponding to "186 Chen Er", "187 Zhang San", and "188 Li Si" are updated to 2.

[0145] 4) After the third data synchronization, the source data table, target data table and deferred memory are as follows Figure 6 As shown. In which, the data "190 Zhao Liu Female 60" is added to the source data table. The third data synchronization is performed at 2024-3-2-18:00. Since the hash value corresponding to "190 Zhao Liu" does not exist in the Bloom filter, the newly added data "190 Zhao Liu Female 60" is inserted into the target data table. Since the extension times corresponding to "186 Chen Er", "187 Zhang San", and "188 Li Si" have reached the extension threshold of 2, the data "186 Chen Er Male 28", "187 Zhang San Male 40", and "188 Li Si Female 40" in the target data table are deleted, and the data "186 Chen Er Female 28", "187 Zhang San Male 40", "188 Li Si Female 40", and "188 Li Si Male 61" in the source data table are inserted into the target data table. The data "189 Wang Wu 1" is added to the extension memory, the extension times corresponding to "186 Chen Er", "187 Zhang San", and "188 Li Si" are reset to 0, and the extension times corresponding to "120 Liu Yi" are updated to 2.

[0146] 5) Execute the scan task at 2024-3-2-24:00. After scanning the target data table, the source data table and the target data table are as follows: Figure 7 As shown. Among them, the update time of the data "185 Liu Yi Nan 26" in the target data table is 40 hours, which is greater than the time threshold of 24 hours. The source data table is queried to determine that "185 Liu Yi" does not exist in the source data table, and the data "185 Liu Yi Nan 26" is deleted in the target data table. At this time, the data in the source data table is consistent with the data in the target data table.

[0147] It is understandable that the above combination Figures 3 to 7 This is merely an illustrative description and should not be construed as limiting the embodiments of the present application.

[0148] See also Figure 8 , a schematic diagram of the structure of a data synchronization device disclosed in an embodiment of the present application, the device includes:

[0149] Determining module 801, used to determine a first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is a value corresponding to the distinguishing field combination, and the distinguishing field combination is obtained by concatenating the fields in the source data table;

[0150] The determination module 801 is further configured to determine a first hash value according to the value of the first distinguishing field combination;

[0151] The synchronization module 802 is used to synchronize the first data to the target data table if the first hash value does not exist in the multiple second hash values; the multiple second hash values ​​are obtained according to the initial data table, and the full amount of data in the initial data table has been synchronized to the target data table.

[0152] It can be seen that in the embodiment of the present application, the corresponding hash value is calculated through the fields existing in the source data table, and combined with the multiple second hash values ​​corresponding to the initial data table to determine whether the first data is newly added or modified data in the source data table, and the corresponding data synchronization is performed. In this way, when the database itself does not support update logs and the data table has no incremental fields to use, it is possible to avoid full data synchronization every time incremental data is synchronized, thereby improving the efficiency of data synchronization.

[0153] In a possible implementation, the data synchronization device provided in the embodiment of the present application further includes:

[0154] an updating module, configured to update the deferred memory using the first distinguishing field combination value if the first hash value exists in the plurality of second hash values ​​and the first distinguishing field combination value does not exist in the deferred memory;

[0155] a counting module, configured to increase the number of extensions by one if the first hash value exists in the plurality of second hash values ​​and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is less than or equal to the extension threshold;

[0156] The synchronization module is also used to update the target data table using the first data if the first hash value exists in the multiple second hash values ​​and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is greater than the extension threshold.

[0157] In a possible implementation, the synchronization module provided in the embodiment of the present application includes:

[0158] A deleting unit, used for deleting the second data in the target data table that can be matched by the first distinguishing field combination value;

[0159] A synchronization unit is used to synchronize the first data to the target data table.

[0160] In a possible implementation, the data synchronization device provided in the embodiment of the present application further includes:

[0161] A scanning module, configured to scan the target data table when a set time is reached, and determine third data in the target data table whose update time is greater than a time threshold;

[0162] The deleting module is used to delete the third data if the second distinguishing field combination value corresponding to the third data does not exist in the source data table.

[0163] In a possible implementation, the determination module in the data synchronization device provided in the embodiment of the present application is further used to determine the total number of data items of the fourth data in the initial data table;

[0164] The data synchronization device also includes:

[0165] A deduplication module is used to remove duplicate values ​​corresponding to each field in the initial data table to obtain the deduplicated total amount corresponding to each field;

[0166] The determination module is further used to determine the discrimination corresponding to each field according to the total number of data items and the total amount after deduplication;

[0167] The determination module is further used to determine the field corresponding to the degree of distinction greater than the distinction threshold as the distinction field;

[0168] The data synchronization device also includes:

[0169] The splicing module is used to splice multiple distinguishing fields to obtain the distinguishing field combination.

[0170] In a possible implementation, the splicing module in the data synchronization device provided in the embodiment of the present application includes:

[0171] A determining unit, configured to determine a plurality of target distinguishing fields from a plurality of the distinguishing fields according to the priorities of the distinguishing fields;

[0172] A sorting unit, used for sorting the plurality of target distinguishing fields according to the priority and distinction of the distinguishing fields to obtain a sorting result;

[0173] A concatenation unit is used to concatenate the plurality of target distinguishing fields according to the sorting result to obtain the distinguishing field combination.

[0174] In a possible implementation, the determination module in the data synchronization device provided in the embodiment of the present application is further used to determine a third distinguishing field combination value corresponding to the fourth data in the initial data table;

[0175] The determination module is further used to determine the second Hash value according to the third distinguishing field combination value;

[0176] The data synchronization device also includes:

[0177] A setting unit, configured to set the Bloom filter according to a plurality of the second Hash values;

[0178] The synchronization module is specifically configured to synchronize the first data to a target data table if the first hash value does not exist in the Bloom filter.

[0179] Furthermore, the embodiment of the present application also provides a data synchronization device, including:

[0180] A memory for storing instructions;

[0181] A processor is used to execute instructions in a memory to perform any one of the implementation methods of the above-mentioned data synchronization method.

[0182] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned data synchronization method.

[0183] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned data synchronization method.

[0184] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0185] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0186] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization method, characterized in that: The method comprises: Determine a first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is a value corresponding to the distinguishing field combination of the first data, and the distinguishing field combination is obtained by concatenating the fields in the source data table; Determine a first Hash value according to the first distinguishing field combination value; If the first hash value does not exist in the multiple second hash values, the first data is synchronized to the target data table; the multiple second hash values ​​are obtained according to the initial data table, and the full amount of data in the initial data table has been synchronized to the target data table.

2. The method according to claim 1, characterized in that The method further comprises: If the first Hash value exists in the plurality of second Hash values ​​and the first distinguishing field combination value does not exist in the deferred memory, updating the deferred memory with the first distinguishing field combination value; If the first Hash value exists in the plurality of second Hash values, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is less than or equal to the extension threshold, the number of extensions is increased by one; If the first hash value exists in the plurality of second hash values, and the number of extensions corresponding to the first distinguishing field combination value in the extension memory is greater than the extension threshold, the target data table is updated using the first data.

3. The method according to claim 2, characterized in that The updating of the target data table by using the first data includes: Deleting the second data in the target data table that can be matched by the first distinguishing field combination value; The first data is synchronized to the target data table.

4. The method according to claim 2, characterized in that: The method further comprises: When the set time is reached, scanning the target data table to determine third data in the target data table whose update time is greater than the time threshold; If the second distinguishing field combination value corresponding to the third data does not exist in the source data table, the third data is deleted.

5. The method according to claim 1, characterized in that The method further comprises: Determine the total number of data items of the fourth data in the initial data table; Deduplicating the values ​​corresponding to each field in the initial data table to obtain the deduplicated total amount corresponding to each field; Determine the discrimination degree corresponding to each field according to the total number of data items and the total amount after deduplication; The field corresponding to the degree of distinction greater than the distinction threshold is determined as the distinction field; A plurality of the distinguishing fields are concatenated to obtain the distinguishing field combination.

6. The method according to claim 5, characterized in that The step of concatenating the plurality of distinguishing fields to obtain a distinguishing field combination includes: Determining a plurality of target distinguishing fields from a plurality of the distinguishing fields according to the priorities of the distinguishing fields; Sorting the plurality of target distinguishing fields according to the priority and the degree of distinction of the distinguishing fields to obtain a sorting result; According to the sorting result, a plurality of the target distinguishing fields are concatenated to obtain the distinguishing field combination.

7. The method according to claim 1, characterized in that The method further comprises: Determine a third distinguishing field combination value corresponding to the fourth data in the initial data table; Determine a second Hash value according to the third distinguishing field combination value; Setting the Bloom filter according to the plurality of the second hash values; Correspondingly, if the first hash value does not exist in the plurality of second hash values, synchronizing the first data to the target data table includes: If the first hash value does not exist in the Bloom filter, the first data is synchronized to the target data table.

8. A data synchronization device, characterized in that: The device comprises: A determination module, used to determine a first distinguishing field combination value corresponding to the first data in the source data table; the first distinguishing field combination value is a value corresponding to the distinguishing field combination, and the distinguishing field combination is obtained by concatenating the fields in the source data table; The determination module is further used to determine a first Hash value according to the value of the first distinguishing field combination; A synchronization module is used to synchronize the first data to a target data table if the first hash value does not exist in multiple second hash values; the multiple second hash values ​​are obtained according to an initial data table, and all data in the initial data table has been synchronized to the target data table.

9. A data synchronization device, characterized in that: include: A memory for storing instructions; A processor, configured to execute the instructions in the memory to execute the data synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the data synchronization method according to any one of claims 1 to 7.

Images