A data synchronization method, device, medium and equipment

Abstract

The present invention relates to the field of data processing, and in particular to a data synchronization method, device, medium and equipment, comprising: based on the second attribute information of the target application data table and the first attribute information of the initial application data table, a first intermediate data table is screened out, and when the data volume of the first intermediate data table is small, the first intermediate data table is directly sent to the target client to assist the target client to directly verify and synchronously update the application data in the target application data table one by one according to the various application data in the first intermediate data table, thereby improving the accuracy of data synchronization; when the data volume of the first intermediate data table is large, the application data corresponding to the reference field is grouped and hashed, so as to assist the target client to verify and synchronously update the application data in the target application data table according to the various reference hash data in the second intermediate data table, thereby reducing the amount of data that needs to be verified and improving the efficiency of data synchronization.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data synchronization method, device, medium and equipment. Background Art

[0002] In today's digital age, data processing and application are becoming increasingly complex and important. Different clients and servers often need to interact and synchronize data to ensure data consistency and accuracy to meet various business needs.

[0003] As enterprise businesses expand and diversify, data exchange between clients and servers is frequent and large in volume. However, due to differences in the data table structures between the client and server, data synchronization faces many challenges. Due to the difference in data volume, it may exceed the client's processing capacity or fail to meet expected requirements, thus affecting the effect of data synchronization and the normal operation of the business.

[0004] For example, in a large enterprise management system, the server may store a large number of application data tables, while different target clients may have their own target application data tables with specific formats and requirements. Traditional data synchronization methods often cannot accurately match the data tables on the client and server, resulting in low accuracy and efficiency of data synchronization.

[0005] Therefore, how to improve the accuracy and efficiency of data synchronization between the server and the client has become an urgent problem to be solved. Summary of the invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is a data synchronization method, which includes the following steps:

[0007] S1, obtains several initial application data tables stored on the server side, the first attribute information of each initial application data table, and the target application data table provided by the target client and the second attribute information of the target application data table, wherein the first attribute information includes the initial table name corresponding to the initial application data table, several initial field names and the field type corresponding to each initial field name, and the second attribute information includes the target table name of the target application data table, several target field names and the field type corresponding to each target field name.

[0008] S2: Acquire a matching degree between the target application data table and each initial application data table according to the second attribute information of the target application data table and the first attribute information of each initial application data table.

[0009] S3, determining the initial application data table corresponding to the maximum matching degree as the designated application data table corresponding to the target application data table.

[0010] S4, according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table, a first intermediate data table is obtained by screening from the designated application data table.

[0011] S5. According to the first intermediate data table, obtain the data magnitude corresponding to the first intermediate data table.

[0012] S6, comparing the data level with a preset data level threshold to obtain a data comparison result.

[0013] S7, processing the first intermediate data table according to a preset data processing method corresponding to the data comparison result to obtain a second intermediate data table, wherein the preset data processing method includes keeping the first intermediate data table unchanged and performing group hash conversion on the first intermediate data table.

[0014] S8, sending the second intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the second intermediate data table.

[0015] The present invention also provides a data synchronization device, the data synchronization device comprising:

[0016] A data acquisition module is used to acquire several initial application data tables stored on the server side, first attribute information of each initial application data table, and a target application data table provided by a target client and second attribute information of the target application data table, wherein the first attribute information includes an initial table name corresponding to the initial application data table, several initial field names and a field type corresponding to each initial field name, and the second attribute information includes a target table name of the target application data table, several target field names and a field type corresponding to each target field name.

[0017] The information matching module is used to obtain the matching degree between the target application data table and each initial application data table according to the second attribute information of the target application data table and the first attribute information of each initial application data table.

[0018] The data table determination module is used to determine the initial application data table corresponding to the maximum matching degree as the designated application data table corresponding to the target application data table.

[0019] The data table screening module is used to screen the first intermediate data table from the designated application data table according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table.

[0020] The magnitude acquisition module is used to acquire the data magnitude corresponding to the first intermediate data table according to the first intermediate data table.

[0021] The threshold comparison module is used to compare the data level with a preset data level threshold to obtain a data comparison result.

[0022] The data table processing module is used to process the first intermediate data table according to the preset data processing method corresponding to the data comparison result to obtain the second intermediate data table, wherein the preset data processing method includes keeping the first intermediate data table unchanged and performing group hash conversion on the first intermediate data table.

[0023] The data table sending module is used to send the second intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the second intermediate data table.

[0024] The present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned data synchronization method.

[0025] The present invention also provides an electronic device, comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0026] The present invention has at least the following beneficial effects: according to the second attribute information of the target application data table and the first attribute information of each initial application data table, the matching degree between the target application data table and each initial application data table is obtained, the initial application data table corresponding to the maximum matching degree is determined as the designated application data table corresponding to the target application data table, according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table, a first intermediate data table is screened from the designated application data table, according to the first intermediate data table, the data magnitude corresponding to the first intermediate data table is obtained, the data magnitude is compared with a preset data magnitude threshold, and a data comparison result is obtained, according to the preset data processing method corresponding to the data comparison result, the first intermediate data table is processed to obtain a second intermediate data table, and the second intermediate data table is sent to the target client. It can be seen that based on the target application Based on the similarity between the second attribute information of the data table and the first attribute information of each initial application data table, the first intermediate data table is selected as the basis for consistency verification and synchronous update of the target application data, and when the amount of data in the first intermediate data table is small, the first intermediate data table is directly sent to the target client to assist the target client to directly verify and synchronously update the application data in the target application data table one by one according to the various application data in the first intermediate data table, thereby improving the accuracy of data synchronization; when the amount of data in the first intermediate data table is large, the application data corresponding to each reference field is grouped and hashed, so as to assist the target client to verify and synchronously update the hash data converted from the application data in the target application data table one by one according to the various reference hash data in the second intermediate data table, thereby reducing the amount of data that needs to be verified and improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flowchart of a data synchronization method provided in Embodiment 1 of the present invention;

[0029] Figure 2 A flowchart of a data synchronization method based on a hash algorithm provided in Embodiment 2 of the present invention;

[0030] Figure 3 A flowchart of a data sending method provided in Embodiment 3 of the present invention;

[0031] Figure 4A structural diagram of a data synchronization device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the above-mentioned terms used to distinguish similar objects can be interchanged so that the present invention can also implement other embodiments other than the above-mentioned illustrated embodiments or described embodiments. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] This embodiment provides a data synchronization method, which includes the following steps: Figure 1 As shown:

[0036] S1, obtains several initial application data tables stored on the server side, the first attribute information of each initial application data table, and the target application data table provided by the target client and the second attribute information of the target application data table, wherein the first attribute information includes the initial table name corresponding to the initial application data table, several initial field names and the field type corresponding to each initial field name, and the second attribute information includes the target table name of the target application data table, several target field names and the field type corresponding to each target field name.

[0037] Among them, the server side stores several initial application data tables. The target client can purchase data from an initial application data table on the server side based on its own needs to form its own target application data table, and make consistency judgments on the target client's data and the server's data. Therefore, when abnormal data appears on the target client or the server side updates the data, the target client's data is verified and synchronously updated to meet the various business needs of the target client.

[0038] For example, in an e-commerce platform, the commodity inventory data table on the server side needs to be synchronized with the client data table in each region to ensure that clients in each region can obtain accurate inventory information in real time to avoid overselling or inaccurate inventory. Another example is in the financial field, the transaction data on the server side needs to be synchronized with different clients (such as mobile terminals and web terminals) to ensure that users can obtain the latest and consistent transaction records on different terminals.

[0039] As corporate businesses expand and diversify, data exchange between the target client and the server is frequent and large in volume, and differences may exist in data table structure, data table naming, field naming, field type and other data information between the target client and the server.

[0040] Therefore, in this embodiment, the client that needs to perform data consistency verification and synchronization is used as the target client to obtain the first attribute information of each initial application data table, that is, the initial table name of the initial application data table, several initial field names and the field type corresponding to each initial field name, and the second attribute information of the target application data table, that is, the target table name of the target application data table, several target field names and the field type corresponding to each target field name, so as to determine the initial application data table with the greatest correlation with the target application data table by measuring the similarity of the table name, field name and field type between the initial application data table and the target application data table, as the basis for measuring data consistency.

[0041] The specific field type may be set by the implementer according to the actual situation. For example, the field type may include integer type, floating point type, character type, date and time type, Boolean type, etc.

[0042] S2: Acquire a matching degree between the target application data table and each initial application data table according to the second attribute information of the target application data table and the first attribute information of each initial application data table.

[0043] The matching degree may represent the matching degree between the target application data table and the corresponding initial application data table, that is, the degree to which the target application data table is derived from the corresponding initial application data table.

[0044] In a specific implementation, S2 further includes the following steps:

[0045] S21 , for any initial application data table, according to the initial table name and the target table name of the current initial application data table, obtain a first matching score corresponding to the current initial application data table.

[0046] S22, obtaining a second matching score corresponding to the current initial application data table according to a number of initial field names corresponding to the current initial application data table and a field type corresponding to each initial field name, and a number of target field names and a field type corresponding to each target field name.

[0047] S23: Acquire the matching degree between the target application data table and the current initial application data table according to the first matching score and the second matching score.

[0048] The first matching score is used to characterize the similarity between the target application data table and the current initial application data table regarding the data table name, and the second matching score is used to characterize the similarity between the target application data table and the current initial application data table regarding the field name and field type.

[0049] In a specific implementation, S22 further includes the following steps:

[0050] S221 : For any target field name, calculate a first similarity between the current target field name and each initial field name in the current initial application data table.

[0051] S222, performing consistency matching on the field type corresponding to the current target field name and the field type corresponding to each initial field name in the current initial application data table, and obtaining a second similarity between the current target field name and each initial field name in the current initial application data table.

[0052] S223: Acquire a third similarity between the current target field name and each initial field name in the current initial application data table according to the first similarity and the second similarity.

[0053] S224: Determine the maximum value of the third similarities between the current target field name and all the initial field names in the current initial application data table as the matching degree between the current initial application data table and the current target field name.

[0054] S225, traversing all target field names, and determining the average value of the matching degree between the current initial application data table and all target field names as the second matching score corresponding to the current initial application data table.

[0055] Among them, the target field name and the initial field name are converted into corresponding word vectors according to a preset word vector model, and the first similarity between the target field name and the initial field name is represented according to the similarity between the word vectors.

[0056] The specific word vector model can be set by the implementer according to the actual situation. For example, the preset word vector model can refer to the word vector model such as Global Vectors for Word Representation (GloVe), Word2Vec and FastText. Among them, those skilled in the art know that any word vector model in the prior art falls within the protection scope of the present invention, and will not be repeated here.

[0057] If the field type corresponding to the target field name is consistent with the field type corresponding to the initial field name in the current initial application data table, the corresponding second similarity is determined to be 1. If the field type corresponding to the target field name is consistent with the field type corresponding to the initial field name in the current initial application data table, the corresponding second similarity is determined to be 0.

[0058] Based on the first mapping weight corresponding to the field name and the second preset weight corresponding to the field type, the first similarity and the second similarity between the current target field name and each initial field name in the current initial application data table are weightedly summed to obtain the corresponding third similarity.

[0059] As described above, based on the second attribute information of the target application data table and the first attribute information of each initial application data table, the degree of matching between the target application data table and each initial application data table is measured, which is used to characterize the degree to which the target application data table is derived from the corresponding initial application data table, thereby providing a verification object for consistency verification of the target application data of the target client, thereby improving the verification accuracy.

[0060] S3, determining the initial application data table corresponding to the maximum matching degree as the designated application data table corresponding to the target application data table.

[0061] S4, according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table, a first intermediate data table is obtained by screening from the designated application data table.

[0062] Among them, the target client may have purchased part of the application data in the specified application data table to form the target application data table. Therefore, this embodiment filters the initial field names corresponding to the specified application data table according to the target field names corresponding to the target application data table and the initial field names corresponding to the specified application data table, and then obtains the first intermediate data table from the specified application data table as the basis for consistency verification and synchronous update of the target application data.

[0063] In a specific implementation, S4 further includes the following steps:

[0064] S41: For any target field name, obtain a third similarity between the current target field name and each initial field name in the specified application data table.

[0065] S42, determining the initial field name corresponding to the largest third similarity in the specified application data table as the reference field name corresponding to the current target field name.

[0066] S43, traverse all target field names, and obtain the reference field name corresponding to each target field name.

[0067] S44, filtering out application data corresponding to each reference field name from the specified application data table.

[0068] S45, forming a first intermediate data table according to all the extracted application data.

[0069] In the above, by measuring the similarity between the target field name and the initial field name corresponding to the specified application data table, the initial field name with higher similarity to the target field name is screened out from all the initial field names corresponding to the specified application data table, and then the first intermediate data table is obtained by screening from the specified application data table as a basis for consistency verification and synchronous update of the target application data. While reducing the amount of data that needs to be verified, the interference of application data corresponding to initial field names with lower similarity on data consistency verification is eliminated, thereby improving the accuracy and efficiency of data synchronization.

[0070] S5. According to the first intermediate data table, obtain the data magnitude corresponding to the first intermediate data table.

[0071] The data magnitude may represent the amount of data in the first intermediate data table. This embodiment selects a processing method for the first intermediate data table according to the amount of data to balance the accuracy and efficiency of data verification.

[0072] In a specific implementation, S5 further includes the following steps:

[0073] S51, obtaining a second data volume of application data in the first intermediate data table.

[0074] S52, performing a logarithm operation on the second data volume to obtain a data volume level corresponding to the first intermediate data table.

[0075] S6, comparing the data level with a preset data level threshold to obtain a data comparison result.

[0076] The data comparison result is that the data magnitude is greater than the threshold, which is used to indicate that the data volume of the first intermediate data table is large, or the data magnitude is less than or equal to the threshold, which is used to indicate that the data volume of the first intermediate data table is small.

[0077] The specific value of the preset data level threshold can be set by the implementer according to the actual situation. For example, the preset data level threshold can be set to 5.

[0078] In the above, the data level is compared with the preset data level threshold to obtain a data comparison result, which is used to characterize the data volume of the first intermediate data table and provide a basis for selecting a processing method corresponding to the first intermediate data table.

[0079] S7, processing the first intermediate data table according to a preset data processing method corresponding to the data comparison result to obtain a second intermediate data table, wherein the preset data processing method includes keeping the first intermediate data table unchanged and performing group hash conversion on the first intermediate data table.

[0080] In a specific implementation, the data comparison result includes that the data level is greater than a preset data level threshold and the data level is less than or equal to the preset data level threshold, and S7 further includes the following steps:

[0081] S71, if the data comparison result is that the data level is less than or equal to the preset data level threshold, the preset data processing method corresponding to the data comparison result is to keep the first intermediate data table unchanged and determine the first intermediate data table as the second intermediate data table.

[0082] S72, if the data comparison result is that the data magnitude is greater than the preset data magnitude threshold, the preset data processing method corresponding to the data comparison result is to perform group hash conversion on the first intermediate data table to obtain the second intermediate data table, wherein S72 also includes the following steps:

[0083] S721 obtains the first data volume of the application data corresponding to each reference field name in the first intermediate data table.

[0084] S722: For any reference field name, obtain a reference group length corresponding to the current reference field name according to the first data volume corresponding to the current reference field name and a preset number of groups.

[0085] S723, grouping the application data corresponding to the current reference field name according to the reference group length corresponding to the current reference field name, and acquiring all reference application data groups corresponding to the current reference field name.

[0086] S724, process each reference application data group according to a preset hash algorithm to obtain reference hash data corresponding to each reference application data group.

[0087] S725, traverse all reference fields to obtain all reference hash data.

[0088] S726, forming a second intermediate data table according to all the reference hash data.

[0089] Among them, if the data comparison result is that the data level is less than or equal to the preset data level threshold, the first intermediate data table is kept unchanged, and the first intermediate data table is determined as the second intermediate data table. Correspondingly, the second intermediate data table is sent to the target client, that is, the first intermediate data table is directly sent to the target client, so as to assist the target client to directly verify and synchronously update the application data in the target application data table one by one according to the various application data in the first intermediate data table, thereby improving the accuracy of data synchronization.

[0090] If the data comparison result is that the data level is greater than the preset data level threshold, the application data corresponding to each reference field is grouped to obtain a reference application data group with a smaller data volume, and each reference application data group is hashed according to a preset hash algorithm. A second intermediate data table is formed according to all the reference hash data, and the second intermediate data table is sent to the target client to assist the target client in verifying and synchronously updating the hash data converted from the application data in the target application data table one by one according to each reference hash data in the second intermediate data table, thereby reducing the amount of data that needs to be verified and improving the efficiency of data synchronization.

[0091] Among them, those skilled in the art know that any hash algorithm in the prior art falls within the protection scope of the present invention, and will not be described in detail here.

[0092] In the above, the first intermediate data table is processed according to the data comparison result. When the amount of data in the first intermediate data table is small, the first intermediate data table is directly sent to the target client to assist the target client to directly verify and synchronously update the application data in the target application data table one by one according to the various application data in the first intermediate data table, thereby improving the accuracy of data synchronization. When the amount of data in the first intermediate data table is large, the application data corresponding to each reference field is grouped and hash-converted to assist the target client to verify and synchronously update the hash data converted from the application data in the target application data table one by one according to the various reference hash data in the second intermediate data table, thereby reducing the amount of data that needs to be verified and improving the efficiency of data synchronization.

[0093] S8, sending the second intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the second intermediate data table.

[0094] In a specific implementation, the target client is used to compare and verify the second intermediate data table and the target application data table, and update the target application data table according to the comparison and verification result, so that the updated target application data table is synchronized with the first intermediate data table.

[0095] Among them, the comparison verification results include consistency and inconsistency.

[0096] Furthermore, when the first intermediate data table is kept unchanged to obtain the second intermediate data table, if the comparison and verification result is consistent, the target application data table is kept unchanged. If the comparison and verification result is inconsistent, the target application data table is updated according to the first intermediate data table, so that the updated target application data table is synchronized with the first intermediate data table, thereby meeting various business requirements corresponding to the target client.

[0097] When the first intermediate data table is grouped and hashed to obtain the second intermediate data table, if the comparison and verification result is consistent, the target application data table is kept unchanged. If the comparison and verification result is inconsistent, the actual group length corresponding to each target field name in the target application data is obtained according to the sub-data volume corresponding to each target field name and the preset number of groups, and the application data in all target field names are grouped and hashed according to the preset hash algorithm and the actual group length corresponding to each target field name to obtain the actual hash data table, and the target hash data table and the actual hash data table are compared and verified, and the target application data table is updated according to the comparison and verification result, so that the updated target application data table is synchronized with the first intermediate data table, thereby meeting various business needs corresponding to the target client.

[0098] Further, if the comparison and verification result is consistent, the target application data table is kept unchanged. If the comparison and verification result is inconsistent, the inconsistent abnormal hash value and the application data corresponding to the abnormal hash value in the first intermediate data table are located, and the application data corresponding to the abnormal hash value in the target application data table is updated according to the application data corresponding to the abnormal hash value in the first intermediate data table, so that the updated target application data table is synchronized with the first intermediate data table.

[0099] In the above, according to the second attribute information of the target application data table and the first attribute information of each initial application data table, the matching degree between the target application data table and each initial application data table is obtained, and the initial application data table corresponding to the maximum matching degree is determined as the designated application data table corresponding to the target application data table, and according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table, the first intermediate data table is screened from the designated application data table, and according to the first intermediate data table, the data magnitude corresponding to the first intermediate data table is obtained, and the data magnitude is compared with a preset data magnitude threshold to obtain a data comparison result, and according to the preset data processing method corresponding to the data comparison result, the first intermediate data table is processed to obtain a second intermediate data table, and the second intermediate data table is sent to the target client. It can be seen that based on the first intermediate data table, Based on the similarity between the second attribute information and the first attribute information of each initial application data table, the first intermediate data table is selected as the basis for consistency verification and synchronous update of the target application data, and when the amount of data in the first intermediate data table is small, the first intermediate data table is directly sent to the target client to assist the target client to directly verify and synchronously update the application data in the target application data table one by one according to the various application data in the first intermediate data table, thereby improving the accuracy of data synchronization; when the amount of data in the first intermediate data table is large, the application data corresponding to each reference field is grouped and hashed, so as to assist the target client to verify and synchronously update the hash data converted from the application data in the target application data table one by one according to the various reference hash data in the second intermediate data table, thereby reducing the amount of data that needs to be verified and improving the efficiency of data synchronization.

[0100] Embodiment 2

[0101] The second embodiment provides a data synchronization method based on a hash algorithm, and the data synchronization method based on a hash algorithm includes: Figure 2 As shown:

[0102] S10, obtain the target application data table provided by the target client, the first intermediate data table corresponding to the target application data table stored on the server, several reference field names in the first intermediate data table, the total data volume corresponding to the first intermediate data table and the corresponding number of data updates within a preset time period, and the corresponding number of selections and exceptions for each reference field name within the preset time period.

[0103] The specific time of the preset time period can be set by the implementer according to the actual situation. For example, the preset time period can refer to a time period such as one month, six months, or one year before the current time point.

[0104] The number of data updates refers to the number of times the server actively updates the corresponding application data in the first intermediate data table.

[0105] The number of selections refers to the number of times the corresponding reference field name is purchased by the client within a preset time period.

[0106] The number of exceptions refers to the number of times that the corresponding reference field name is inconsistent with the corresponding application data in the target application data table when participating in data consistency verification within a preset time period.

[0107] With the expansion and diversification of enterprise business, the data exchange between the target client and the server is frequent and large in volume. When the target application data table is consistency verified and synchronously updated based on the first intermediate data table, the more total data volume corresponds to the first intermediate data table, the higher the number of data updates corresponding to the first intermediate data table within the preset time period, and the more selection times and exception times corresponding to the reference field name in the first intermediate data table within the preset time period, the higher the cost of verifying the application data in the first intermediate data table and the target application data table one by one, and the lower the cost of performing hash conversion on the application data in the first intermediate data table and the target application data table and then verifying it.

[0108] Therefore, this embodiment obtains the total amount of data corresponding to the first intermediate data table and the corresponding number of data updates within the preset time period, the corresponding number of selections and the number of exceptions for each reference field name within the preset time period, to measure the cost of verifying the application data in the first intermediate data table and the target application data table one by one, that is, the degree of need for hash conversion of the first intermediate data table, and selects the processing form of the first intermediate data table accordingly, so as to balance and improve the accuracy and efficiency of data synchronization.

[0109] S20, obtaining the hash conversion requirement degree corresponding to the first intermediate data table according to the number of data updates corresponding to the first intermediate data table, the number of selections and the number of exceptions corresponding to each reference field name within a preset time period.

[0110] In a specific implementation, S20 further includes the following steps:

[0111] S201, obtaining a first preset weight corresponding to the number of data updates, a second preset weight corresponding to the number of selections, and a third preset weight corresponding to the number of exceptions.

[0112] S202, respectively normalize the data update times, selection times and exception times to obtain corresponding data update reference times, reference selection times and reference exception times.

[0113] S203, obtaining the hash conversion requirement degree corresponding to the first intermediate data table according to the reference number of data updates, the reference number of selections, the reference number of exceptions, the first preset weight, the second preset weight and the third preset weight.

[0114] Among them, the average data update times corresponding to all initial application data tables stored on the server side within the preset time period, the average selection times and the average exception times corresponding to all initial field names corresponding to all initial application data tables within the preset time period are obtained, and the ratio of the data update times and the average data update times is determined as the corresponding data update reference times, the ratio of the selection times and the average selection times is determined as the corresponding reference selection times, and the ratio of the exception times and the average exception times is determined as the corresponding reference exception times, thereby unifying the scales of the reference times, reference selection times and reference exception times, and improving the accuracy of the hash conversion requirement degree.

[0115] The specific values ​​of the first preset weight, the second preset weight and the third preset weight can be set by the implementer according to actual conditions.

[0116] As described above, based on the number of data updates corresponding to the first intermediate data table, the number of selections and the number of exceptions corresponding to each reference field name within a preset time period, the degree of hash conversion demand corresponding to the first intermediate data table is obtained to analyze the cost of verifying the application data in the first intermediate data table and the target application data table one by one, and the cost of performing hash conversion and then verifying the application data in the first intermediate data table and the target application data table, and accordingly select the processing form for the first intermediate data table to balance and improve the accuracy and efficiency of data synchronization.

[0117] S30, determining a first data processing method corresponding to the first intermediate data table according to the hash conversion requirement and a preset requirement threshold, wherein the first data processing method includes keeping unchanged and performing hash conversion.

[0118] In a specific implementation, S30 further includes the following steps:

[0119] S301: If the hash conversion requirement is less than a preset requirement threshold, determine that the first data processing method corresponding to the first intermediate data table remains unchanged.

[0120] S302: If the hash conversion requirement is greater than or equal to a preset requirement threshold, determine that the first data processing method corresponding to the first intermediate data table is to perform hash conversion.

[0121] Among them, if the hash conversion requirement degree is less than the preset requirement degree threshold, it means that the cost of verifying the application data in the first intermediate data table and the target application data table one by one is less than the cost of verifying the application data in the first intermediate data table and the target application data table after hash conversion, then it is determined that the first data processing method corresponding to the first intermediate data table remains unchanged.

[0122] If the degree of hash conversion requirement is greater than or equal to a preset degree of requirement threshold, it means that the cost of verifying the application data in the first intermediate data table and the target application data table one by one is greater than or equal to the cost of verifying the application data in the first intermediate data table and the target application data table after hash conversion, then it is determined that the first data processing method corresponding to the first intermediate data table is hash conversion.

[0123] The specific value of the preset demand level threshold can be set by the implementer according to actual conditions.

[0124] S40, if the first data processing method is to perform hash conversion, then determine the second data processing method corresponding to the first intermediate data table according to the total data volume corresponding to the first intermediate data table and the preset data volume threshold, wherein the second data processing method includes overall hash conversion and grouped hash conversion.

[0125] In a specific implementation, S40 further includes the following steps:

[0126] S401: If the total data volume is less than a preset data volume threshold, determine that the second data processing method corresponding to the first intermediate data table is overall hash conversion.

[0127] S402: If the total data volume is greater than or equal to a preset data volume threshold, determine that the second data processing method corresponding to the first intermediate data table is group hash conversion.

[0128] Among them, when the amount of data to be hashed is too small, the amount of data information contained is small, and the probability of hash collision occurring during hash conversion is relatively high, that is, the probability that different data corresponds to the same hash value is relatively high.

[0129] Therefore, in order to improve the accuracy of the target hash data table, if the total data volume is less than the preset data volume threshold, the first intermediate data table can be hashed as a whole. If the total data volume is greater than or equal to the preset data volume threshold, the first intermediate data table can be grouped and then hashed, which can improve the efficiency of locating abnormal data during data consistency verification while ensuring the accuracy of the target hash data table, thereby comprehensively improving the accuracy and efficiency of data synchronization.

[0130] The specific value of the preset data volume threshold can be set by the implementer according to actual conditions.

[0131] As described above, when the total data volume is less than the preset data volume threshold, the first intermediate data table is hash converted as a whole; when the total data volume is greater than or equal to the preset data volume threshold, the first intermediate data table is grouped and then hash converted. While ensuring the accuracy of the target hash data table, the efficiency of locating abnormal data can be improved during data consistency verification, thereby comprehensively improving the accuracy and efficiency of data synchronization.

[0132] S50, if the second data processing method is group hash conversion, then according to the amount of sub-data corresponding to each reference field name and the preset number of groups, the target group length corresponding to each reference field name is obtained.

[0133] The specific value of the preset number of groups can be set by the implementer according to actual conditions.

[0134] Specifically, for any reference field name, the ratio of the amount of sub-data corresponding to the current reference field name and the preset number of groups is determined as the candidate group length corresponding to the current reference field name. The candidate group length is rounded up to obtain the target group length corresponding to the current reference field name, which is used to group the application data corresponding to the current reference field name.

[0135] S60, performing group hash conversion on the application data in each reference field name according to a preset hash algorithm and a target group length corresponding to each reference field name, and obtaining a target hash data table.

[0136] In a specific implementation, S60 further includes the following steps:

[0137] S601, for any reference field name, group the application data corresponding to the current reference field name according to the target group length corresponding to the current reference field name, and obtain all target application data groups corresponding to the current reference field name.

[0138] S602: Process each target application data group according to a preset hash algorithm to obtain hash conversion data corresponding to each target application data group.

[0139] S603, traverse all reference field names to obtain all hash conversion data.

[0140] S604, forming a target hash data table according to all hash conversion data.

[0141] Among them, those skilled in the art know that any hash algorithm in the prior art falls within the protection scope of the present invention, and will not be described in detail here.

[0142] As described above, according to the preset hash algorithm and the target group length corresponding to each reference field name, the application data in each reference field name is grouped and hashed to obtain the target hash data table as the basis for verifying and synchronizing the target application data table.

[0143] S70, sending the target hash data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the target hash data table.

[0144] In a specific embodiment, the target client is used to obtain the actual group length corresponding to each target field name based on the amount of sub-data corresponding to each target field name in the target application data and the preset number of groups, and perform group hash conversion on the application data in all target field names according to the preset hash algorithm and the actual group length corresponding to each target field name to obtain the actual hash data table, compare and verify the target hash data table and the actual hash data table, and update the target application data table based on the comparison and verification results, so that the updated target application data table and the first intermediate data table are synchronized.

[0145] Among them, the comparison verification results include consistency and inconsistency.

[0146] Furthermore, if the comparison and verification result is consistent, the target application data table is kept unchanged. If the comparison and verification result is inconsistent, the abnormal hash value and the application data corresponding to the abnormal hash value in the first intermediate data table are located, and the application data corresponding to the abnormal hash value in the target application data table is updated according to the application data corresponding to the abnormal hash value in the first intermediate data table, so that the updated target application data table is synchronized with the first intermediate data table, thereby meeting various business needs corresponding to the target client.

[0147] In a specific implementation, S40 also includes the following content: if the first data processing mode remains unchanged, sending the first intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the first intermediate data table.

[0148] S50 also includes the following content: if the second data processing method is overall hash conversion, the application data in the first intermediate data table is overall hash converted according to a preset hash algorithm to obtain a target hash data table.

[0149] In the above, according to the number of data updates corresponding to the first intermediate data table, the number of selections and the number of exceptions corresponding to each reference field name within a preset time period, the hash conversion requirement degree corresponding to the first intermediate data table is obtained, and according to the hash conversion requirement degree and the preset requirement degree threshold, the first data processing method corresponding to the first intermediate data table is determined. If the first data processing method is hash conversion, then according to the total data volume corresponding to the first intermediate data table and the preset data volume threshold, the second data processing method corresponding to the first intermediate data table is determined. If the second data processing method is group hash conversion, then according to the sub-data volume corresponding to each reference field name and the preset number of groups, the target group length corresponding to each reference field name is obtained. According to the preset hash algorithm and the target group length corresponding to each reference field name, group hash conversion is performed on the application data in each reference field name to obtain the target Hash data table, sends the target hash data table to the target client. It can be known that according to the number of data updates corresponding to the first intermediate data table, the number of selections and the number of exceptions corresponding to each reference field name within the preset time period, the cost of verifying the application data in the first intermediate data table and the target application data table one by one, and the cost of performing hash conversion and then verifying the application data in the first intermediate data table and the target application data table are analyzed. Based on this, the processing form of the first intermediate data table is selected, and when the total data volume is less than the preset data volume threshold, the first intermediate data table is hash converted as a whole. When the total data volume is greater than or equal to the preset data volume threshold, the first intermediate data table is grouped and then hash converted. While ensuring the accuracy of the target hash data table, the efficiency of locating abnormal data can be improved during data consistency verification, thereby comprehensively improving the accuracy and efficiency of data synchronization.

[0150] Embodiment 3

[0151] The third embodiment provides a data transmission method, which includes: Figure 3 As shown:

[0152] S100, obtain the target application data table provided by the target client, the first intermediate data table corresponding to the target application data table stored on the server, the number of selections and the number of exceptions corresponding to each reference field name in the first intermediate data table within a preset time period, the reference field name corresponding to each target field name in the target application data table, and the comparison and verification results between the target application data table provided by the target client and the first intermediate data table, wherein the comparison and verification results include consistency and inconsistency.

[0153] In a specific implementation, when the comparison and verification result is consistent, the data related to the first intermediate data table is not sent to the target client to synchronously update the target data table.

[0154] S200, if the comparison and verification result is inconsistent, obtain the first number of comparison and verification results provided by the target client within a preset time period, the total amount of data corresponding to the first intermediate data table, and the number of data updates corresponding to the first intermediate data table within the preset time period.

[0155] The first number of comparison verification results provided by the target client within the preset time period can represent the number of verification requests made by the target client to the server within the preset time period. The number of data updates refers to the number of times the server actively updates the corresponding application data in the first intermediate data table.

[0156] In this embodiment, when the first number is larger, the total data volume is larger, and the data update times are more, the cost of sending the first intermediate data as a whole to the target client each time is higher, and the cost of sending the application data part corresponding to the exception-related data in the target application data table in the first intermediate data table to the target client is lower.

[0157] Therefore, this embodiment obtains the first number of comparison and verification results provided by the target client within a preset time period, the total amount of data corresponding to the first intermediate data table, and the number of data updates corresponding to the first intermediate data table within the preset time period, to measure the cost of sending the first intermediate data as a whole to the target client each time, so as to determine the overall demand degree of the target client for the first intermediate data.

[0158] S300, for any target field name in the target application data table, if the application data corresponding to the current target field name in the target application data table is inconsistent with the application data corresponding to the reference field name corresponding to the current target field name in the first intermediate data table, the current target field name is determined as an abnormal field name.

[0159] Among them, by traversing all target field names, all abnormal field names can be filtered out from all target field names as the basis for data sending.

[0160] S400, based on the reference field name corresponding to each abnormal field name, the number of selections and the number of abnormalities corresponding to each reference field name within a preset time period, the first quantity, the total data volume and the number of data updates, the overall demand degree of the target client for the first intermediate data is obtained.

[0161] In a specific implementation, S400 further includes the following steps:

[0162] S410, according to the reference field name corresponding to each abnormal field name, and the number of selections and the number of abnormalities corresponding to each reference field name within a preset time period, the abnormality degrees corresponding to all abnormal field names are obtained.

[0163] S420: Obtain the importance corresponding to the first intermediate data table according to the first quantity, the total data volume, and the number of data updates.

[0164] S430: Obtain the overall demand level corresponding to the first intermediate data table according to the abnormality level and the importance level.

[0165] Among them, the higher the abnormality degree corresponding to the abnormal field name and the higher the importance corresponding to the first intermediate data table, the higher the possibility and frequency of verifying and synchronizing the target application data table based on the first intermediate data table. Correspondingly, compared with sending the application data part corresponding to the abnormal related data in the target application data table in the first intermediate data table to the target client, sending the entire first intermediate data table to the target client each time can provide the target client with complete and accurate data, meet the various application requirements corresponding to the target client, and avoid the target client being unable to use the corresponding data normally due to too low data validity.

[0166] On the contrary, the lower the abnormality degree corresponding to the abnormal field name and the lower the importance corresponding to the first intermediate data table, the lower the possibility and frequency of verifying and synchronizing the target application data table based on the first intermediate data table. Correspondingly, compared with sending the entire first intermediate data table to the target client every time, sending the abnormal related data in the target application data table to the target client in the application data part corresponding to the first intermediate data table can improve the efficiency of data sending, reduce the cost of data sending, and improve the timeliness of data synchronization update by the target client.

[0167] Therefore, based on the degree of abnormality and importance, the overall demand level corresponding to the first intermediate data table is obtained, which is used to characterize the priority of sending the first intermediate data table as a whole to the target client. The sending method of the first intermediate data table is determined according to the overall demand level, thereby improving the efficiency of data sending while ensuring the validity of the data.

[0168] In a specific implementation, S410 further includes the following steps:

[0169] S411, for any abnormal field name, according to the number of selections and the number of abnormalities corresponding to the reference field name corresponding to the current abnormal field name within a preset time period, obtain the abnormal sub-degree corresponding to the current abnormal field name.

[0170] S412, traverse all abnormal field names, and obtain the abnormal sub-level corresponding to each abnormal field name.

[0171] S413: The sum of all abnormal sub-degrees is determined as the abnormal degree corresponding to all abnormal field names.

[0172] Among them, the average number of selections and the average number of exceptions corresponding to all initial field names corresponding to all initial application data tables within a preset time period are obtained, the ratio of the number of selections to the average number of selections is determined as the corresponding reference number of selections, and the ratio of the number of exceptions to the average number of exceptions is determined as the corresponding reference number of exceptions, thereby unifying the scales of the reference number, reference number of selections and reference number of exceptions, and improving the accuracy of the degree of abnormality.

[0173] The second preset weight corresponding to the number of selections and the third preset weight corresponding to the number of exceptions are obtained, and the reference number of selections and the reference number of exceptions are weighted and summed according to the second preset weight and the third preset weight to obtain the abnormal sub-degree corresponding to each abnormal field name.

[0174] The specific values ​​of the second preset weight and the third preset weight can be set by the implementer according to actual conditions.

[0175] In the above, based on the reference field name corresponding to each abnormal field name, the number of selections and the number of abnormalities corresponding to each reference field name within a preset time period, the first quantity, the total data volume and the number of data updates, the priority of sending the first intermediate data as a whole to the target client is measured, and the overall demand degree of the target client for the first intermediate data is obtained, which is used to determine the sending method of the first intermediate data to achieve efficient data sending effect.

[0176] S500, determining a data sending mode corresponding to the first intermediate data table according to the overall demand level and a preset overall demand level threshold, wherein the data sending mode includes sending overall data and sending abnormality-related data.

[0177] In a specific implementation, S500 includes the following steps:

[0178] S510: If the overall demand level is greater than a preset overall demand level threshold, determine that the data sending method corresponding to the first intermediate data table is to send the overall data.

[0179] S520: If the overall demand level is less than or equal to a preset overall demand level threshold, determine that the data sending method corresponding to the first intermediate data table is to send abnormality-related data.

[0180] S600, acquiring a data table to be sent according to a data sending mode, a first intermediate data table, and a reference field name corresponding to each abnormal field name.

[0181] In a specific implementation, S600 further includes the following steps:

[0182] S610: If the data sending method is to send the entire data, the first intermediate data table is determined as the data table to be sent.

[0183] S620: If the data sending mode is to send exception-related data, the position identifier of each exception field name in the target application data table is obtained.

[0184] S630: Determine the application data corresponding to the reference field name corresponding to each abnormal field name in the first intermediate data table as the first data to be sent.

[0185] S640, acquiring second data to be sent corresponding to each first data to be sent according to each first data to be sent and the corresponding position identifier of the abnormal field name corresponding to each first data to be sent in the target application data table.

[0186] S650: compose a table of data to be sent according to all the second data to be sent.

[0187] The position identifier is used to indicate the position of the abnormal field name in the target application data table. For example, the position identifier corresponding to each abnormal field name may refer to the row number or column number corresponding to each abnormal field name in the target application data table.

[0188] In the above, according to the data sending mode, the first intermediate data table and the reference field name corresponding to each abnormal field name, the data table to be sent is obtained as the data basis for synchronizing the target application data.

[0189] S700, sending the data table to be sent to the target client, wherein the target client is used to synchronize the target application data table according to the data table to be sent.

[0190] If the data table to be sent is the entire first intermediate data table, the first intermediate data table is used to replace the target application data table, so that the target application data table is synchronized with the first intermediate data table.

[0191] If the data table to be sent consists of the second data to be sent, each first data to be sent is used to replace the application data corresponding to the corresponding abnormal field name in the target application data, so that the target application data table is synchronized with the first intermediate data table.

[0192] As mentioned above, if the comparison verification result is inconsistent, then for any target field name in the target application data table, if the application data corresponding to the current target field name in the target application data is inconsistent with the application data corresponding to the reference field name corresponding to the current target field name in the first intermediate data table, then the current target field name is determined as an abnormal field name, and the overall demand degree of the target client for the first intermediate data is obtained according to the reference field name corresponding to each abnormal field name, the number of selections and the number of abnormalities corresponding to each reference field name within a preset time period, the first quantity, the total data volume and the number of data updates, and the data sending method corresponding to the first intermediate data table is determined according to the overall demand degree and the preset overall demand degree threshold. According to the data sending method, the first intermediate data table and the reference field name corresponding to each abnormal field name, the data table to be sent is obtained, and the data table to be sent is sent to the target client. It can be seen that by analyzing the data sending effect between the data sending form of sending the first intermediate data table as a whole to the target client each time and the data sending form of sending the application data part corresponding to the abnormal related data in the target application data table in the first intermediate data table to the target client, the priority of sending the first intermediate data table as a whole to the target client is measured, and the data sending method is determined accordingly and the corresponding data to be sent is sent to the target client, which improves the efficiency of data sending while ensuring the validity of the data.

[0193] Embodiment 4

[0194] The fourth embodiment provides a data synchronization device, the data synchronization device includes: Figure 4 As shown:

[0195] The data acquisition module 41 is used to acquire several initial application data tables stored on the server side, the first attribute information of each initial application data table, and the target application data table provided by the target client and the second attribute information of the target application data table, wherein the first attribute information includes the initial table name of the corresponding initial application data table, several initial field names and the field type corresponding to each initial field name, and the second attribute information includes the target table name of the target application data table, several target field names and the field type corresponding to each target field name.

[0196] The information matching module 42 is used to obtain the matching degree between the target application data table and each initial application data table according to the second attribute information of the target application data table and the first attribute information of each initial application data table.

[0197] The data table determination module 43 is used to determine the initial application data table corresponding to the maximum matching degree as the designated application data table corresponding to the target application data table.

[0198] The data table screening module 44 is used to screen the first intermediate data table from the designated application data table according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table.

[0199] The magnitude acquisition module 45 is used to acquire the data magnitude corresponding to the first intermediate data table according to the first intermediate data table.

[0200] The threshold comparison module 46 is used to compare the data level with a preset data level threshold to obtain a data comparison result.

[0201] The data table processing module 47 is used to process the first intermediate data table according to the preset data processing method corresponding to the data comparison result to obtain the second intermediate data table, wherein the preset data processing method includes keeping the first intermediate data table unchanged and performing group hash conversion on the first intermediate data table.

[0202] The data table sending module 48 is used to send the second intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the second intermediate data table.

[0203] In a specific implementation, the information matching module 42 further includes:

[0204] The first scoring submodule is used to obtain, for any initial application data table, a first matching score corresponding to the current initial application data table according to the initial table name and the target table name of the current initial application data table.

[0205] The second scoring submodule is used to obtain a second matching score corresponding to the current initial application data table based on several initial field names corresponding to the current initial application data table and the field type corresponding to each initial field name, as well as several target field names and the field type corresponding to each target field name.

[0206] The matching degree acquisition submodule is used to acquire the matching degree between the target application data table and the current initial application data table according to the first matching score and the second matching score.

[0207] In a specific implementation, the second scoring submodule further includes:

[0208] The first similarity calculation unit is used to calculate, for any target field name, a first similarity between the current target field name and each initial field name in the current initial application data table.

[0209] The second similarity calculation unit is used to perform consistency matching on the field type corresponding to the current target field name and the field type corresponding to each initial field name in the current initial application data table, and obtain the second similarity between the current target field name and each initial field name in the current initial application data table.

[0210] The third similarity calculation unit is used to obtain a third similarity between the current target field name and each initial field name in the current initial application data table according to the first similarity and the second similarity.

[0211] The matching degree calculation unit is used to determine the maximum value of the third similarities between the current target field name and all the initial field names in the current initial application data table as the matching degree between the current initial application data table and the current target field name.

[0212] The matching score determination unit is used to traverse all target field names and determine the average value of the matching degree between the current initial application data table and all target field names as the second matching score corresponding to the current initial application data table.

[0213] In a specific implementation, the data table screening module 44 further includes:

[0214] The similarity acquisition submodule is used to acquire, for any target field name, the third similarity between the current target field name and each initial field name in the specified application data table.

[0215] The field name determination submodule is used to determine the initial field name corresponding to the largest third similarity in the specified application data table as the reference field name corresponding to the current target field name.

[0216] The field name traversal submodule is used to traverse all target field names and obtain the reference field name corresponding to each target field name.

[0217] The application data filtering submodule is used to filter out the application data corresponding to each reference field name from the specified application data table.

[0218] The first data table forming submodule is used to form a first intermediate data table according to all the extracted application data.

[0219] In a specific implementation, the data comparison result includes that the data level is greater than a preset data level threshold and the data level is less than or equal to the preset data level threshold, and the data table processing module 47 further includes:

[0220] The data table processing submodule is used to keep the first intermediate data table unchanged and determine the first intermediate data table as the second intermediate data table if the data comparison result is that the data level is less than or equal to the preset data level threshold.

[0221] The data volume acquisition submodule is used for, if the data comparison result is that the data volume is greater than the preset data volume threshold, the preset data processing method corresponding to the data comparison result is to perform group hash conversion on the first intermediate data table to obtain the second intermediate data table, wherein the data volume acquisition submodule also includes:

[0222] The first data volume acquisition unit is used to acquire the first data volume of the application data corresponding to each reference field name in the first intermediate data table.

[0223] The packet length acquisition unit is used to acquire, for any reference field name, a reference packet length corresponding to the current reference field name according to the first data volume corresponding to the current reference field name and a preset number of packets.

[0224] The application data grouping unit is used to group the application data corresponding to the current reference field name according to the reference group length corresponding to the current reference field name, and obtain all reference application data groups corresponding to the current reference field name.

[0225] The hash processing unit is used to process each reference application data group according to a preset hash algorithm to obtain reference hash data corresponding to each reference application data group.

[0226] The reference field traversal unit is used to traverse all reference fields and obtain all reference hash data.

[0227] The second data table forming unit is used to form a second intermediate data table according to all the reference hash data.

[0228] In a specific implementation, the magnitude acquisition module 45 further includes:

[0229] The second data volume acquisition submodule is used to acquire the second data volume of the application data in the first intermediate data table.

[0230] The logarithmic processing submodule is used to perform a logarithmic operation on the second data volume to obtain the data magnitude corresponding to the first intermediate data table.

[0231] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0232] Embodiment 5

[0233] Embodiment 5 of the present invention provides a non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the steps:

[0234] S1, obtains several initial application data tables stored on the server side, the first attribute information of each initial application data table, and the target application data table provided by the target client and the second attribute information of the target application data table, wherein the first attribute information includes the initial table name corresponding to the initial application data table, several initial field names and the field type corresponding to each initial field name, and the second attribute information includes the target table name of the target application data table, several target field names and the field type corresponding to each target field name.

[0235] S2: Acquire a matching degree between the target application data table and each initial application data table according to the second attribute information of the target application data table and the first attribute information of each initial application data table.

[0236] S3, determining the initial application data table corresponding to the maximum matching degree as the designated application data table corresponding to the target application data table.

[0237] S4, according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table, a first intermediate data table is obtained by screening from the designated application data table.

[0238] S5. According to the first intermediate data table, obtain the data magnitude corresponding to the first intermediate data table.

[0239] S6, comparing the data level with a preset data level threshold to obtain a data comparison result.

[0240] S7, processing the first intermediate data table according to a preset data processing method corresponding to the data comparison result to obtain a second intermediate data table, wherein the preset data processing method includes keeping the first intermediate data table unchanged and performing group hash conversion on the first intermediate data table.

[0241] S8, sending the second intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the second intermediate data table.

[0242] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchronization Link) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0243] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0244] Embodiment 6

[0245] A sixth embodiment of the present invention provides an electronic device, which includes a processor and the non-transitory computer-readable storage medium in the fifth embodiment of the present invention.

[0246] The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A data synchronization method, characterized in that: The data synchronization method comprises the following steps: S1, obtaining a plurality of initial application data tables stored on the server side, first attribute information of each initial application data table, and a target application data table provided by a target client and second attribute information of the target application data table, wherein the first attribute information includes an initial table name corresponding to the initial application data table, a plurality of initial field names, and a field type corresponding to each initial field name, and the second attribute information includes a target table name of the target application data table, a plurality of target field names, and a field type corresponding to each target field name; S2, acquiring a matching degree between the target application data table and each initial application data table according to the second attribute information of the target application data table and the first attribute information of each initial application data table; S3, determining the initial application data table corresponding to the maximum matching degree as the designated application data table corresponding to the target application data table; S4, according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table, filter out a first intermediate data table from the designated application data table; S5, acquiring the data magnitude corresponding to the first intermediate data table according to the first intermediate data table; S6, comparing the data level with a preset data level threshold to obtain a data comparison result; S7, according to the preset data processing method corresponding to the data comparison result, the first intermediate data table is processed to obtain a second intermediate data table, wherein the preset data processing method includes keeping the first intermediate data table unchanged and performing group hash conversion on the first intermediate data table, and the data comparison result includes data magnitude greater than a preset data magnitude threshold and data magnitude less than or equal to a preset data magnitude threshold, and S7 also includes the following steps: S71, if the data comparison result is that the data magnitude is less than or equal to a preset data magnitude threshold, the preset data processing method corresponding to the data comparison result is to keep the first intermediate data table unchanged and determine the first intermediate data table as the second intermediate data table; S72, if the data comparison result is that the data magnitude is greater than the preset data magnitude threshold, the preset data processing method is to perform group hash conversion on the first intermediate data table to obtain a second intermediate data table, wherein S72 also includes the following steps: S721, obtaining a first data volume of application data corresponding to each reference field name in the first intermediate data table; S722, for any reference field name, according to the first data volume corresponding to the current reference field name and the preset number of groups, obtain a reference group length corresponding to the current reference field name; S723, grouping the application data corresponding to the current reference field name according to the reference group length corresponding to the current reference field name, and acquiring all reference application data groups corresponding to the current reference field name; S724, processing each reference application data group according to a preset hash algorithm to obtain reference hash data corresponding to each reference application data group; S725, traverse all reference fields to obtain all reference hash data; S726, forming a second intermediate data table according to all the reference hash data; S8, sending the second intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the second intermediate data table.

2. The data synchronization method according to claim 1, characterized in that: S2 also includes the following steps: S21, for any initial application data table, obtaining a first matching score corresponding to the current initial application data table according to the initial table name of the current initial application data table and the target table name; S22, obtaining a second matching score corresponding to the current initial application data table according to a plurality of initial field names corresponding to the current initial application data table and a field type corresponding to each initial field name, and a plurality of target field names and a field type corresponding to each target field name; S23: Acquire a matching degree between the target application data table and the current initial application data table according to the first matching score and the second matching score.

3. The data synchronization method according to claim 2, characterized in that: S22 also includes the following steps: S221, for any target field name, calculating a first similarity between the current target field name and each initial field name in the current initial application data table; S222, performing consistency matching on the field type corresponding to the current target field name and the field type corresponding to each initial field name in the current initial application data table, and obtaining a second similarity between the current target field name and each initial field name in the current initial application data table; S223, acquiring a third similarity between the current target field name and each initial field name in the current initial application data table according to the first similarity and the second similarity; S224, determining the maximum value of the third similarities between the current target field name and all the initial field names in the current initial application data table as the matching degree between the current initial application data table and the current target field name; S225, traversing all target field names, and determining the average value of the matching degree between the current initial application data table and all target field names as the second matching score corresponding to the current initial application data table.

4. The data synchronization method according to claim 3, characterized in that: S4 also includes the following steps: S41, for any target field name, obtaining a third similarity between the current target field name and each initial field name in the specified application data table; S42, determining the initial field name corresponding to the largest third similarity in the specified application data table as the reference field name corresponding to the current target field name; S43, traverse all target field names, and obtain the reference field name corresponding to each target field name; S44, filtering out application data corresponding to each reference field name from the specified application data table; S45, forming a first intermediate data table according to all the extracted application data.

5. The data synchronization method according to claim 1, characterized in that: S5 also includes the following steps: S51, acquiring a second data volume of application data in the first intermediate data table; S52: Perform a logarithm operation on the second data volume to obtain a data volume level corresponding to the first intermediate data table.

6. The data synchronization method according to claim 1, characterized in that: The target client is used to compare and verify the second intermediate data table and the target application data table, and update the target application data table according to the comparison and verification result, so that the updated target application data table is synchronized with the first intermediate data table.

7. A data synchronization device, characterized in that: The data synchronization device comprises: A data acquisition module, used to acquire a plurality of initial application data tables stored on the server side, first attribute information of each initial application data table, and a target application data table provided by a target client and second attribute information of the target application data table, wherein the first attribute information includes an initial table name corresponding to the initial application data table, a plurality of initial field names, and a field type corresponding to each initial field name, and the second attribute information includes a target table name of the target application data table, a plurality of target field names, and a field type corresponding to each target field name; an information matching module, configured to obtain a matching degree between the target application data table and each initial application data table according to the second attribute information of the target application data table and the first attribute information of each initial application data table; A data table determination module, used to determine the initial application data table corresponding to the maximum matching degree as the designated application data table corresponding to the target application data table; A data table screening module, used for screening the first intermediate data table from the designated application data table according to the target field name corresponding to the target application data table and the initial field name corresponding to the designated application data table; A magnitude acquisition module, configured to acquire the data magnitude corresponding to the first intermediate data table according to the first intermediate data table; A threshold comparison module, used to compare the data level with a preset data level threshold to obtain a data comparison result; A data table processing module, used to process the first intermediate data table according to a preset data processing method corresponding to the data comparison result, and obtain a second intermediate data table, wherein the preset data processing method includes keeping the first intermediate data table unchanged and performing group hash conversion on the first intermediate data table, and the data comparison result includes a data magnitude greater than a preset data magnitude threshold and a data magnitude less than or equal to a preset data magnitude threshold, and the data table processing module also includes: A data table processing submodule, for, if the data comparison result is that the data magnitude is less than or equal to a preset data magnitude threshold, then the preset data processing method corresponding to the data comparison result is to keep the first intermediate data table unchanged and determine the first intermediate data table as the second intermediate data table; A data volume acquisition submodule is used for, if the data comparison result is that the data volume is greater than a preset data volume threshold, the preset data processing method is to perform group hash conversion on the first intermediate data table to obtain a second intermediate data table, wherein the data volume acquisition submodule also includes: A first data volume acquisition unit, configured to acquire a first data volume of application data corresponding to each reference field name in the first intermediate data table; A packet length acquisition unit, configured to acquire, for any reference field name, a reference packet length corresponding to the current reference field name according to the first data volume corresponding to the current reference field name and a preset number of packets; An application data grouping unit, used to group the application data corresponding to the current reference field name according to the reference group length corresponding to the current reference field name, and obtain all reference application data groups corresponding to the current reference field name; A hash processing unit, used to process each reference application data group according to a preset hash algorithm to obtain reference hash data corresponding to each reference application data group; The reference field traversal unit is used to traverse all reference fields and obtain all reference hash data; A second data table forming unit, used for forming a second intermediate data table according to all reference hash data; A data table sending module is used to send the second intermediate data table to the target client, wherein the target client is used to verify and synchronize the target application data table according to the second intermediate data table.

8. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the non-transitory computer-readable storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the data synchronization method as described in any one of claims 1-6.

9. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 8.

Images