A data compensation method, device and medium based on data quality assessment

Abstract

The present invention discloses a data compensation method, device and medium based on data quality assessment, and relates to the technical field of data compensation. The method is applied to data compensation between two distributed systems, including: sampling data within a specified time period between the two systems to obtain multiple data items; respectively evaluating the data quality score of each data item in the two systems, and the data quality score at least reflects the integrity, accuracy and timeliness of the data; according to the data quality score of each data item in the two systems, determining whether the data item needs to be compensated, as well as the direction and method of compensation, and then performing corresponding compensation operations to copy the data items that need to be compensated from one system to another. By sampling the data and performing data quality score assessment, the present invention does not need to maintain additional metadata between the two systems, reduces the storage and transmission overhead, and simplifies the complexity and uncertainty.

Description

Technical Field

[0001] The present invention relates to the technical field of data compensation, and in particular to a data compensation method, device and medium based on data quality assessment. Background Art

[0002] In a distributed system, different systems need to exchange data to maintain business consistency and coordination. However, due to network delays, failures, anomalies, etc., the data between different systems may be inconsistent or incomplete, resulting in business logic errors or reduced user experience. To solve this problem, it is usually necessary to compensate the data between different systems, that is, by comparing the data differences between the two systems, copy the missing or erroneous data from one system to another.

[0003] Traditional data compensation methods usually determine whether the data between two systems is consistent based on timestamps or version numbers, and then perform compensation operations according to pre-defined rules. This method has the following disadvantages: 1. It is necessary to maintain additional metadata between the two systems, such as timestamps or version numbers, which increases storage and transmission overhead. 2. It is necessary to manually define compensation rules, such as how to choose the compensation direction, when to trigger compensation, how to deal with conflicts, etc., which increases complexity and uncertainty. 3. It is not possible to dynamically compensate based on the quality of the data itself, such as the integrity, accuracy, timeliness, etc. of the data, which may result in compensation results that do not meet user or business expectations. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data compensation method, device and medium based on data quality assessment. By sampling the data and performing data quality score assessment, there is no need to maintain additional metadata between the two systems, thereby reducing storage and transmission overhead; there is no need to manually define compensation rules, thereby simplifying complexity and uncertainty.

[0005] In a first aspect, the present invention provides a data compensation method based on data quality assessment, which is applied to data compensation between two distributed systems, and includes the following steps:

[0006] Step 10: sampling the data within a specified time period between the two systems to obtain multiple data items;

[0007] Step 20: Evaluate the data quality score of each data item in the two systems respectively, wherein the data quality score at least reflects the completeness, accuracy and timeliness of the data;

[0008] Step 30: Determine whether the data item needs to be compensated, as well as the direction and method of compensation, based on the data quality score of each data item in the two systems, and then perform the corresponding compensation operation to copy the data item that needs to be compensated from one system to another.

[0009] Furthermore, the step 20 specifically includes:

[0010] Evaluate the completeness score: The completeness score reflects whether a data item exists in a system and contains all necessary attributes;

[0011] Evaluate accuracy score: Accuracy reflects whether the value of a data item in a system is correct or reasonable;

[0012] Evaluate the timeliness score: The timeliness score reflects whether a data item is updated in a system in a timely manner;

[0013] The data quality score is obtained by calculating the weighted average of the completeness score, accuracy score, and timeliness score. The weight of each score is adjustable.

[0014] Furthermore, the step 30 specifically includes:

[0015] If a data item exists in both systems and can be scored, the data quality scores in the two systems are compared; if the difference in the data quality scores between the two systems is less than the set value, no compensation is performed; if the difference in the data quality scores between the two systems is greater than or equal to the set value, the data item is copied from the system with the higher data quality score to the system with the lower data quality score;

[0016] If a data item exists in one system and can be scored, but does not exist or cannot be scored in another system, copy the data item from the system where it exists and can be scored to the other system;

[0017] If a data item cannot be scored in both systems, it is ignored.

[0018] Furthermore, in step 10, the time period and the sampling ratio are set according to the system data consistency requirement.

[0019] In a second aspect, the present invention provides a data compensation device based on data quality assessment, which is applied to data compensation between two distributed systems, comprising:

[0020] A data sampling module is used to sample data within a specified time period between two systems to obtain multiple data items;

[0021] A data quality assessment module, used to assess the data quality score of each data item in the two systems respectively, wherein the data quality score at least reflects the completeness, accuracy and timeliness of the data;

[0022] The data compensation decision module is used to determine whether the data item needs to be compensated, as well as the direction and method of compensation based on the data quality score of each data item in the two systems, and then perform the corresponding compensation operation to copy the data items that need to be compensated from one system to another.

[0023] Furthermore, the data quality assessment module is specifically used for:

[0024] Evaluate the completeness score: The completeness score reflects whether a data item exists in a system and contains all necessary attributes;

[0025] Evaluate accuracy score: Accuracy reflects whether the value of a data item in a system is correct or reasonable;

[0026] Evaluate the timeliness score: The timeliness score reflects whether a data item is updated in a system in a timely manner;

[0027] The data quality score is obtained by calculating the weighted average of the completeness score, accuracy score, and timeliness score. The weight of each score is adjustable.

[0028] Furthermore, the data compensation decision module is specifically used to:

[0029] If a data item exists in both systems and can be scored, the data quality scores in the two systems are compared; if the difference in the data quality scores between the two systems is less than the set value, no compensation is performed; if the difference in the data quality scores between the two systems is greater than or equal to the set value, the data item is copied from the system with the higher data quality score to the system with the lower data quality score;

[0030] If a data item exists in one system and can be scored, but does not exist or cannot be scored in another system, copy the data item from the system where it exists and can be scored to the other system;

[0031] If a data item cannot be scored in both systems, it is ignored.

[0032] Furthermore, in the data sampling module, the time period and the sampling ratio are set according to the system data consistency requirements.

[0033] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0035] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects:

[0036] 1. No need to maintain additional metadata between the two systems, reducing storage and transmission overhead;

[0037] 2. No need to manually define compensation rules, which simplifies complexity and uncertainty;

[0038] 3. Compensation can be made dynamically based on the quality of the data itself, increasing the probability that the compensation result meets user or business expectations.

[0039] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0041] Figure 1 It is a schematic diagram of the framework of the system of the present invention;

[0042] Figure 2 is a flow chart of the method in Embodiment 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of data compensation based on quantity and quality evaluation in Embodiment 1 of the present invention;

[0044] Figure 4 It is a structural schematic diagram of the device in the second embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device in Embodiment 3 of the present invention;

[0046] Figure 6 Schematic diagram of the structure of the medium in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0047] The embodiments of the present invention provide a data compensation method, device and medium based on data quality assessment. By sampling data and performing data quality score assessment, there is no need to maintain additional metadata between the two systems, thereby reducing storage and transmission overhead; there is no need to manually define compensation rules, thereby simplifying complexity and uncertainty.

[0048] The technical solution in the embodiment of the present invention has the following general idea:

[0049] 1. Sample the data that needs to be compensated between the two systems, and calculate the data quality score of each data item in the two systems. The score reflects the comprehensive evaluation of the data's completeness, accuracy, timeliness, etc. 2. According to the data quality score of each data item in the two systems, determine whether the data item needs to be compensated, as well as the direction and method of compensation, and then perform the corresponding compensation operation to copy the data items that need to be compensated from one system to another. If the score of a data item in one system is higher than that in another system, copy the data item from the system with the higher score to the system with the lower score; if the score of a data item in the two systems is not much different, no compensation is performed; if a data item cannot be scored in both systems, ignore the data item. 3. According to the determined compensation direction and method, perform the corresponding compensation operation to copy the missing data from one system to another.

[0050] Before introducing the specific embodiments, the system framework corresponding to the method of the embodiment of the present application is first introduced. Figure 1 As shown, there are two distributed systems with independent businesses: system A and system B. The data of system A and system B can be created separately.

[0051] Embodiment 1

[0052] This embodiment provides a data compensation method based on data quality assessment, which is applied to data compensation between two distributed systems. Figure 2 As shown, including:

[0053] Step 10: sampling the data within a specified time period between the two systems to obtain multiple data items;

[0054] Step 20: Evaluate the data quality score of each data item in the two systems respectively, wherein the data quality score at least reflects the completeness, accuracy and timeliness of the data;

[0055] Step 30: Determine whether the data item needs to be compensated, as well as the direction and method of compensation, based on the data quality score of each data item in the two systems, and then perform the corresponding compensation operation to copy the data item that needs to be compensated from one system to another.

[0056] In a possible implementation, step 20 specifically includes:

[0057] Evaluation integrity score: The integrity score reflects whether a data item exists in a system and whether it contains all necessary attributes. If a data item does not exist in a system, its integrity score is 0. If a data item exists in a system but lacks some necessary attributes, its integrity score is the ratio of the number of attributes of the data item to the number of necessary attributes. If a data item exists in a system and contains all necessary attributes, its integrity score is 1.

[0058] Evaluation accuracy score: Accuracy reflects whether the value of a data item in a system is correct or reasonable (for example, some values ​​have a normal range of values, and if they deviate too far, the accuracy is low); if the value of a data item in a system is inconsistent with the facts or standards, its accuracy score is 0; if the value of a data item in a system is consistent with the facts or standards, its accuracy score is 1.

[0059] Evaluate the timeliness score: The timeliness score reflects whether a data item is updated in a system in a timely manner. If the update time of a data item in a system exceeds a certain threshold from the current time, its timeliness score is 0. If the update time of a data item in a system is less than a certain threshold from the current time, its timeliness score is 1.

[0060] The data quality score is obtained by calculating the weighted average of the completeness score, accuracy score, and timeliness score. The weight of each score is adjustable, so that compensation can be made dynamically based on the quality of the data itself, increasing the probability that the compensation result meets user or business expectations.

[0061] In one possible implementation, Figure 3 As shown, the data items of system A can be retrieved in system B (not in the sampled data) first, and then the data items of system B can be retrieved in system A. During the entire evaluation process, the evaluated data will be recorded. For example, if a data item extracted from system A is retrieved in system B, this data item will be recorded as evaluated data. In this way, when detecting the sampled data of system B, if this data item is encountered, it will be skipped and will not be retrieved in system A again.

[0062] The step 30 specifically includes:

[0063] If a data item exists in both systems and can be scored, the data quality scores in the two systems are compared; if the difference in the data quality scores between the two systems is less than the set value, no compensation is performed; if the difference in the data quality scores between the two systems is greater than or equal to the set value, the data item is copied from the system with the higher data quality score to the system with the lower data quality score;

[0064] If a data item exists in one system and can be scored, but does not exist or cannot be scored in another system, copy the data item from the system where it exists and can be scored to the other system;

[0065] If a data item cannot be scored in both systems, it is ignored.

[0066] In one possible implementation, in step 10, the time period and the sampling ratio are set according to the system data consistency requirements. For example, when the system data consistency requirements are high, the sampling ratio can be increased, for example, the sampling ratio is set to 100%. However, in actual operation, by not requiring the system data to be completely consistent, in order to reduce computing power consumption, the sampling ratio can be set to the 70%-80% range, and then the time period for multiple executions of data compensation is set to a repeated interval, such as performing data compensation for one month, one quarter, and one year. Due to the randomness of sampling, for data within one month, after three samplings of the month, quarter, and year, the actual proportion of data sampled will be higher than the proportion of a single sampling, and the system data consistency requirements can also be met.

[0067] The method of the embodiment of the present invention does not need to maintain additional metadata between the two systems, reducing storage and transmission overhead; does not need to manually define compensation rules, simplifying complexity and uncertainty; and can dynamically perform compensation based on the quality of the data itself, increasing the probability that the compensation result meets user or business expectations.

[0068] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0069] Embodiment 2

[0070] In this embodiment, a data compensation device based on data quality assessment is provided, which is applied to data compensation between two distributed systems. Figure 4 As shown, including:

[0071] A data sampling module is used to sample data within a specified time period between two systems to obtain multiple data items;

[0072] A data quality assessment module, used to assess the data quality score of each data item in the two systems respectively, wherein the data quality score at least reflects the completeness, accuracy and timeliness of the data;

[0073] The data compensation decision module is used to determine whether the data item needs to be compensated, as well as the direction and method of compensation based on the data quality score of each data item in the two systems, and then perform the corresponding compensation operation to copy the data items that need to be compensated from one system to another.

[0074] In a possible implementation, the data quality assessment module is specifically used for:

[0075] Evaluate the completeness score: The completeness score reflects whether a data item exists in a system and contains all necessary attributes;

[0076] Evaluate accuracy score: Accuracy reflects whether the value of a data item in a system is correct or reasonable;

[0077] Evaluate the timeliness score: The timeliness score reflects whether a data item is updated in a system in a timely manner;

[0078] The data quality score is obtained by calculating the weighted average of the completeness score, accuracy score, and timeliness score. The weight of each score is adjustable.

[0079] In a possible implementation, the data compensation decision module is specifically used to:

[0080] If a data item exists in both systems and can be scored, the data quality scores in the two systems are compared; if the difference in the data quality scores between the two systems is less than the set value, no compensation is performed; if the difference in the data quality scores between the two systems is greater than or equal to the set value, the data item is copied from the system with the higher data quality score to the system with the lower data quality score;

[0081] If a data item exists in one system and can be scored, but does not exist or cannot be scored in another system, copy the data item from the system where it exists and can be scored to the other system;

[0082] If a data item cannot be scored in both systems, it is ignored.

[0083] In a possible implementation, in the data sampling module, the time period and the sampling ratio are set according to the system data consistency requirement.

[0084] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0085] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0086] Embodiment 3

[0087] This embodiment provides an electronic device, such as Figure 5 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0088] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0089] Based on the same inventive concept, the present application provides a storage medium corresponding to the first embodiment, see the fourth embodiment for details.

[0090] Embodiment 4

[0091] This embodiment provides a computer-readable storage medium, such as Figure 6 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, any implementation method in Example 1 can be implemented.

[0092] Since the computer-readable storage medium introduced in this embodiment is a computer-readable storage medium used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the computer-readable storage medium of the present embodiment and its various variations, so how the computer-readable storage medium implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the computer-readable storage medium used in the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0093] The embodiments of the present invention do not require the maintenance of additional metadata between the two systems, thus reducing storage and transmission overheads; do not require manual definition of compensation rules, thus simplifying complexity and uncertainty; and can dynamically perform compensation based on the quality of the data itself, thus increasing the probability that the compensation result meets user or business expectations.

[0094] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A data compensation method based on data quality assessment, characterized in that: Applied to data compensation between two distributed systems, including the following steps: Step 10: sampling the data within a specified time period between the two systems to obtain multiple data items; Step 20: Evaluate the data quality score of each data item in the two systems respectively, wherein the data quality score at least reflects the completeness, accuracy and timeliness of the data; Step 30: Determine whether the data item needs to be compensated, as well as the direction and method of compensation, based on the data quality score of each data item in the two systems, and then perform the corresponding compensation operation to copy the data item that needs to be compensated from one system to another system; Wherein, the step 30 specifically includes: If a data item exists in both systems and can be scored, the data quality scores in the two systems are compared; if the difference in the data quality scores between the two systems is less than the set value, no compensation is performed; if the difference in the data quality scores between the two systems is greater than or equal to the set value, the data item is copied from the system with the higher data quality score to the system with the lower data quality score; If a data item exists in one system and can be scored, but does not exist or cannot be scored in another system, copy the data item from the system where it exists and can be scored to the other system; If a data item cannot be scored in both systems, it is ignored.

2. The method according to claim 1, characterized in that: The step 20 specifically includes: Evaluate the completeness score: The completeness score reflects whether a data item exists in a system and contains all necessary attributes; Evaluate accuracy score: Accuracy reflects whether the value of a data item in a system is correct or reasonable; Evaluate the timeliness score: The timeliness score reflects whether a data item is updated in a system in a timely manner; The data quality score is obtained by calculating the weighted average of the completeness score, accuracy score, and timeliness score. The weight of each score is adjustable.

3. The method according to claim 1, characterized in that: In step 10, the time period and the sampling ratio are set according to the system data consistency requirements.

4. A data compensation device based on data quality assessment, characterized in that: Applied to data compensation between two distributed systems, including: A data sampling module is used to sample data within a specified time period between two systems to obtain multiple data items; A data quality assessment module, used to assess the data quality score of each data item in the two systems respectively, wherein the data quality score at least reflects the completeness, accuracy and timeliness of the data; The data compensation decision module is used to determine whether the data item needs to be compensated, as well as the direction and method of compensation according to the data quality score of each data item in the two systems, and then perform the corresponding compensation operation to copy the data item that needs to be compensated from one system to another system; Wherein, the data compensation decision module is specifically used for: If a data item exists in both systems and can be scored, the data quality scores in the two systems are compared; if the difference in the data quality scores between the two systems is less than the set value, no compensation is performed; if the difference in the data quality scores between the two systems is greater than or equal to the set value, the data item is copied from the system with the higher data quality score to the system with the lower data quality score; If a data item exists in one system and can be scored, but does not exist or cannot be scored in another system, copy the data item from the system where it exists and can be scored to the other system; If a data item cannot be scored in both systems, it is ignored.

5. The device according to claim 4, characterized in that: The data quality assessment module is specifically used for: Evaluate the completeness score: The completeness score reflects whether a data item exists in a system and contains all necessary attributes; Evaluate accuracy score: Accuracy reflects whether the value of a data item in a system is correct or reasonable; Evaluate the timeliness score: The timeliness score reflects whether a data item is updated in a system in a timely manner; The data quality score is obtained by calculating the weighted average of the completeness score, accuracy score, and timeliness score. The weight of each score is adjustable.

6. The device according to claim 4, characterized in that: In the data sampling module, the time period and the sampling ratio are set according to the system data consistency requirements.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images