Method and device for detecting and repairing to-be-repaired data, computer equipment and readable storage medium

Abstract

The invention relates to a to-be-repaired data detection and repair method and device, computer equipment and a readable storage medium. The method comprises the following steps: detecting to-be-repaired data from business data, and extracting initial repair target data from the to-be-repaired data according to the detected data features of the to-be-repaired data; processing the initial repair target data, and detecting first repair target data and second repair target data in the processed initial repair target data; under the condition that the to-be-repaired type of the first repair target data is a logic type, generating a repair suggestion file matched with the data feature of the first repair target data; updating a data processing component of the first repair target data based on the repair suggestion file, and performing secondary processing on the first repair target data based on the updated data processing component; and obtaining the repaired target business data. By adopting the method, data recovery can be efficiently carried out.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for detecting and repairing data to be repaired. Background Art

[0002] With the development of science and technology, the requirements for data quality are getting higher and higher. By repairing data, the integrity, accuracy, completeness and consistency of the data can be ensured. Therefore, detecting and repairing the data to be repaired is a key link in ensuring data quality.

[0003] In traditional technology, the detection and repair of data to be repaired is generally achieved through historical experience. That is to say, first, developers rely on historical experience in discovering data to be repaired, check the data, and determine whether there is data to be repaired in the data. If there is data to be repaired, based on historical experience in repairing data and business domain expertise, the existing data to be repaired are repaired one by one.

[0004] However, the current methods for detecting and repairing data to be repaired are not efficient enough. Summary of the invention

[0005] Based on this, it is necessary to provide an efficient method, apparatus, computer device, computer-readable storage medium and computer program product for detecting and repairing data to be repaired in order to solve the above technical problems.

[0006] In a first aspect, the present application provides a method for detecting and repairing data to be repaired, comprising:

[0007] Detecting the data to be repaired from the business data, and extracting initial repair target data from the data to be repaired according to the data features of the detected data to be repaired;

[0008] Processing the initial repair target data, and detecting first repair target data and second repair target data in the processed initial repair target data, wherein the first repair target data is data that indicates that the first repair state has not been successfully repaired, and the second repair target data is data that indicates that the first repair state has been successfully repaired;

[0009] When the to-be-repaired type of the first repair target data is a logical type, generating a repair suggestion file matching the data features of the first repair target data;

[0010] Based on the repair suggestion file, the data processing component of the first repair target data is updated, and the first repair target data is reprocessed based on the updated data processing component;

[0011] The second-processed first repair target data and the second repair target data are used as the target business data after the repair is completed.

[0012] In one embodiment, detecting data to be repaired from business data includes:

[0013] In response to a configuration operation on a data verification rule, obtaining at least one of a database design configuration rule, a business configuration rule, and a data quality configuration rule;

[0014] According to at least one of database design configuration rules, business configuration rules, and data quality configuration rules, data to be repaired is detected from business data.

[0015] In one embodiment, when the problem type of the first repair target data is a logic type, generating a repair suggestion file matching the data feature of the first repair target data includes:

[0016] In the case where the problem type of the first repair target data is a logical type, generating an initial repair suggestion file matching the data characteristics of the repair target data;

[0017] Acquire blood relationship data of the first repair target data, the blood relationship data being obtained from the first repair target data and associated data of the first repair target data;

[0018] From the blood relationship data, select the program field, affected data range and verification time corresponding to the first repair target data;

[0019] Based on the program field, the affected data range, and the verification time, the field content of the dynamic data field in the initial repair suggestion file is updated, and the updated initial repair suggestion file is used as the repair suggestion file.

[0020] In one embodiment, extracting initial repair target data from the data to be repaired according to data features of the data to be repaired includes:

[0021] Obtaining a second repair state of the data to be repaired;

[0022] When the second repair state represents that repair is not performed, the data to be repaired is sampled according to data features of the data to be repaired to obtain initial repair target data.

[0023] In one embodiment, the second processed first repair target data is used as the repair completed target business data, including:

[0024] Evaluate the first repair target data of the secondary processing to generate a data repair evaluation result of the first repair target data;

[0025] When the data repair evaluation result indicates that the repair is successful, the secondarily processed first repair target data is used as the target business data after the repair is completed.

[0026] In one embodiment, detecting the first repair target data and the second repair target data in the processed initial repair target data includes:

[0027] Acquire first data repair information, and determine a first repair state of the processed initial repair target data, wherein the first data repair information is data repair information representing a fourth repair state of the initial repair target data without performing repair;

[0028] generating second data repair information when the first repair state indicates that the repair has been performed;

[0029] In the case that the first data repair information is inconsistent with the second data repair information, detecting the processed initial repair target data according to at least one of the database design configuration rule, the business configuration rule and the data quality configuration rule, and updating the first repair state based on the detected result;

[0030] In a case where the updated first repair state indicates that the repair is not successful, determining the processed initial repair target data as the first repair target data;

[0031] When the updated first repair state indicates that the repair is successful, the processed initial repair target data is determined as the second repair target data.

[0032] In a second aspect, the present application also provides a device for detecting and repairing data to be repaired, comprising:

[0033] A data acquisition module is used to detect the data to be repaired from the business data, and extract the initial repair target data from the data to be repaired according to the data characteristics of the detected data to be repaired;

[0034] A processing module, used to process the initial repair target data, and detect first repair target data and second repair target data in the processed initial repair target data, wherein the first repair target data is data that indicates that the first repair state has not been successfully repaired, and the second repair target data is data that indicates that the first repair state has been successfully repaired;

[0035] A first logic processing module, configured to generate a repair suggestion file matching the data features of the first repair target data when the to-be-repaired type of the first repair target data is a logic type;

[0036] A second logic processing module is used to update the data processing component of the first repair target data based on the repair suggestion file, and perform secondary processing on the first repair target data based on the updated data processing component;

[0037] The repair completion module is used to use the first repair target data and the second repair target data of the secondary processing as the target business data of the repair completion.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Detecting the data to be repaired from the business data, and extracting initial repair target data from the data to be repaired according to the data features of the detected data to be repaired;

[0040] Processing the initial repair target data, and detecting first repair target data and second repair target data in the processed initial repair target data, wherein the first repair target data is data that indicates that the first repair state has not been successfully repaired, and the second repair target data is data that indicates that the first repair state has been successfully repaired;

[0041] When the to-be-repaired type of the first repair target data is a logical type, generating a repair suggestion file matching the data features of the first repair target data;

[0042] Based on the repair suggestion file, the data processing component of the first repair target data is updated, and the first repair target data is reprocessed based on the updated data processing component;

[0043] The second-processed first repair target data and the second repair target data are used as the target business data after the repair is completed.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0045] Detecting the data to be repaired from the business data, and extracting initial repair target data from the data to be repaired according to the data features of the detected data to be repaired;

[0046] Processing the initial repair target data, and detecting first repair target data and second repair target data in the processed initial repair target data, wherein the first repair target data is data that indicates that the first repair state has not been successfully repaired, and the second repair target data is data that indicates that the first repair state has been successfully repaired;

[0047] When the to-be-repaired type of the first repair target data is a logical type, generating a repair suggestion file matching the data features of the first repair target data;

[0048] Based on the repair suggestion file, the data processing component of the first repair target data is updated, and the first repair target data is reprocessed based on the updated data processing component;

[0049] The second-processed first repair target data and the second repair target data are used as the target business data after the repair is completed.

[0050] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0051] Detecting the data to be repaired from the business data, and extracting initial repair target data from the data to be repaired according to the data features of the detected data to be repaired;

[0052] Processing the initial repair target data, and detecting first repair target data and second repair target data in the processed initial repair target data, wherein the first repair target data is data that indicates that the first repair state has not been successfully repaired, and the second repair target data is data that indicates that the first repair state has been successfully repaired;

[0053] When the to-be-repaired type of the first repair target data is a logical type, generating a repair suggestion file matching the data features of the first repair target data;

[0054] Based on the repair suggestion file, the data processing component of the first repair target data is updated, and the first repair target data is reprocessed based on the updated data processing component;

[0055] The second-processed first repair target data and the second repair target data are used as the target business data after the repair is completed.

[0056] In the above-mentioned method, apparatus, computer device, computer-readable storage medium and computer program product for detecting and repairing data to be repaired, during the whole process, when the data to be repaired in the business data is detected, it is not necessary to repair all the data to be repaired. Instead, the initial repair target data is extracted according to the data characteristics of the data to be repaired for selective repair, thereby shortening the time for data repair and improving the efficiency of data repair. Furthermore, firstly, all the initial repair target data are processed, and then for the first repair target data that has not been successfully repaired after processing and the type to be repaired is a logical type, the data processing component of the first repair target data is updated through a repair suggestion file that matches the data characteristics of the first repair target data, and then the first repair target data is efficiently repaired based on the updated data processing component to obtain the first target business data that is finally repaired. Moreover, by performing targeted repair on the repair target data of the logical type, the problem of slow repair caused by invalid repair can also be reduced. Finally, the process of obtaining the repaired target business data based on the second-processed first repair target data and the successfully repaired second repair target data is also simpler and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A diagram of an application environment of a method for detecting and repairing data to be repaired in an embodiment;

[0059] Figure 2 A schematic diagram of a flow chart of a method for detecting and repairing data to be repaired in one embodiment;

[0060] Figure 3 is a flow chart of a method for detecting and repairing data to be repaired in another embodiment;

[0061] Figure 4 A flowchart of a method for detecting and repairing data to be repaired in a detailed application embodiment;

[0062] Figure 5 is a structural block diagram of a device for detecting and repairing data to be repaired in an embodiment;

[0063] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are used to explain the present application and are not used to limit the present application.

[0065] The method for detecting and repairing data to be repaired provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0066] The user uploads business data in the terminal 102 and triggers the data detection button. The terminal 102 responds to the data detection button, generates a data detection request carrying the business data, and sends the data detection request to the server 104. The server 104 detects the data to be repaired from the business data, and extracts the initial repair target data from the data to be repaired according to the data characteristics of the detected data to be repaired; processes the initial repair target data, and detects the first repair target data and the second repair target data in the processed initial repair target data, wherein the first repair target data is data that has not been successfully repaired in the first repair state, and the second repair target data is data that has been successfully repaired in the first repair state; when the type of the first repair target data to be repaired is a logical type, a repair suggestion file matching the data characteristics of the first repair target data is generated; based on the repair suggestion file, the data processing component of the first repair target data is updated, and the first repair target data is re-processed based on the updated data processing component; the re-processed first repair target data and the second repair target data are used as the target business data after the repair is completed.

[0067] Furthermore, the server 104 may also send the data to be repaired together with the target service data that has been repaired to the terminal 102 , and the terminal 102 may display the data to the user.

[0068] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0069] In an exemplary embodiment, Figure 2 As shown, a method for detecting and repairing data to be repaired is provided, and the method is applied to Figure 1 The server 104 in FIG. 1 is used as an example for explanation. Among them:

[0070] S100, detecting data to be repaired from business data, and extracting initial repair target data from the data to be repaired according to data features of the detected data to be repaired.

[0071] Specifically, business data is processed raw data, which can be business information or user personal information. Data to be repaired refers to business data with problems or defects. The data features of data to be repaired are multidimensional, generally a combination of various limited and enumerable dimensions. For example, the data features of data to be repaired include but are not limited to business source, business type, and data source.

[0072] Specifically, the processed business data is obtained, and data verification is performed on the business data to obtain data verification results of the business data. The data verification results of the business data can indicate the presence of defects or the absence of defects. Data to be repaired that are indicated by the data verification results to be defective are screened from the business data, and the checked data to be repaired can be stored in a pool of data to be repaired. Generally, data verification of business data is achieved through pre-configured data verification rules.

[0073] Furthermore, the data verification cycle may be configured, and the acquired business data may be regularly verified according to the configured cycle to automatically detect the data to be repaired in the business data.

[0074] In an exemplary embodiment, the data verification type of the business data is determined, such as a sampling verification type, and the business data is verified by implementing a data verification method that matches the data verification type to determine the data to be repaired in the business data.

[0075] Furthermore, for the detected data to be repaired, if the amount of the data to be repaired is too large, directly repairing the data to be repaired will result in low efficiency or even system crash. Therefore, it is necessary to extract initial repair target data from the data to be repaired to reduce the amount of the data to be repaired.

[0076] Generally, extracting initial repair target data from the data to be repaired can be achieved through the detected data features of the data to be repaired, that is, based on the detected data features of the data to be repaired, the initial repair target data is extracted from the data to be repaired, so as to perform data repair on the initial repair target data, improve the efficiency of data repair, and then determine the problem type of the initial repair target data, and add the initial repair target data to the repair queue.

[0077] There are many extraction methods. For example, the data to be repaired can be sampled according to the detected data features to obtain the initial repair target data. The data volume of the data to be repaired can also be reduced by filtering outliers in the data to be repaired to obtain the initial repair target data. It should be noted that when the method of sampling the data to be repaired is used to extract the initial repair target data, random sampling of the data to be repaired will make the initial repair target data obtained after sampling unable to cover various situations and lack representativeness. Therefore, the method of sampling the data to be repaired is generally stratified sampling. The stratified sampling method, also called the type sampling method, is to divide the population units into several types or layers according to their attribute characteristics, and then randomly select sample units from the types or layers.

[0078] S200, processing the initial repair target data, and detecting the first repair target data and the second repair target data in the processed initial repair target data.

[0079] The first repair target data is data whose first repair state indicates that the repair is not successful, and the second repair target data is data whose first repair state indicates that the repair is successful.

[0080] Specifically, the initial repair target data to be repaired is processed, and the initial repair target data is repaired for the first time. Further, the processing can repair some initial repair target data of non-logical type to be repaired.

[0081] It should be explained that the types of data to be repaired in the initial repair target data include logical types and non-logical types. The non-logical types are generally types to be repaired that are related to the operating mechanism, such as inaccurate dependencies between tables, conflicts caused by mixed storage of business types, and errors reported during log storage.

[0082] The reason why the initial repair target data has non-logical type problems is that an error occurs during the initial processing of the original data. Therefore, the initial repair target data is reprocessed, and the initial repair target data of non-logical type to be repaired can be automatically repaired.

[0083] Furthermore, after the initial repair target data is processed, some data still cannot be repaired, and therefore, a first repair state of the processed initial repair target data is detected, and the first repair state includes successful repair or unsuccessful repair.

[0084] From the processed initial repair target data, first repair target data whose first repair state is unsuccessful repair is detected, and from the processed initial repair target data, second repair target data whose first repair state is unsuccessful repair is detected.

[0085] S300: When the to-be-repaired type of the first repair target data is a logical type, generate a repair suggestion file matching the data feature of the first repair target data.

[0086] Specifically, when the type of the first repair target data to be repaired is a logical type, the first repair target data cannot be repaired even if it is reprocessed, and therefore, a repair suggestion for the first repair target data needs to be generated first.

[0087] The repair suggestion is essentially a repair suggestion file, that is, a repair suggestion file matching the first repair target data is first generated. The repair suggestion file is a template file matching the data features of the first repair target data. One type of data feature matches one type of template file, and there is also dynamically modifiable data in the template file.

[0088] S400, based on the repair suggestion file, updating the data processing component of the first repair target data, and performing secondary processing on the first repair target data based on the updated data processing component.

[0089] Specifically, the data processing component of the first repair target data is updated through the repair suggestion file, so that the first repair target data is reprocessed based on the updated data processing component to solve the logic problems that occur during the processing. Further, for example, the logic type problem can be incomplete and inaccurate logic and some business logic is not connected.

[0090] Generally, in the process of updating the data processing component of the first repair target data through the repair suggestion file, the data processing component of the first repair target data is essentially located based on the repair suggestion file, and then the data processing component of the first repair target data is updated.

[0091] S500, taking the secondary processed repair target data and the second repair target data as the repair-completed target business data.

[0092] Specifically, the secondary processing repair target data is the data for repairing logical errors, and the second repair target data is the data for repairing non-logical errors. The secondary processing repair target data and the second repair target data are used as the target business data for repair. Furthermore, the data to be repaired and the target business data for repair can also be pushed to the terminal, and displayed to the developer by the terminal, so that the developer can clearly know the repair progress of the processed data.

[0093] In the above-mentioned method for detecting and repairing data to be repaired, during the whole process, when the data to be repaired in the business data is detected, it is not necessary to repair all the data to be repaired. Instead, the initial repair target data is extracted according to the data characteristics of the data to be repaired for selective repair, thereby shortening the data repair time and improving the efficiency of data repair. Furthermore, firstly, all the initial repair target data are processed, and then for the first repair target data that has not been successfully repaired after processing and the type to be repaired is a logical type, the data processing component of the first repair target data is updated through a repair suggestion file that matches the data characteristics of the first repair target data, and then the first repair target data is efficiently repaired based on the updated data processing component to obtain the first target business data that is finally repaired. Moreover, by performing targeted repair on the repair target data of the logical type, the problem of slow repair caused by invalid repair can also be reduced. Finally, the process of obtaining the repaired target business data based on the second-processed first repair target data and the successfully repaired second repair target data is also simpler and more efficient.

[0094] In an exemplary embodiment, detecting data to be repaired from business data includes:

[0095] In response to the configuration operation of the data verification rules, at least one of the database design configuration rules, business configuration rules and data quality configuration rules is obtained, and the data to be repaired is detected from the business data according to at least one of the database design configuration rules, business configuration rules and data quality configuration rules.

[0096] Specifically, a data verification rule configuration interface is displayed in the terminal interface, and the user configures the data verification rule in the data verification rule configuration interface. The configured data verification rules include at least one of the following categories, specifically:

[0097] 1. Database design configuration rules, for example: check whether the zipper table time range is continuous and compliant.

[0098] 2. Business configuration rules, for example: the termination date of the terminated business is not empty, and a certain product code is not empty.

[0099] 3. Data quality configuration rules, for example: employee ID is 14 digits, code value is defined, field does not contain special characters such as Chinese and spaces, etc.

[0100] Furthermore, according to at least one of the database design configuration rules, business configuration rules and data quality configuration rules, data verification is performed on the business data to determine whether the business data meets the configured data verification rules. When the business data does not meet the configured data verification rules, the business data is determined to be data to be repaired; when the business data meets the configured data verification rules, the business data is determined to be data not to be repaired.

[0101] In this embodiment, by comprehensively configuring the data verification rules, for example, configuring the database design configuration rules, configuring the business configuration rules, and configuring the data quality configuration rules, etc., accurate detection of the data to be repaired in the business data is achieved.

[0102] In an exemplary embodiment, Figure 3 As shown, S300 includes:

[0103] S310 , when the to-be-repaired type of the first repair target data is a logical type, generating an initial repair suggestion file matching the data features of the first repair target data.

[0104] S320, obtaining blood relationship data of the first repair target data.

[0105] S330, screening the program field, affected data range and verification time corresponding to the first repair target data from the blood relationship data.

[0106] S340, based on the program field, the affected data range and the verification time, updating the field content of the dynamic data field in the repair suggestion file, and using the updated initial repair suggestion file as the repair suggestion file.

[0107] The blood relationship data is obtained from the first repair target data and the associated data of the first repair target data.

[0108] Specifically, when the to-be-repaired type of the first repair target data is a logical type, an initial repair suggestion file matching the data feature of the first repair target data is queried, and based on the first repair target data, the initial repair suggestion file is updated to generate a repair suggestion file.

[0109] Herein, updating the initial repair suggestion file refers to updating the field content in the dynamic data field in the initial repair suggestion file.

[0110] Furthermore, based on the first repair target data, the initial repair suggestion file is updated, including: obtaining the blood relationship data of the first repair target data, screening the program fields, impact data ranges and verification time corresponding to the first repair target data from the blood relationship data, and updating the field content of the dynamic data fields in the initial repair suggestion file based on the program fields, impact data ranges and verification time to obtain a repair suggestion file.

[0111] Obtaining the blood relationship data of the first repair target data includes: scanning the objects corresponding to the first repair target data in the data warehouse and storing them as metadata; configuring the homology or synonymy relationship between the metadata according to the content defined by the metadata; traversing the indicator processing logic according to the metadata object (and its homology or synonymy relationship) to obtain the associated data of the first repair target data.

[0112] Then, the program fields, affected data ranges and verification time corresponding to the first repair target data are screened from the blood relationship data, and based on the program fields, affected data ranges and verification time, the field contents of the dynamic data fields in the repair suggestion file are overwritten to obtain the repair suggestion file.

[0113] In an exemplary embodiment, a repair suggestion file is as follows:

[0114] Repair (xxx) program, (xxx) module; involving fields (xxx), (xxx), involving data volume: (xxx);

[0115] Verification time: (xxx).

[0116] Among them, those enclosed in brackets are dynamic data fields in the repair suggestion file, which support dynamic adjustment, and those not enclosed are static data fields in the repair suggestion file.

[0117] In an exemplary embodiment, the repair suggestion file can also be used for notifications during release deployment. The server generates a repair message based on the field content of the dynamic data fields in the repair suggestion file, and sends the repair message to the user group. The sending method can be email, fax, etc., to inform the user group that the field content of these dynamic data fields will be repaired and the user group needs to pay attention to the impact in advance.

[0118] In the above embodiment, an initial repair suggestion file is generated through the data characteristics of the first repair target data, and based on the blood relationship data between the first repair target data and the first repair target data, the field content of the dynamic data field in the initial repair suggestion file can be accurately updated, thereby accurately generating a repair suggestion file that matches the data characteristics of the first repair target data.

[0119] In an exemplary embodiment, extracting initial repair target data from the data to be repaired according to data features of the data to be repaired includes:

[0120] A second repair state of the data to be repaired is obtained. When the second repair state indicates that repair has not been performed, the data to be repaired is sampled according to data features of the data to be repaired to obtain initial repair target data.

[0121] The second repair status includes but is not limited to: not sampled, not repaired, added to the repair queue, repaired, successfully repaired, and not successfully repaired.

[0122] Specifically, the second repair state of the data to be repaired is obtained to determine whether the data to be repaired has been automatically repaired. In the case where the second repair state indicates that the repair has not been performed, the data to be repaired can be sampled to obtain initial repair target data, and the data can be added to the repair queue. In addition, when the data to be repaired is added to the repair queue, the second repair state of the data to be repaired can be updated from not having performed repair to having been added to the repair queue.

[0123] In the above embodiment, by acquiring the second repair status of the data to be repaired, it is determined whether the data to be repaired needs to be repaired, thereby improving the accuracy and efficiency of data repair.

[0124] In an exemplary embodiment, the secondarily processed first repair target data is used as the repair-completed target business data, including:

[0125] The secondarily processed first repair target data is evaluated to generate a data repair evaluation result of the first repair target data. When the data repair evaluation result indicates that the repair is successful, the secondarily processed first repair target data is used as the target business data after the repair is completed.

[0126] Specifically, the first repair target data of the secondary processing is evaluated. The means of evaluation may be to check the first repair target data of the secondary processing again to determine whether there are still defects in the first repair target data of the secondary processing, and generate a data repair evaluation result of the first repair target data by judging whether there are still defects in the first repair target data of the secondary processing.

[0127] That is to say, when the data repair evaluation result indicates that the second-processed first repair target data still has defects, the data repair evaluation result for generating the first repair target data is unsuccessful repair; when the data repair evaluation result indicates that the second-processed first repair target data does not have defects, the data repair evaluation result for generating the first repair target data is successful repair. At this time, the second-processed first repair target data can be used as the target business data after the repair is completed.

[0128] In the above embodiment, when the data repair evaluation result indicates that the repair is successful, the secondarily processed first repair target data is used as the repaired target business data, so that the data repair process is more accurate and the target business data finally obtained is also more accurate.

[0129] In an exemplary embodiment, detecting the first repair target data and the second repair target data in the processed initial repair target data includes:

[0130] Acquire first data repair information, and determine a first repair state of the processed initial repair target data; if the first repair state indicates that the repair has been performed, generate second data repair information; if the first data repair information is inconsistent with the second data repair information, detect the processed initial repair target data according to at least one of the database design configuration rules, the business configuration rules, and the data quality configuration rules, and update the first repair state based on the detected result; if the updated first repair state indicates that the repair was not successful, determine the processed initial repair target data as the first repair target data; if the updated first repair state indicates that the repair was successful, determine the processed initial repair target data as the second repair target data.

[0131] The first data repair information is data repair information when the second repair state of the data to be repaired does not perform repair. The first data repair information and the second data repair information can be represented in the form of a table, which includes a number of data to be repaired and the data repair states corresponding to the data to be repaired.

[0132] The second repair status is the same as the first repair status, including but not limited to: not sampled, not repaired, added to the repair queue, repaired, successfully repaired, and not successfully repaired.

[0133] Specifically, before repairing the data to be repaired, first data repair information within a historical time period is obtained, and a first repair state of the initial repair target data after processing is determined.

[0134] If the first repair state indicates that the repair has been performed, the second data repair information in the current time period is generated. If the first repair state indicates that the repair has not been performed, the repair target data of the secondary processing is added to the repair queue for repair.

[0135] When the first data repair information is inconsistent with the second data repair information, that is, when the data repair information of the historical time period and the current time period are inconsistent, the processed initial repair target data is secondary checked according to the configured data verification rules, that is, at least one of the database design configuration rules, business configuration rules and data quality configuration rules, and the first repair status of the processed initial repair target data is updated based on the secondary verification results.

[0136] Based on the secondary verification result, the first repair state of the processed initial repair target data is updated, including: when the secondary verification result indicates that the repair is not successful, determining that the first repair state of the processed initial repair target data indicates that the repair is not successful; when the secondary verification result indicates that the repair is successful, determining that the first repair state of the processed initial repair target data indicates that the repair is successful.

[0137] Further, when the updated first repair state indicates that the repair has been successful, the processed initial repair target data is determined to be the second repair target data; when the updated first repair state indicates that the repair has not been successful, the processed initial repair target data is determined to be the first repair target data.

[0138] In the above embodiment, first, it is determined whether the repair target data has been repaired. If the repair has been performed, the first repair state of the processed initial repair target data is further updated through data verification rules, so that the first repair target data and the second repair target data can be accurately distinguished based on the first repair state.

[0139] In an exemplary embodiment, after the second-processed first repair target data and the second repair target data are used as the repair-completed target business data, the method further includes:

[0140] A data repair log is generated based on the data to be repaired, the target business data corresponding to the data to be repaired, and the data repair status of the target business data.

[0141] Specifically, based on the data to be repaired, the target business data corresponding to the data to be repaired, and the data repair status of the target business data, a data repair log is generated. Furthermore, the data repair log may also include data verification time, data processing time, and data secondary processing time.

[0142] In the above embodiment, by generating a data repair log, the traceability of the data repair process can be achieved, and the data repair process can be accurately and conveniently analyzed and evaluated in the future time period.

[0143] In an exemplary embodiment, data repair can ensure data integrity, data accuracy, completeness and consistency. Data repair is inseparable from the discovery and handling of problems. In order to improve the level of data repair and reduce the cost of data repair, this application proposes a data repair solution based on rule-based data verification and automatic repair. The solution connects data verification and repair in series, automatically repairs the data to be repaired detected by the rules, provides auxiliary development program modification suggestions for problems that cannot be repaired, and generates a repair report to dynamically and automatically display data quality and data repair results.

[0144] like Figure 4 As shown, the detection and repair method of the data to be repaired in this application mainly includes two stages: automatic inspection and repair process and auxiliary manual repair process. The following are the specific execution steps:

[0145] 1. Automatic verification and repair process:

[0146] A. Prerequisites: Data verification rule configuration and verification cycle configuration.

[0147] For example, data verification rules include:

[0148] 1. Database design, for example: a zipper table records the state changes of an entity, and whether the zipper table time range is continuous and compliant is checked.

[0149] 2. Business rule category, for example: the termination date of the termination business is not empty, and a certain product code is not empty.

[0150] 3. Data quality, for example: employee ID is 14 digits, code value is defined, field does not contain Chinese characters, spaces and other special characters, etc.

[0151] B. Steps:

[0152] 1) Determine the verification type, implement verification according to the verification type, and generate a pool of data to be repaired. The pool of data to be repaired contains a number of data to be repaired. Further, determine whether to start a new round of data repair. If it is started, generate problematic data table 1, which contains the data to be repaired and its corresponding repair status. The repair status at this time is unsampled. If there is no need to start a new round of data repair, generate problematic data table 2. At this time, the repair status of the data to be repaired in problematic data table 2 is updated to automatic repair has been performed. In addition, the problematic data table 1 that matches the problematic data table 2 and is before the repair is performed can be determined and compared. If there is inconsistency, the problematic data in the problematic data table 2 is checked again to modify the repair status of the data to be repaired to successful repair or unsuccessful repair.

[0153] 2) In the process of starting a new round of data repair, batch sampling is performed based on the data features of the data to be repaired, that is, stratified sampling, to form a list of data to be repaired. The list of data to be repaired contains multiple initial repair target data. Among them, the data features are multidimensional, generally a combination of various limited enumerable dimensions, including but not limited to business sources, business types, and data sources.

[0154] 3) Add the initial repair target data in the problem data list to the repair queue. At this time, the repair status of the repair target data is added to the repair queue.

[0155] 4) Main processing flow operation. The initial repair target data is reprocessed to repair the data to be repaired for the operation mechanism, that is, the non-logical type of data to be repaired, such as inaccurate dependencies between tables, mixed storage conflicts of business types, and log storage errors. For the data to be repaired for the logical type, such as incomplete and inaccurate logic or some business not connected, it cannot be repaired in this step, but repaired in the auxiliary development module.

[0156] 5) Repair status update: The repair status of the initial repair target data after processing is updated to "repair has been executed".

[0157] 6) Check the repair results. Determine whether the repair status of the processed initial repair target data is successful or unsuccessful, and generate a new table of data to be repaired. The unsuccessful data in the processed initial repair target data is used as the first repair target data, and the successfully repaired data is used as the second repair target data.

[0158] C. The sub-modules of the automatic verification and repair module include: verification module, sampling module and status check module.

[0159] 2. Assisted manual repair process:

[0160] A. Preconditions: Extract the list of data to be repaired, data features and corresponding program map relationships to obtain the program fields corresponding to the data features.

[0161] B. Steps:

[0162] 1) Generation of modification suggestions. Each type of data feature corresponds to an initial repair suggestion file. For the first repair target data that cannot be repaired in the main processing flow, that is, the first repair target data whose status is unsuccessful repair, obtain the associated data of the first repair target data, and based on the associated data of the first repair target data and the first repair target data, obtain the blood relationship data of the first repair target data, and from the blood relationship data, screen the program fields, affected data ranges, and verification time corresponding to the first repair target data, and based on the program fields, affected data ranges, and verification time, update the field content of the dynamic data fields in the repair suggestion file to obtain the repair suggestion file.

[0163] For example: repair (xxx) program, (xxx) module; involving fields (xxx), (xxx), involving data volume: (xxx);

[0164] Verification time: (xxx).

[0165] Among them, the data fields enclosed in brackets are dynamic data fields in the repair suggestion file, and the data fields not enclosed in brackets are static data fields in the repair suggestion file.

[0166] 2) Modify the main component based on the auxiliary suggestion. Based on the program field in the modification suggestion, locate the corresponding component, and the developer performs logical repair on the field content corresponding to the component. The first repair target data is added to the repair queue again. At this time, the repair status of the first repair target data is updated to have been added to the repair queue, and the first repair target data is processed again based on the field content corresponding to the repaired program field.

[0167] 3) Repair report generation.

[0168] C. The sub-modules of the auxiliary development module include: auxiliary development module and repair report generation module.

[0169] Based on the above analysis, this method connects data verification and repair in series. There is no need to establish strong dependencies between modules. With the help of status bit communication, it has the advantages of flexibility and intervention. The self-healing process and the generated auxiliary development suggestions can effectively reduce the cost of data repair.

[0170] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0171] Based on the same inventive concept, the embodiment of the present application also provides a device for detecting and repairing data to be repaired for implementing the method for detecting and repairing data to be repaired involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more devices for detecting and repairing data to be repaired provided below can refer to the limitations of the method for detecting and repairing data to be repaired above, and will not be repeated here.

[0172] In an exemplary embodiment, Figure 5 As shown, a device for detecting and repairing data to be repaired is provided, comprising: a data acquisition module 100, a processing module 200, a first logic processing module 300, a second logic processing module 400 and a repair completion module 500, wherein:

[0173] The data acquisition module 100 is used to detect the data to be repaired from the business data, and extract the repair target data from the data to be repaired according to the data characteristics of the detected data to be repaired;

[0174] The processing module 200 is used to process the initial repair target data and detect the first repair target data and the second repair target data in the processed initial repair target data, wherein the first repair target data is data that indicates that the repair is not successful in the first repair state, and the second repair target data is data that indicates that the repair is successful in the first repair state;

[0175] The first logic processing module 300 is used to generate a repair suggestion file matching the data feature of the first repair target data when the to-be-repaired type of the first repair target data is a logical type;

[0176] The second logic processing module 400 is used to update the data processing component of the first repair target data based on the repair suggestion file, and perform secondary processing on the first repair target data based on the updated data processing component;

[0177] The repair completion module 500 is used to use the second-processed first repair target data and the second repair target data as repair completion target business data.

[0178] In one embodiment, the data acquisition module 100 is also used to respond to the configuration operation of the data verification rules, obtain at least one of the database design configuration rules, business configuration rules and data quality configuration rules; and detect the data to be repaired from the business data according to at least one of the database design configuration rules, business configuration rules and data quality configuration rules.

[0179] In one embodiment, the first logic processing module 300 is also used to generate an initial repair suggestion file that matches the data characteristics of the first repair target data when the problem type of the first repair target data is a logic type; obtain the blood relationship data of the first repair target data, the blood relationship data is obtained by the first repair target data and the associated data of the first repair target data; from the blood relationship data, screen the program fields, affected data ranges and verification time corresponding to the first repair target data; based on the program fields, affected data ranges and verification time, update the field content of the dynamic data fields in the initial repair suggestion file, and use the updated initial repair suggestion file as the repair suggestion file.

[0180] In one embodiment, the data acquisition module 100 is further used to acquire a second repair state of the data to be repaired; when the second repair state indicates that repair has not been performed, the data to be repaired is sampled according to data features of the data to be repaired to obtain initial repair target data.

[0181] In one embodiment, the repair completion module 500 is also used to evaluate the second-processed first repair target data and generate a data repair evaluation result for the first repair target data; when the data repair evaluation result indicates that the repair is successful, the second-processed first repair target data is used as the target business data for the completion of the repair.

[0182] In one embodiment, the processing module 200 is also used to obtain first data repair information and determine a first repair state of the processed initial repair target data, the first data repair information being the data repair information when the second repair state of the data to be repaired represents the case where the repair has not been performed; when the first repair state represents that the repair has been performed, the second data repair information is generated; when the first data repair information is inconsistent with the second data repair information, the processed initial repair target data is detected according to at least one rule among the database design configuration rules, the business configuration rules and the data quality configuration rules, and the first repair state is updated based on the detected result; when the updated first repair state represents that the repair has not been successful, the processed initial repair target data is determined to be the first repair target data; when the updated first repair state represents that the repair is successful, the processed initial repair target data is determined to be the second repair target data.

[0183] Each module in the above-mentioned detection and repair device for data to be repaired can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0184] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting and repairing data to be repaired is implemented.

[0185] Those skilled in the art will understand that Figure 6The structure shown in the figure is a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0186] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0187] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0188] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0189] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0190] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0191] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0192] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting and repairing data to be repaired, characterized in that: The method comprises: Detecting data to be repaired from the business data, and extracting initial repair target data from the data to be repaired according to data features of the detected data to be repaired; Processing the initial repair target data, and detecting first repair target data and second repair target data in the processed initial repair target data, wherein the first repair target data is data indicating that the repair is not successful in the first repair state, and the second repair target data is data indicating that the repair is successful in the first repair state; In a case where the to-be-repaired type of the first repair target data is a logical type, generating a repair suggestion file matching the data features of the first repair target data; Based on the repair suggestion file, updating the data processing component of the first repair target data, and performing secondary processing on the first repair target data based on the updated data processing component; The second-processed first repair target data and the second repair target data are used as the target business data after the repair is completed.

2. The method according to claim 1, characterized in that The detecting the data to be repaired from the business data includes: In response to a configuration operation on a data verification rule, obtaining at least one of a database design configuration rule, a business configuration rule, and a data quality configuration rule; According to at least one of the database design configuration rule, the business configuration rule and the data quality configuration rule, the data to be repaired is detected from the business data.

3. The method according to claim 1, characterized in that The step of generating a repair suggestion file matching the data feature of the first repair target data when the problem type of the first repair target data is a logic type includes: In the case where the problem type of the first repair target data is a logical type, generating an initial repair suggestion file matching the data features of the first repair target data; Acquire blood relationship data of the first repair target data, wherein the blood relationship data is obtained from association data between the first repair target data and the first repair target data; From the blood relationship data, screening the program field, affected data range and verification time corresponding to the first repair target data; Based on the program field, the affected data range, and the verification time, the field content of the dynamic data field in the initial repair suggestion file is updated, and the updated initial repair suggestion file is used as the repair suggestion file.

4. The method according to claim 1, characterized in that The extracting initial repair target data from the data to be repaired according to the data features of the data to be repaired comprises: Acquire a second repair state of the data to be repaired; When the second repair state indicates that repair has not been performed, the data to be repaired is sampled according to data features of the data to be repaired to obtain initial repair target data.

5. The method according to claim 1, characterized in that The first repair target data of the secondary processing is used as the target business data of the repair completion, including: Evaluate the first repair target data of the secondary processing to generate a data repair evaluation result of the first repair target data; When the data repair evaluation result indicates that the repair is successful, the secondarily processed first repair target data is used as the repaired target business data.

6. The method according to claim 1, characterized in that The first repair target data and the second repair target data in the initial repair target data after the detection and processing include: Acquire first data repair information, and determine a first repair state of the processed initial repair target data, wherein the first data repair information is data repair information representing the second repair state of the data to be repaired when no repair is performed; generating second data repair information when the first repair state indicates that repair has been performed; In the case that the first data repair information is inconsistent with the second data repair information, detecting the processed initial repair target data according to at least one rule among database design configuration rules, business configuration rules and data quality configuration rules, and updating the first repair state based on the detected result; In a case where the updated first repair state indicates that the repair is not successful, determining the processed initial repair target data as the first repair target data; In a case where the updated first repair state indicates that the repair is successful, the processed initial repair target data is determined as the second repair target data.

7. A device for detecting and repairing data to be repaired, characterized in that: The device comprises: A data acquisition module, used to detect the data to be repaired from the business data, and extract initial repair target data from the data to be repaired according to the data features of the detected data to be repaired; a processing module, used to process the initial repair target data, and detect first repair target data and second repair target data in the processed initial repair target data, wherein the first repair target data is data in which the first repair state indicates that the repair is not successful, and the second repair target data is data in which the first repair state indicates that the repair is successful; A first logic processing module, configured to generate a repair suggestion file matching the data feature of the first repair target data when the to-be-repaired type of the first repair target data is a logic type; A second logic processing module is used to update the data processing component of the first repair target data based on the repair suggestion file, and perform secondary processing on the first repair target data based on the updated data processing component; The repair completion module is used to use the second-processed first repair target data and the second repair target data as target business data for repair completion.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images