Method and system for automatic data verification and repair based on scheme configuration

Abstract

The application relates to a method and system for automatic data verification and repair based on a scheme configuration, and belongs to the technical field of data processing, and solves the problem that an integrated processing flow of data verification and repair is lacked in the prior art. The method comprises the following steps: according to a verification rule configured in a data processing scheme, performing a data verification operation, generating a rule violation detail and a processing record; according to the rule violation detail, performing a batch data repair operation on to-be-processed data, and updating the rule violation detail; when a failure exists in an execution state in the rule violation detail, obtaining a repair rule of a rule violation field with the execution state being the failure, adjusting the repair rule, and performing the data repair operation again; the operation is repeated for multiple times until the execution state in the rule violation detail is all updated to be successful, final processed data is generated, otherwise, a data processing scheme is adjusted, and a data processing operation is re-executed. The method realizes the integration of data verification and repair, and improves the data repair efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for automatic data verification and repair based on scheme configuration. Background Technology

[0002] As enterprises develop and their IT infrastructure expands, the number of business systems built within enterprises continues to increase, and their data relationships become increasingly complex. The volume of object-type data within each business system—data that possesses business characteristics, commonalities, and uniqueness—is growing exponentially. Furthermore, due to business needs, there are instances of separate management, duplicate management, mutual referencing, and cross-referencing between systems. Simultaneously, data exchange between systems is becoming more frequent, leading to inaccuracies, omissions, and inconsistencies in data flow and management between business systems. Manually processing such data presents challenges due to the large volume and lack of standardized data. Therefore, a unified processing solution is needed to automatically verify and repair the data.

[0003] Existing technologies typically rely on validation rules, which determine whether the data passes or fails validation. The main focus is on cleaning the data that passes validation, such as format conversion and standard mapping. Data that fails validation is usually handled by personnel from the source system, or can only be repaired using business rules.

[0004] Existing technologies for data validation often result in a single data record appearing in multiple failed validation records, making it impossible to show the overall situation of failed validations from the perspective of the data table. Furthermore, when the source system personnel handle the data that fails validation, there is a lack of unified data processing standards and procedures, and they cannot control the progress of data processing, making it difficult to promote data repair and rectification, thus making it difficult to improve the overall data quality. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a method and system for automatic data verification and repair based on scheme configuration, so as to solve the problem of the lack of an integrated data verification and repair process in the prior art.

[0006] On one hand, embodiments of the present invention provide a method for automatic data verification and repair based on scheme configuration, including the following steps:

[0007] The system acquires the data to be processed, performs data verification operations according to the verification rules configured in the data processing scheme, and generates violation details and processing records. The violation details include the data primary key, violation field, verification rule identifier, repair rule identifier, and execution status.

[0008] Obtain the primary key of the data by using the violation field and the validation rule identifier as dimensions, obtain the repair rule according to the corresponding repair rule identifier, perform batch data repair operations on the data to be processed, and update the execution status and processing record in the violation details;

[0009] When an execution status failure is detected in the violation details, the failure rule adjustment operation key is displayed, the repair rule for the violation field with the execution status failure is obtained, the adjusted repair rule is received, and the data repair operation is performed on the violation field with the execution status failure; the repair rule is adjusted and the repair operation is performed repeatedly until the execution status of all violations in the violation details is updated to success, the failure rule adjustment operation key is hidden, and the final processed data is generated; otherwise, the data processing scheme is adjusted and the data processing operation is re-executed.

[0010] Based on the further improvement of the above method, the processing record includes the number of violations, the number of violation fields, and the verification time. After all the execution statuses in the violation details are updated to success, the data re-verification operation key is displayed in the processing record. Based on the data primary key in the violation details, the record in the final processed data is obtained, and the data verification operation is executed again to generate a data re-verification record, including the number of re-verified violations, the number of re-verified violation fields, and the re-verification status.

[0011] Based on further improvements to the above method, the states of both verification rules and repair rules include two categories: one is the usage state, including enabled and disabled, and the other is the execution state, including locked and unlocked. When performing a data verification operation, the verification rules with the usage state of enabled are obtained, and the execution state of the verification rules is set to locked. After the execution is completed, it is set to unlocked. When performing a data repair operation, the repair rules with the usage state of enabled are obtained, and the execution state of the repair rules is set to locked. After the execution is completed, it is set to unlocked.

[0012] Based on further improvements to the above method, the data processing scheme is configured based on the data model of the data object to be processed and its associated objects;

[0013] In the data model configuration interface, the system receives data model configuration commands, parses the data model information based on the selected table name of the data object to be processed and its associated table name, or the imported data records, and matches the preset data standard items according to the field names or field names in the data model information. The system then receives the modified data model information and the fields to be verified set from it, thus completing the data model configuration of the data object to be processed.

[0014] Based on further improvements to the above methods, the data processing scheme includes verification rules and repair rules;

[0015] In the verification rules interface, check if there is a configured verification rule list. If it exists, retrieve the configured verification rule list. If it does not exist, generate an initial verification rule list based on the preset data standard items matched in the data model configuration.

[0016] For unlocked verification rules, receive the verification rule configuration command, list the preset rule templates for each type of verification rule in the rule configuration interface, receive the selected verification rule type, the selected field name to be verified, the preset rule template or the configuration of the custom verification expression and its parameters, and complete the verification rule configuration operation.

[0017] For unlocked verification rules, receive commands to disable or enable them, and complete the configuration of the usage status of the verification rules.

[0018] Based on the above method, the repair rules are further improved by configuring the enabled and unlocked verification rules and their corresponding fields.

[0019] In the repair rules interface, obtain the list of enabled and unlocked verification rules and their corresponding repair rules. For unlocked repair rules, receive the repair rule configuration command. Based on the fields in the verification rules, receive the configuration of the repair condition expression, repair method and its parameters, and complete the data repair rule configuration for the fields.

[0020] For unlocked repair rules, receive commands to disable or enable them, and complete the configuration of the repair rule's usage status;

[0021] The enabled verification rules and enabled repair rules are merged to generate a data processing scheme.

[0022] Based on further improvements to the above method, the configuration of custom rule expressions and repair condition expressions includes: fields in the data model, preset functions, operators, operational relationships, and constants, ultimately generating expressions similar to natural language.

[0023] Based on the further improvement of the above method, a test operation is provided for the data processing scheme. After receiving the test command, the data object table corresponding to the data processing scheme and the total number of records in the current table are displayed in the pop-up window. The number of test data records entered is received. If it is less than or equal to the preset maximum number of test records, the test operation is started and the test data violation details and test data processing records are displayed.

[0024] On the other hand, embodiments of the present invention provide a system for automatic data verification and repair based on scheme configuration, including a data model configuration module, a data processing scheme module, a data processing execution module, a rule template management module, and a data standard management module, wherein...

[0025] The data model configuration module is used to parse the data model of the data object to be processed, match the data standard items, modify and configure the parsed data model information, and set the fields to be verified.

[0026] The data processing solution module is used to configure verification rules and repair rules for the fields to be verified in the data model, enable and disable verification rules and repair rules, generate data processing solutions, perform test operations, and record test data violation details and test data processing records.

[0027] Data processing execution module: used to perform data processing operations, failure rule adjustment operations and re-verification operations, display processing records and violation details, and export all data after processing and data after violation repair.

[0028] Rule template management module: Used to manage preset verification rules and repair rules;

[0029] Data Standards Management Module: Used to manage data standard items and the corresponding validation rules for data standard items.

[0030] Further improvements to the above system also include a scheduled task management module, which manages scheduled repair tasks for pending data objects, configures scheduled scheduling rules and exception notification methods, and displays the execution status of scheduled tasks.

[0031] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0032] 1. By integrating data models, verification rules, and repair rules, an integrated data processing solution is constructed, forming a closed-loop processing flow from data analysis, solution configuration, problem discovery to problem repair. This enhances data control capabilities, improves the execution efficiency of data repair, and facilitates the improvement of overall data quality.

[0033] 2. Provides a re-verification operation for the repaired problematic data, with built-in auditing to enhance verification capabilities;

[0034] 3. Provides the option to select fields to be validated in the data model, and allows for enabling and disabling of validation and repair rules, enabling faster adaptation to changes in business needs, facilitating adjustments to data processing solutions, and improving the efficiency of data repair.

[0035] 4. The automatic matching capability of data models and data standards, along with general rule templates, simplifies the configuration of data processing solutions and improves configuration efficiency; customizable natural language-like expressions facilitate configuration by business personnel, expanding the scope of use and the business scenarios covered.

[0036] 5. Multiple repair operations can be performed on the non-compliant data. Alternatively, the data processing plan can be adjusted again after the data has been processed in stages, and in-depth verification and repair can be carried out. The processing method is flexible and the repair effect is strong.

[0037] 6. During data processing, violation details and processing records are generated. Violation details show the violation situation of the overall data records and data fields, while processing records summarize the violation situation and repair status, which facilitates the analysis of data problems and the control of data repair progress.

[0038] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0039] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0040] Figure 1 This is a flowchart of the automatic data verification and repair method in Embodiment 1 of the present invention;

[0041] Figure 2 This is a flowchart illustrating the configuration of the data processing scheme in Embodiment 1 of the present invention;

[0042] Figure 3 This is an example diagram of the general verification rule configuration interface in Embodiment 1 of the present invention;

[0043] Figure 4 This is an example diagram of the configuration interface for the custom validation expression in Embodiment 1 of the present invention;

[0044] Figure 5 This is an example diagram of the configuration interface for repairing conditional expressions in Embodiment 1 of the present invention;

[0045] Figure 6 Example diagram of the interface configuration for the "encoding" repair method in Embodiment 1 of the present invention;

[0046] Figure 7 This is a flowchart of the overall data processing in Embodiment 1 of the present invention. Detailed Implementation

[0047] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0048] Example 1,

[0049] A specific embodiment of the present invention discloses a method for automatic data verification and repair based on scheme configuration, such as... Figure 1 As shown, it includes the following steps:

[0050] S11: Obtain the data to be processed, perform data verification operations according to the verification rules configured in the data processing scheme, and generate violation details and processing records; the violation details include the data primary key, violation field, verification rule identifier, repair rule identifier, and execution status;

[0051] S12: Obtain the primary key of the data by the dimensions of the violation field and the validation rule identifier, obtain the repair rule according to the corresponding repair rule identifier, perform batch data repair operation in the data to be processed, and update the execution status and processing record in the violation details;

[0052] S13: When the execution status in the violation details shows failure, display the failure rule adjustment operation key, obtain the repair rule for the violation field with the execution status of failure, receive the adjusted repair rule, and perform data repair operation on the violation field with the execution status of failure; cyclically adjust the repair rule and perform repair operation until the execution status in the violation details is updated to success, hide the failure rule adjustment operation key, and generate the final processed data; otherwise, adjust the data processing scheme and re-execute the data processing operation.

[0053] In implementation, a data processing scheme needs to be pre-configured before step S11, meaning step S10 is included before S11. Optionally, in step S10, the data processing scheme is configured based on the data model of the data object to be processed and its associated objects, including verification rules and repair rules, such as... Figure 2 As shown, step S10 specifically includes the following steps:

[0054] S101: Parse and configure the data model, and match data standard items.

[0055] Specifically, in the data model configuration interface, a data model configuration command is received. Based on the selected table name of the data object to be processed and its associated table name, or the imported data records, the data model information is parsed. For example, the drop-down list of the data object to be processed displays the business data table name and Chinese name in the data cleanup database. After selecting one of the tables, the associated table name is selected in the associated data object table. It can be left unselected, or one or more can be selected. Alternatively, an Excel table of imported data records can be used, which includes at least the table name of the data object to be processed, field names, and / or the associated table name and field names.

[0056] After parsing the data model, pre-defined data standard items are matched based on the field names. It's important to note that data standards define attribute standards for core data objects or public data, serving as the primary reference for data standardization and eliminating data ambiguity. For example, the data standard for "customer" defines the maximum length and illegal characters for customer names, standard enumeration values ​​for customer and document types, and the length range and illegal characters for document numbers corresponding to different document types. All systems involving "customer" must establish data models and store data according to these data standards. Some data standards are more stringent, defining specifications for attribute field names. Here, each field name in the data model of the object to be processed is matched against the field names in the data standard items. When a field name in the data model is modified, the matching operation is triggered again, and the matched data standard item is displayed in the corresponding data standard item. Alternatively, it can be manually selected or adjusted. Fields that match the data standard items are set as fields to be validated by default. Other fields can be selected as fields to be validated at the same time. This completes the data model configuration of the data object to be processed and stores it in the data model table, including data processing scheme identifier, data object identifier, field name, field type, field length, whether it is a primary key, data standard item identifier, and whether it is validated.

[0057] Next, configure the validation rules for the fields to be validated.

[0058] S102: Configure validation rules based on the fields to be validated in the data model, and set the usage status of the validation rules.

[0059] It should be noted that in the data model, one field to be validated corresponds to one or more validation rules. In the validation rules interface, the system checks if a configured list of validation rules exists. If it does, the system retrieves the configured list of validation rules; otherwise, it generates an initial list of validation rules based on the preset data standard items matched in the data model configuration.

[0060] Based on the standard specifications for each attribute defined in the data standard, the system pre-sets the verification rules for each data standard item. This not only facilitates user configuration and speeds up the configuration process, but also serves as a rule configuration example for user reference.

[0061] The rule configuration interface lists the preset rule templates for each type of validation rule, including:

[0062] 1. Integrity: Fields must be non-nullable, sub-table records must be non-nullable, parent table records must be non-nullable, etc.

[0063] 2. Uniqueness: Content deduplication, sub-table content deduplication, combined deduplication, etc.

[0064] 3. Authority: Telephone format, time format, identity format, taxpayer format, etc.;

[0065] 4. Legality: Length constraints, content constraints, date constraints, numerical constraints, etc.;

[0066] 5. Consistency: equal to a certain value, equal to a certain field, referencing a certain field, etc.

[0067] 6. Other: Custom validation expressions.

[0068] The configuration of custom validation expressions includes: fields in the data model, preset functions, operators, operational relationships, and constants, ultimately generating expressions similar to natural language, which facilitates configuration by business personnel and expands the scope of use and the business scenarios covered.

[0069] For example, the general verification rule configuration interface is as follows: Figure 3 As shown, the configuration interface for custom validation expressions is as follows: Figure 4 As shown.

[0070] The system receives the selected validation rule type, the selected field name to be validated, the preset rule template or the custom rule expression and its parameters, and completes the validation rule configuration operation.

[0071] It's important to note that the status of validation rules falls into two categories: usage status (enabled / disabled) and execution status (locked / unlocked). When performing data validation, the system retrieves validation rules with an enabled usage status and sets their execution status to locked. After execution, the status is reset to unlocked. Unlocked validation rules can be modified, thus preventing inconsistencies and errors caused by modifying validation rules during data processing.

[0072] For unlocked verification rules, receive commands to disable or enable them, and complete the configuration of the usage status of the verification rules.

[0073] The configured verification rules are stored in the verification rule table, which includes data processing scheme identifier, data object identifier, verification rule identifier, field name, rule category, verification method identifier, verification parameters, usage status, and execution status.

[0074] S103: Configure repair rules based on enabled and unlocked verification rules, and set the usage status of repair rules.

[0075] It should be noted that one verification rule corresponds to one or more repair rules; the repair rules are configured based on the enabled and unlocked verification rules and their corresponding fields.

[0076] The repair rules interface displays enabled and unlocked verification rules. After selecting a verification rule, configure the corresponding repair rule, including: repair condition expression, repair method identifier, and parameter configuration. Repair methods include:

[0077] 1. Add: Add prefixes, add suffixes, add characters at specified positions, etc.;

[0078] 2. Deletion: Specify the character to delete, specify the position and length of characters to delete, etc.;

[0079] 3. Modification: Sequential modification, forced modification, content replacement, specified replacement, etc.;

[0080] 4. Encoding: Re-encoding, adding encoding at a specified position, etc.;

[0081] 5. Conversion: Pinyin conversion, uppercase / lowercase conversion, initial letter capitalization, uppercase / lowercase reversal, etc.

[0082] For example, the conditional expression configuration interface is as follows: Figure 5 As shown, the "Encoding" repair method configuration interface is as follows: Figure 6 As shown.

[0083] Receive the configuration of the repair condition expression, repair method and its parameters, and complete the configuration of the data repair rules for the field.

[0084] It's worth noting that the status of repair rules also includes two categories: usage status (enabled, disabled) and execution status (locked, unlocked). When performing a data repair operation, the system retrieves repair rules with an enabled usage status and sets their execution status to locked. After execution, the status is set back to unlocked.

[0085] For unlocked repair rules, receive deactivation or activation commands to complete the configuration of the repair rule's usage status. The configured repair rules are stored in the repair rule table, including data processing scheme identifier, data object identifier, verification rule identifier, repair rule identifier, repair condition expression, repair condition identifier, repair rule parameters, usage status, and execution status.

[0086] The enabled verification rules and enabled repair rules are merged to generate a data processing solution. During iterative data processing, the verification rules and repair rules can be enabled or disabled to more quickly meet changes in business needs, facilitate adjustments to the data processing solution, and improve the efficiency of data repair.

[0087] S104: Test data processing plan, generating test data violation details and test data processing records.

[0088] To facilitate understanding of the accuracy and correctness of the data processing scheme configuration and to gain a preliminary understanding of the standardization of the data to be processed, a test operation is provided for the data processing scheme. After receiving the test command, a pop-up window displays the data object table corresponding to the data processing scheme and the total number of records in the current table. The number of test data records entered is received. If it is less than or equal to the preset maximum number of test records, the test operation will begin.

[0089] Data is retrieved based on the configured number of records and the field names in the validation rules of the data processing scheme. A validation expression is constructed from the rule template based on the validation method identifier and validation parameters. The data is validated sequentially. For data that fails validation, its corresponding primary key, field name, validation rule identifier, and repair rule identifier are recorded in the test data violation details table. The number of violation records is counted based on different primary keys, and the number of violation fields is counted based on different field names. The current time is recorded as the validation time in the test data processing record. Then, based on the violation details, the primary key is obtained by using the violation field and validation rule identifier as dimensions. The corresponding repair rule is obtained based on the repair rule identifier, and batch data repair is performed on the violation test data, recording the repair start time. If the repair is successful, the execution status in the violation details is set to success; otherwise, the execution status is set to failure, and the next step is skipped. After all steps are performed, the repair end time is recorded, and the number of repaired records and fields is updated in the data record table based on the execution status.

[0090] After the test is completed, the test data processing record is displayed, and details of test data violations can be viewed. Based on the test results, the data processing plan can be adjusted.

[0091] After the test passes, proceed to step S11 to process all the data to be processed. The overall data processing flowchart is as follows: Figure 7 As shown.

[0092] In step S11, it should be noted that the data to be processed can be synchronized with the source data through an ETL tool, imported through a database client tool, or obtained from a backup table through a database connection.

[0093] The data processing process is similar to the data processing scheme test in step S104. First, a data validation operation is performed to generate violation details and processing records. The violation details include the data primary key, violation fields, validation rule identifier, repair rule identifier, and execution status. The processing records include the number of violations, the number of violation fields, the validation time, the number of repaired records, the number of repaired fields, the repair start time, and the repair end time. Then, in step S12, the data primary key is obtained based on the violation fields and validation rule identifiers. That is, data records in the data to be processed that do not conform to the same validation rule for the same field are grouped into the same category. This category corresponds to the same repair rule identifier. The repair rule is found based on the repair rule identifier, and then it is identified whether it meets the repair condition expression. For the data that meets the repair condition, a batch data repair operation is performed, updating the execution status in the violation details and the number of repaired records, the number of repaired fields, the repair start time, and the repair end time in the processing records. The violation record facilitates viewing the violation status of all data from the perspective of the data object and understanding the standardization level of the data object. The processing record facilitates understanding the repair status of the violation data and controlling the data repair progress.

[0094] Preferably, when there are many data processing operations to be performed simultaneously, multi-threading or scheduled tasks can be used.

[0095] Unlike the test data processing solution, the record of failed execution of violation details can be repaired multiple times, which is convenient for flexible adjustment and quick adaptation according to the actual situation.

[0096] Specifically, in step S13, when the execution status of the violation details is found to be failed, the failure rule adjustment operation key is displayed, the repair rule of the violation field with the execution status of failure is obtained, the adjusted repair rule is received, the data repair operation is performed on the violation field with the execution status of failure, and the violation details and processing records are updated; the repair rule is adjusted in a loop, and the repair operation is performed until the execution status of all violations in the violation details is updated to success, the failure rule adjustment operation key is hidden, and the final processed data is generated.

[0097] If multiple failed rule adjustments fail to update the execution status of all violation details to success, adjust the data processing scheme for the data after the phased processing, perform data processing operations, conduct in-depth verification and repair, or restore the initial source data, readjust the data processing scheme, and re-execute the data processing.

[0098] Preferably, a threshold for manual modification can be set. When the number of unrepaired entries or fields is less than the threshold, manual modification can be provided on the interface. The data before and after each modification, the person who made the modification, and the time of the modification can be recorded to make it traceable.

[0099] Preferably, after all execution statuses in the violation details are updated to successful, the data re-verification operation key is displayed in the processing record. Based on the data primary key in the violation details, the records in the final processed data are retrieved, and the data verification operation is performed again to generate a data re-verification record, including the number of re-verified violations, the number of re-verified violation fields, and the re-verification status.

[0100] Compared with existing technologies, the data automatic verification and repair method based on scheme configuration provided in this embodiment, according to an integrated data processing scheme, sequentially performs data verification, generates violation details, generates data processing records, repairs data, updates violation details, and updates data processing records on the data to be processed. After completing one data processing cycle, the corresponding repair rules can be adjusted multiple times for unrepaired violation data to perform data repair. Alternatively, the data processing scheme can be adjusted again for deeper verification and repair of the data after phased processing, offering flexible processing methods. A re-verification operation is provided for the repaired problematic data, with built-in auditing, enhancing verification capabilities. The entire method enhances data control capabilities, improves the execution efficiency of data repair, and facilitates the improvement of overall data quality.

[0101] Example 2,

[0102] A system for automatic data verification and repair based on scheme configuration is provided to implement the automatic data verification and repair method in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. The system includes: a data model configuration module, a data processing scheme module, a data processing execution module, a rule template management module, and a data standard management module.

[0103] The data model configuration module is used to parse the data model of the data object to be processed, match the data standard items, modify and configure the parsed data model information, and set the fields to be verified.

[0104] The data processing solution module is used to configure verification rules and repair rules for the fields to be verified in the data model, enable and disable verification rules and repair rules, generate data processing solutions, perform test operations, and record test data violation details and test data processing records.

[0105] Data processing execution module: used to perform data processing operations, failure rule adjustment operations and re-verification operations, display processing records and violation details, and export all data after processing and data after violation repair.

[0106] Rule template management module: Used to manage preset verification rules and repair rules;

[0107] Data Standards Management Module: Used to manage data standard items and the corresponding validation rules for data standard items.

[0108] Preferably, it also includes a scheduled task management module, which is used to manage scheduled repair tasks for data objects to be processed, configure scheduled scheduling rules and exception notification methods, and display the execution status of scheduled tasks.

[0109] Compared with existing technologies, the data automatic verification and repair system based on scheme configuration provided in this embodiment offers a visual interface. It can configure simple and general verification and repair rules, as well as complex verification and repair rules through custom expressions similar to natural language. As long as verification and repair can be achieved through programs or SQL, the data processing scheme can be configured through configuration, thus achieving unified data processing standards. By flexibly selecting multiple repairs or re-verifications, the processing progress can be tracked and the repair accuracy can be controlled, enhancing the ability to manage illegal data and improving overall data quality.

[0110] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0111] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic data verification and repair based on scheme configuration, characterized in that, Includes the following steps: In the data model configuration interface, the data model of the data object to be processed is parsed, and each field name or field name of the data model of the object to be processed is matched with the field name or field name in the data standard item. The field that matches the data standard item is set as the field to be verified by default, or other fields can be selected as fields to be verified at the same time. One field to be verified corresponds to one or more verification rules, and one verification rule corresponds to one or more repair rules; the status of the verification rules and the repair rules includes two categories: one is the usage status, including enabled and disabled, and the other is the execution status, including locked and unlocked; the enabled verification rules and the enabled repair rules are merged to generate a data processing scheme. The system acquires the data to be processed, performs data verification operations according to the verification rules configured in the data processing scheme, and generates violation details and processing records. The violation details include the data primary key, violation fields, verification rule identifier, repair rule identifier, and execution status. The processing records include the number of violations, the number of violation fields, and the verification time. When performing the data verification operation, the system acquires the verification rules whose usage status is enabled, sets the execution status of the verification rules to locked, and sets them to unlocked after execution. The primary key of the data is obtained by using the violation field and the verification rule identifier as dimensions. The repair rule is obtained according to the corresponding repair rule identifier. A batch data repair operation is performed on the data to be processed, and the execution status and the processing record in the violation details are updated. When the data repair operation is performed, the repair rule with the usage status of enabled is obtained, and the execution status of the repair rule is set to locked. After the execution is completed, it is set to unlocked. When a failure is detected in the execution status of the violation details, the failure rule adjustment operation key is displayed, the repair rule for the violation field with the execution status of failure is obtained, the adjusted repair rule is received, and the data repair operation is performed on the violation field with the execution status of failure; the repair rule is adjusted in a loop and the repair operation is performed until the execution status of all violations in the violation details is updated to success, the failure rule adjustment operation key is hidden, and the final processed data is generated; otherwise, the data processing scheme is adjusted and the data processing operation is re-executed. After all execution statuses in the violation details are updated to successful, the data re-verification operation key is displayed in the processing record. Based on the data primary key in the violation details, the record in the final processed data is retrieved, and the data verification operation is performed again to generate a data re-verification record, including the number of re-verified violations, the number of re-verified violation fields, and the re-verification status.

2. The method for automatic data verification and repair based on scheme configuration according to claim 1, characterized in that, The data processing scheme is configured based on the data model of the data object to be processed and its associated objects; In the data model configuration interface, the system receives a data model configuration command, parses the data model information based on the selected table name of the data object to be processed and its associated table name, or the imported data records, and matches the preset data standard items with the field names or field names in the data model information. The system then receives the modified data model information and the fields to be verified set therefrom, thus completing the data model configuration for the data object to be processed.

3. The method for automatic data verification and repair based on scheme configuration according to claim 2, characterized in that, The data processing scheme includes verification rules and repair rules; In the verification rules interface, identify whether there is a configured verification rule list. If it exists, obtain the configured verification rule list. If it does not exist, generate an initial verification rule list based on the preset data standard items matched in the data model configuration. For unlocked verification rules, receive the verification rule configuration command, list the preset rule templates for each type of verification rule in the rule configuration interface, receive the selected verification rule type, the selected field name to be verified, the preset rule template or the configuration of the custom verification expression and its parameters, and complete the verification rule configuration operation. For unlocked verification rules, receive commands to disable or enable them, and complete the configuration of the usage status of the verification rules.

4. The method for automatic data verification and repair based on scheme configuration according to claim 3, characterized in that, The repair rules are configured based on enabled and unlocked verification rules and their corresponding fields; In the repair rules interface, obtain the enabled and unlocked verification rules and their corresponding repair rule list. For the unlocked repair rules, receive the repair rule configuration command. Based on the fields in the verification rules, receive the configuration of the repair condition expression, repair method and its parameters, and complete the data repair rule configuration for the fields. For unlocked repair rules, receive commands to disable or enable them, and complete the configuration of the repair rule's usage status; The enabled verification rules and enabled repair rules are combined to generate the data processing scheme.

5. The method for automatic data verification and repair based on scheme configuration according to claim 4, characterized in that, The configuration of the custom validation expression and the repair condition expression includes: fields in the data model, preset functions, operators, operational relationships, and constants, ultimately generating a natural language-like expression.

6. The method for automatic data verification and repair based on scheme configuration according to claim 4, characterized in that, The system provides test operations for the data processing scheme. After receiving the test command, a pop-up window displays the data object table corresponding to the data processing scheme and the total number of records in the current table. It receives the number of test data records entered. If the number of records is less than or equal to the preset maximum number of test records, the test operation is started, and the test data violation details and test data processing records are displayed.

7. A system for automatic data verification and repair based on scheme configuration, used to execute the method for automatic data verification and repair based on scheme configuration as described in any one of claims 1-6, characterized in that, It includes a data model configuration module, a data processing scheme module, a data processing execution module, a rule template management module, and a data standard management module. The data model configuration module is used to parse the data model of the data object to be processed in the data model configuration interface, match the data standard items, modify and configure the parsed data model information, and set the fields to be verified. Specifically, it includes: matching each field name or field name of the data model of the object to be processed with the field name or field name in the data standard items. Fields that match the data standard items are set as fields to be verified by default, or other fields can be selected as fields to be verified at the same time. The data processing solution module is used to configure verification rules and repair rules for the fields to be verified in the data model. One field to be verified corresponds to one or more verification rules, and one verification rule corresponds to one or more repair rules. The module enables and disables the verification and repair rules, merges the enabled verification rules and enabled repair rules to generate a data processing solution, and performs test operations, records test data violation details and test data processing records. Data processing execution module: used to perform data processing operations, failure rule adjustment operations and re-verification operations, display processing records and violation details, and export all data after processing and data after violation repair. Rule template management module: Used to manage preset verification rules and repair rules; Data Standards Management Module: Used to manage data standard items and the corresponding validation rules for data standard items.

8. The system for automatic data verification and repair based on scheme configuration according to claim 7, characterized in that, It also includes a scheduled task management module, which is used to manage scheduled repair tasks for pending data objects, configure scheduled scheduling rules and exception notification methods, and display the execution status of scheduled tasks.

Images