Data quality guarantee method and system based on AI capability and data rules
By using a dynamic generation and verification method based on AI capabilities and data rules, the flexibility and efficiency issues of traditional data quality assurance methods are solved. This enables intelligent and comprehensive data quality detection, adapting to business changes and complex data environments, reducing manual intervention, and improving detection efficiency and accuracy.
Patent Information
- Application Number
- CN202511177137.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-21
- Publication Date
- 2025-12-05
AI Technical Summary
Existing data quality assurance methods lack flexibility, making it difficult to adapt to business changes and complex data scenarios. Furthermore, relying on manually written rules is inefficient and cannot comprehensively detect potential data quality problems.
By acquiring interface information, utilizing AI models and dynamic rule generation mechanisms, and combining data collection tools and the ES system, data rules are automatically generated and verified. Combined with manual confirmation, intelligent detection of data quality is achieved.
It improves the flexibility and efficiency of data quality inspection, reduces manual workload, improves inspection accuracy, can discover deep-seated data correlation problems, and ensures the reliability of data quality.
Smart Images
Figure CN121071302A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, in particular to a data quality guarantee method and system based on AI capability and data rules. BACKGROUND
[0002] In the field of data processing, traditional data quality guarantee methods mainly rely on manual writing of fixed rules for data verification. Usually, based on known data format specifications and business logic, the value range, data type and other rules of key data fields are manually sorted out, and then the data is checked one by one. This way is low in efficiency and prone to human error when facing complex data structures and large amounts of data.
[0003] Some other existing technologies are static rule-based data quality detection tools. These tools verify data through a predefined rule set, such as setting length limits, non-empty constraints and other rules for specific fields in database tables. When data flows into the system, it is checked according to these pre-set rules, and if the data does not meet the rules, it is marked as abnormal. However, this method is relatively rigid in rule making, and is difficult to adapt to dynamic changes in business needs and complex data correlation.
[0004] From the analysis of the prior art, the following shortcomings exist: first, the rule making lacks flexibility, which cannot timely respond to business changes and complex data scenarios, resulting in a large number of potential data quality problems that cannot be effectively detected; second, the degree of human dependence is high, and a large amount of manpower is needed in the process of rule writing and data checking, which is not only low in efficiency but also high in cost; third, for some quality problems hidden in the deep correlation of data, such as when the "certificate type" is "ID card", the "certificate number" needs to be checked for "ID card" rules, i.e. "certificate number length is 18 digits; except the last digit which allows X, the rest are all digits". The traditional method is difficult to find and cannot fully guarantee the data quality. The present application aims to solve these technical problems and achieve more efficient, intelligent and comprehensive data quality guarantee. SUMMARY
[0005] The present application provides a data quality guarantee method and system based on AI capability and data rules, which solves the technical problems of fixed data rules in existing methods, lack of flexibility, low efficiency and difficulty in effectively detecting potential data quality problems.
[0006] A data quality guarantee method based on AI capability and data rules, comprising the following steps:
[0007] S1, acquiring different interface information;
[0008] S2: based on different interface information, data flow recording and information extraction of corresponding interface are carried out, and the information is converted into structured data;
[0009] Based on different interface information, different data rules are generated by using the data source of the interface specification system, the dynamic rule generation mechanism of the enterprise master data and the AI model;
[0010] S3, the structured data in S2 and the generated data rules are transmitted to the AI model for checking, and feedback is carried out according to the checking result.
[0011] The application preferably provides a data quality guarantee method based on AI capability and data rules, and S2 comprises the following steps:
[0012] S2.1, using a data acquisition tool, a monitoring point is set on a data transmission channel, data packets flowing through are captured, and the data packets are parsed and classified, the data packets related to the target interface are screened out, and initial interface request and response information are obtained;
[0013] S2.2, the extracted information is converted into a structured data format (JSON) through the captured interface request and response information.
[0014] Preferably, the interface request is checked before being sent to AI verification, if the interface response is rejected, the data is considered as not passing the interface, and the data is sent to AI verification if the interface response is passed.
[0015] The application preferably provides a data quality guarantee method based on AI capability and data rules, and the dynamic rule generation mechanism of S2 comprises three data rule generation modes:
[0016] The first mode is that the data source of the interface specification system is connected, the data information of the interface is extracted by using quality technology-AI interface verification tool, and the interface specification system is ApiDoc system;
[0017] The second mode is that the data source of the enterprise internal master data management specification is connected, and the data information of the enterprise in management is extracted;
[0018] The third mode is that the AI is used to automatically derive or assist in generating rules for checking the correctness of future interactive data according to the data characteristics of the sampling data.
[0019] The application preferably provides a data quality guarantee method based on AI capability and data rules, and S3 comprises the following steps:
[0020] S3.1, the AI model is pre-trained
[0021] The AI model is pre-trained by learning the normal mode and abnormal features of the data through the model using a large amount of sample data that meets the data quality rules and has common data problems, so as to form a relatively complete knowledge base.
[0022] S3.2, the data to be checked and the rules are input into the AI model trained in S3.1, the model extracts features and matches patterns of the data, judges whether the data meets the rule requirements and feeds back the checking result.
[0023] The application preferably provides a data quality guarantee method based on AI capability and data rules, and S3 further includes S3.3: when the AI model finds information that does not meet the rules, a professional person confirms it manually and feeds back the checking result.
[0024] Preferably, manual confirmation of whether it meets the requirements is roughly divided into three categories.
[0025] 1, if the AI checking does not meet the requirements, but the data is normal, the prompt word training model needs to be optimized.
[0026] 2, if the AI checking data is normal, but the rule does not meet the requirements and the interface user is correct, that is, the rules in the ApiDoc system do not meet the actual interface rules, the business system needs to update the ApiDoc system rules.
[0027] 3, if the AI checking does not meet the requirements, the interface user inputs does not meet the requirements, which belongs to the data quality problem checked out, the upstream and downstream of the interface need to be contacted for rectification, such as adding access control verification to enhance the robustness of the interface.
[0028] The application preferably provides a data quality guarantee method based on AI capability and data rules, and when the rules in the ApiDoc system are incomplete or unclear, the missing or unclear rules are manually edited and supplemented.
[0029] The application can quickly adapt to business changes and complex data environments through dynamic data flow recording and multi-source rule generation, and is more flexible than traditional static rule setting methods. The application uses the intelligent learning and analysis capability of the AI model to greatly reduce the workload of manual data quality checking, improve the detection efficiency and accuracy, and finally combines the manual confirmation link to ensure the reliability of data quality problem processing and avoid the possible misjudgment of the AI model.
[0030] A data quality guarantee system based on AI capability and data rules is used to implement the data quality guarantee method based on AI capability and data rules, and includes:
[0031] The flow recording system is used to test and record the data flow of the corresponding interface in the system based on the interface information in the test environment.
[0032] ES system, for storing data captured by the traffic recording system from different interfaces;
[0033] Quality technology-AI interface checking tool, for extracting corresponding interface information from the ES system, connecting the ApiDoc system, data governance management system, rule generation / configuration, pushing AI model checking, and AI checking report;
[0034] AI model: for feature extraction and pattern matching of data, to determine whether the data meets the rule requirements and feedback the checking results.
[0035] The quality technology-AI interface checking tool is also used to query all data of the corresponding interface information in the ES and push the data to the AI model, and the AI model generates rules according to the data and feeds back to the quality technology-AI interface checking tool, and the quality technology-AI interface checking tool stores the rules.
[0036] The beneficial effects of the present application include:
[0037] 1. The present application can quickly adapt to business changes and complex data environment through dynamic data traffic recording and multi-source rule generation, and has dynamic and comprehensive rule generation, which is more flexible than traditional static rule making method.
[0038] 2. The present application uses the intelligent learning and analysis ability of AI model to greatly reduce the workload of artificial data quality inspection, improve the detection efficiency and accuracy, and combine with the artificial confirmation link to ensure the reliability of data quality problem processing and avoid the possible misjudgment of AI model.
[0039] 3. The present application has stronger ability to find data quality problems, not only can detect surface rule violation, but also can excavate deep-seated data correlation problems based on AI model deep learning and feature extraction ability. BRIEF DESCRIPTION OF DRAWINGS
[0040] Figure 1 The flowchart of the present application based on AI capability and data rule data quality guarantee method.
[0041] Figure 2 The block diagram of the present application based on AI capability and data rule data quality guarantee system. DETAILED DESCRIPTION
[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0043] A data quality assurance method based on AI capabilities and data rules, such as Figure 1 As shown, it includes the following steps:
[0044] S1. Obtain different interface information;
[0045] S2: Data Traffic Recording and Information Extraction: In the test environment, the data traffic in the system is recorded based on different interface information. The request and response information of the interface is accurately extracted from the recorded data and converted into a structured data format.
[0046] For example, when a customer registers their contact information, the contact information interface in the user center is called. During this step, traffic recording will extract the corresponding parameter information. Some simplified formatted data information is as follows:
[0047] [{"body":{"class":"com.xwbank.urc.puser.model.PersonUserContactsSaveRequest","userContacts":[{"class":com.xwbank.urc.puser.model.PersonUserContactsInRequest",
[0048] "contactDocId":"53060****0019","contactDocType":"B01","contactName":"spouse",
[0049] "contactPhone":"156****1222","contactWordUnit":"Spouse's Work Unit","contactWorkAddr":null}],"userId":"2001704036"}……).
[0050] The accurate extraction here refers to capturing the context information of code execution (including input parameters, return values, exception values, call links, running SQL, etc.) by injecting monitoring code on the key execution path of the application through the instrumentation technology. The captured data result set is stored in ES.
[0051] The specific steps of S2 are as follows:
[0052] S2.1: Based on the interface information of S1, use the data collection tool to set up a monitoring point on the data transmission channel, capture the data packets flowing through, analyze and classify the data, filter out the data packets related to the target interface, and obtain the preliminary interface request and response information;
[0053] The data collection tool uses the instrumentation technology to record data and store the results in ES.
[0054] Among them, data analysis and classification are to extract and analyze, classify the data related to the interface in ES.
[0055] S2.2: Convert the extracted information into structured data format (JSON) and prepare it for uploading AI model checking through the captured interface request and response information;
[0056] The extracted information is extracted from ES by the "quality technology-AI interface checking" tool, including the functions of checking ApiDoc system, rule generation / configuration, pushing AI model checking, and AI checking report.
[0057] The extracted information refers to the incoming parameters of the interface, such as "contactDocId" in the above example data:
[0058] "53060****0019", ID information, length and range. Because it involves sensitive information, * is used directly.
[0059] The response request will be checked before uploading AI verification, such as interface response rejection. If such parameter data is considered as interface failure, it will not be sent to AI checking.
[0060] S2 also includes: based on different interface information, using the data source of the interface specification system, enterprise master data and the dynamic rule generation mechanism of AI model to generate different data rules;
[0061] The specific generation method of data rules in S2 is as follows:
[0062] Method 1: Connect to the data source of the interface specification system (ApiDoc system), and use the quality technology - AI interface verification tool to extract information such as the input and output parameter definitions, data types, and business meanings of the interface. If the rules in the ApiDoc system are incomplete or unclear, the missing or unclear rules can be supplemented manually.
[0063] Taking the above "interface for contact information" as an example, the data rules include the verification of contactName. The ApiDoc system specification is "Contact name, field length less than 20", and the front - end specification is supplemented manually as "The name cannot contain special characters, contains Chinese or '.', and does not contain keywords: 'guess', 'guess what', 'your name', 'guess it', 'guess', 'don't say', 'don't know', 'won't tell', 'nothing', 'what', 'ok', '囧囧', etc.".
[0064] Method 2: Connect to the data source of the enterprise - internal master data management specification, extract the data information managed by the unit, and the master data management specification uses the Yuanyang system. Such as: the unified standard of customer information, product coding rules, etc., and integrate them into the interface field rules.
[0065] Method 3: Automatically deduce or assist in generating rules for verifying the correctness of future interaction data based on the data characteristics of the sampled data. For example, after configuring the interface to be monitored, by clicking "AI rule generation", the quality technology - AI interface verification tool collects some sample data and then pushes it to the AI model to generate rules in reverse.
[0066] Specifically, through the interface information configured with monitoring, click "Automatically obtain", and the quality technology - AI interface verification tool will query all the data of this interface information in ES and push it to the AI model to generate rules. If there is no data in ES, the generation will fail. For example, in the case of no interface calls recently, generally only data for 7 days is retained in the test environment.
[0067] S3: AI model verification and feedback
[0068] Send the structured data and data rules in S2 to the AI model for verification. During the data sending process, the AI model can learn the characteristics and patterns of a large amount of historical data and perform intelligent analysis on new data.
[0069] The specific steps of S3 are as follows:
[0070] S3.1: Pre - train the AI model
[0071] Use sample data containing a large number of data that conform to data quality rules and have common data problems to let the model learn the normal patterns and abnormal characteristics of the data, and form a relatively complete knowledge base.
[0072] If the submission parameter is the name, the model only needs to be told that the parameter is the name. The model needs to meet the knowledge base rule conditions during the check, such as the length range, which special characters are not allowed, which keywords, etc.
[0073] S3.2: Determine whether the data to be checked meets the rules
[0074] The data to be checked and the rules are input into the trained AI model. The model extracts features and pattern matches the data to determine whether the data meets the rule requirements.
[0075] For example, the "contact information interface" described above, the AI check is "not passed", and the analysis result is "the value of contactName is'spouse', which contains the keyword 'no' (although it does not appear directly, it checks whether there are non-compliant keywords, which may be incorrect here)"; if only traditional rule checking is used, it may not be able to identify the data quality problem.
[0076] In this embodiment, the large model accesses DeepSeek-R1, and accurately judges it by modifying the prompt words of the pre-agent agent. The prompt words are obtained through AI pre-training, and the optimization of the prompt words is modified and optimized by manual confirmation or adjusted and optimized through the feedback structure of the AI model.
[0077] S3.3: Artificial confirmation and problem handling
[0078] For information that does not meet the rules found by the AI model, professional personnel perform artificial confirmation. The "contact information interface" example described above can be referred to.
[0079] Artificial confirmation of whether it meets the requirements is divided into three categories:
[0080] 1. If the AI check does not meet the requirements, but the data is normal, the prompt words need to be optimized to train the model.
[0081] 2. If the AI check data is normal, but the rules are not met and the interface user is correct, that is, the rules in the ApiDoc system do not meet the actual interface rules, the business system needs to update the ApiDoc system rules. For example: the contactDocId description of the interface information in the ApiDoc system is an ID number, but there may be a social credit code for a public in the actual check. According to the type of the above-mentioned certificate, it is necessary to judge, so the inaccuracy of the interface information in the ApiDoc system needs to be corrected to promote the accuracy of the ApiDoc system.
[0082] 3. If the AI check does not meet the requirements, the interface user inputs does not meet the requirements, which belongs to the data quality problem of the check, and needs to contact the upstream and downstream of the interface for rectification, such as adding access control verification to enhance the robustness of the interface.
[0083] The method of the present application is applied to the test environment, and the current test environment has configured 100+ interface information, and the comprehensive checking pass rate is 89%, the AI checking does not pass, and the part needing manual confirmation is 11%, compared with the original manual writing of fixed rules, the work efficiency is greatly improved.
[0084] And in the smoke test of a certain new service online, the new service here refers to the interface that is not changed, and the new project needs to reuse the interface. In the case of unclear interface of the user side, there may be random calling, unclear input parameters, resulting in low data quality of writing to the downstream, therefore, data quality guarantee is needed.
[0085] The smoke test of the method of the present application on the new service takes 0.5 hours, compared with 3.2 hours of manual writing of fixed rules for smoke test, the test efficiency is effectively improved by 84.4%.
[0086] In another embodiment, the data corresponding to the result fed back by the AI model is also included as a sample for AI training, and the AI model is further updated and corrected, so that the efficiency and checking pass rate of the AI model are improved.
[0087] In another embodiment, a data quality guarantee system based on AI capability and data rules, as shown in Figure 2 , comprises:
[0088] The flow recording system, that is, the above-mentioned data collection tool, is used for recording the data flow of the corresponding interface in the system based on the interface information by using the plug-in technology in the test environment;
[0089] The ES system is used for storing the data captured from different interfaces by the flow recording system;
[0090] The quality technology-AI interface checking tool is used for extracting the corresponding interface information from the ES system, checking the ApiDoc system, the data governance management system, the rule generation / configuration, pushing the AI model checking, and the AI checking report;
[0091] The AI model is used for feature extraction and pattern matching of data, judging whether the data meets the rule requirements and feeding back the checking result.
[0092] In another embodiment, the quality technology-AI interface checking tool is also used for querying all the data of the corresponding interface information in the ES and pushing the data to the AI model, and the AI model feeds back the generated rules to the quality technology-AI interface checking tool after generating the rules according to the data, and the quality technology-AI interface checking tool stores the generated rules.
[0093] The above embodiments only express the specific implementation of the present application, which is described in more detail and specifically, but cannot be understood as a limitation to the protection scope of the present application. It should be noted that for those skilled in the art, without departing from the technical concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.
Claims
1. An AI capability and data rule based data quality assurance method, characterized in that, The method comprises the following steps: S1, obtaining different interface information; S2: based on different interface information, recording data flow and extracting information of corresponding interfaces, and converting the information into structured data; Based on different interface information, different data rules are generated by using the data source of the interface specification system, the enterprise master data and the dynamic rule generation mechanism of the AI model; S3, the structured data in S2 and the generated data rules are transmitted to the AI model for verification, and the feedback is made according to the verification result. 2.The AI capability and data rule based data quality assurance method of claim 1, wherein, The S2 comprises the following steps: S2.1, using a data acquisition tool, setting a monitoring point on a data transmission channel, capturing a data packet flowing through, analyzing and classifying it, screening out the data packet related to the target interface, and obtaining the initial interface request and response information; S2.2, through the captured interface request and response information, the extracted information is converted into a structured data format (JSON). 3.The AI capability and data rule based data quality assurance method of claim 1, wherein, The dynamic rule generation mechanism of the S2 comprises three data rule generation methods: The first method is to connect the data source of the interface specification system, extract the data information of the interface by using quality technology-AI checking tool, and the interface specification system is ApiDoc system; The second method is to connect the data source of the enterprise internal master data management specification, and extract the data information managed by the enterprise; The third method is to use AI to automatically derive or assist in generating rules for checking the correctness of future interaction data according to the data characteristics of the sample data. 4.The AI capability and data rule based data quality assurance method of claim 1, wherein, The S3 comprises the following steps: S3.1, pre-training the AI model A large number of sample data conforming to the data quality rules and existing common data problems are used to learn the normal mode and abnormal characteristics of the data through the model, form a relatively complete knowledge base, and thus complete the pre-training of the AI model; S3.2, input the data to be checked and the rules into the AI model trained in S3.1, the model extracts features and matches patterns of the data, judges whether the data meets the rule requirements and feeds back the checking result.
5. The data quality assurance method based on AI capability and data rules according to claim 4, characterized in that, S3.3 is further included in S3: when the AI model finds information that does not meet the rules, a professional person confirms it manually and feeds back the checking result.
6. The data quality assurance method based on AI capability and data rules according to claim 3, characterized in that, When the rules in the ApiDoc system are incomplete or unclear, the missing or unclear rules are supplemented by manual editing.
7. The data quality assurance method based on AI capability and data rules according to claim 1, characterized in that, Further comprising: Before the interface request is sent to AI verification, it is checked whether it passes, such as interface response rejection, the data is considered as interface failure, and AI checking is not performed, and if the interface response is passed, the data is sent to AI verification.
8. An AI capability and data rule based data quality assurance system, characterized in that, The method for implementing the AI capability and data rule based data quality guarantee method of any one of claims 1-7 comprises: A flow recording system for recording data flow of corresponding interfaces in the system based on interface information in a test environment; An ES system for storing data captured from different interfaces by the flow recording system; A quality technology-AI interface checking tool for extracting corresponding interface information from the ES system, connecting the ApiDoc system, the data governance management system, the rule generation / configuration, pushing the AI model checking, and the AI checking report; AI model: used for feature extraction and pattern matching of data, to determine whether the data meets the rule requirements and feedback the review results. 9.The AI capability and data rule based data quality assurance system according to claim 8, wherein, The quality technology-AI interface review tool is also used to query all data of the corresponding interface information in the ES and push the data to the AI model, and the AI model generates rules according to the data and feeds back to the quality technology-AI interface review tool.