Data quality detection method, device, medium, and program product

Abstract

The application discloses a data quality detection method, device, medium and program product. The method comprises the following steps: acquiring to-be-detected data, and identifying the data type of the to-be-detected data; determining an initial detection rule matched with the data type from a preset quality detection rule library according to the data type of the to-be-detected data; detecting the data quality of the to-be-detected data according to the initial detection rule, and obtaining a quality detection result of the to-be-detected data; inputting the quality detection result into a pre-trained data analysis model, and obtaining a data analysis result output by the data analysis model; adjusting the initial detection rule according to a first rule adjustment mode, obtaining an adjusted detection rule, and detecting the to-be-detected data according to the adjusted detection rule, and obtaining a quality detection update result. The scheme of the application avoids the case that data quality detection can only be performed through existing rules, and improves the accuracy of data quality detection.

Description

Technical Field

[0001] This application relates to the field of data quality inspection technology, and in particular to a data quality inspection method, equipment, medium and program product. Background Technology

[0002] With the development of the information age and the advancement of digital transformation, the amount of data held by enterprises and organizations is growing exponentially. Currently, when enterprises and organizations exchange massive and diverse amounts of data, they need the data to meet quality requirements such as accuracy and completeness.

[0003] Currently, the data quality of the data in the corresponding database is mainly tested through a variety of pre-set data quality detection rules corresponding to the database. Data exchange between enterprises and organizations can only be carried out if the data quality test is passed.

[0004] However, the data quality of the data in the corresponding database is detected by a variety of pre-set data quality detection rules corresponding to the database. However, the data quality detection can only be performed using the existing rules. Therefore, after the data changes, the detection rules cannot be applied to the changed data, resulting in inaccurate data quality detection and a high false alarm rate and false negative rate. Summary of the Invention

[0005] This application provides a data quality detection method, device, medium, and program product to solve the problem that in the prior art, data quality detection is performed on data in a corresponding database by using a variety of pre-set data quality detection rules corresponding to the database. However, this method can only perform data quality detection based on existing rules, and therefore the detection rules cannot be applied to the changed data after the data changes. This results in inaccurate data quality detection and a high false alarm rate and false negative rate in the quality detection results.

[0006] Firstly, this application provides a data quality detection method, the method comprising:

[0007] Acquire the data to be detected and identify the data type of the data to be detected;

[0008] Based on the data type of the data to be detected, an initial detection rule matching the data type is determined from a preset quality detection rule base;

[0009] The data quality of the data to be tested is detected according to the initial detection rules, and the quality detection result of the data to be tested is obtained.

[0010] The quality inspection results are input into a pre-trained data analysis model to obtain the data analysis results output by the data analysis model; wherein, the data analysis results include, but are not limited to, the first rule adjustment method;

[0011] The initial detection rule is adjusted according to the first rule adjustment method to obtain the adjusted detection rule, and the data to be detected is detected according to the adjusted detection rule to obtain the quality detection update result.

[0012] Secondly, this application provides a data quality inspection device, the device comprising:

[0013] The acquisition module is used to acquire the data to be detected and identify the data type of the data to be detected;

[0014] The determination module is used to determine an initial detection rule that matches the data type from a preset quality detection rule library based on the data type to be detected.

[0015] The detection module is used to detect the data quality of the data to be detected according to the initial detection rules, and obtain the quality detection result of the data to be detected.

[0016] An input module is used to input the quality inspection results into a pre-trained data analysis model to obtain the data analysis results output by the data analysis model; wherein, the data analysis results include, but are not limited to, the first rule adjustment method;

[0017] The adjustment module is used to adjust the initial detection rule according to the first rule adjustment method to obtain the adjusted detection rule, and to detect the data to be detected according to the adjusted detection rule to obtain the quality detection update result.

[0018] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data quality detection method as described in any embodiment of this application.

[0019] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data quality detection method as described in any embodiment of this application.

[0020] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data quality detection method as described in any embodiment of this application.

[0021] The solution of this application involves acquiring data to be tested and identifying its data type; determining an initial detection rule matching the data type from a pre-set quality detection rule library; detecting the data quality of the data to be tested according to the initial detection rule to obtain a quality detection result; inputting the quality detection result into a pre-trained data analysis model to obtain the data analysis result output by the data analysis model; wherein the data analysis result includes, but is not limited to, a first rule adjustment method; adjusting the initial detection rule according to the first rule adjustment method to obtain an adjusted detection rule; and detecting the data to be tested according to the adjusted detection rule to obtain an updated quality detection result. In other words, the solution of this application analyzes the data using the data quality detection result obtained after using the initial detection rule, and adjusts the initial detection rule based on the data analysis result, thereby avoiding the situation where data quality detection can only be performed using existing rules, improving the accuracy of data quality detection, and reducing the false positive and false negative rates of the quality detection results obtained after detecting data quality according to the detection rules. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the data quality testing method provided in this application;

[0024] Figure 2 This is a schematic diagram of the training process of the data analysis model for the data quality detection method provided in this application;

[0025] Figure 3 This is a schematic diagram of the data quality testing device provided in this application;

[0026] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Figure 1 This is a flowchart illustrating a data quality inspection method provided in this application. This method can be executed by a data quality inspection device, which can be implemented using software and / or hardware. In a specific embodiment, the device can be applied in an electronic device, which can be a computer. The following embodiments will illustrate this using the application of the device in an electronic device as an example. Figure 1 The method may specifically include the following steps:

[0030] Step 101: Obtain the data to be detected and identify the data type of the data to be detected.

[0031] Specifically, the data to be tested can be provided by an upstream system to the electronic device executing this embodiment. For example, the electronic device executing this embodiment is a quality inspection system, and the upstream system is a directory system used to dispatch quality inspection tasks to the quality inspection system. The data to be tested may come from different data sources, such as databases, file systems, or data interfaces. The upstream system will transmit the data to the data quality inspection system in a specific format, such as structured data, semi-structured data, or unstructured data. The electronic device executing this embodiment receives the data to be tested. Data type identification is used to identify the data type of the data to be tested, so that appropriate detection rules can be selected according to the data type. For example, the detection rules for structured data and unstructured data are different. The data type identification of the data to be tested can be based on file extension identification, data content analysis identification, structured data identification, semi-structured data identification, unstructured data identification, or metadata identification, etc. File extension identification determines the data type by the file extension, such as identifying files with the .csv extension as structured data. Data content analysis can identify structured data such as database tables or similar structured files by analyzing their format. This format can include column names, data type definitions, etc. For example, checking if the data conforms to a table format and has clearly defined column names and data types. Semi-structured data can be identified through its format; for example, JSON data is typically in key-value pair format, and identifying this format determines the data type. Unstructured data can be identified through file header information or file content characteristics. Metadata-based identification can identify data types by analyzing metadata, which usually contains a detailed description of the data.

[0032] Step 102: Based on the data type of the data to be tested, determine the initial detection rule that matches the data type from the preset quality detection rule library.

[0033] Specifically, the pre-defined quality inspection rule base is a collection of pre-defined data quality inspection rules. These rules are used to evaluate data integrity, consistency, uniqueness, accuracy, timeliness, and other quality indicators. Rules in the rule base are categorized and stored according to different data types to quickly match appropriate inspection rules for those data types. For example, for data in a database table, inspection rules might include field null checks, data type checks, field value range checks, and foreign key association checks. The rule base supports rule creation, modification, deletion, version control, and permission management. It also has a user interface, allowing staff to define and maintain the rule base according to actual needs. Initial inspection rules matching the data type are retrieved from the rule base. Rule instances can also be dynamically generated based on the specific characteristics of the data to be inspected. For example, for a database table with multiple fields, corresponding null check rules and data type check rules are generated based on the definition of each field. The rule templates retrieved from the rule base are instantiated into specific inspection rules. The instantiation process includes replacing placeholders in the rule template with the specific characteristics of the data to be inspected, such as field names and key names. After the matching process is complete, the system will generate an initial detection rule that can completely detect the data, or generate a set of initial detection rules that match the data type to be detected. These rule sets will serve as the basis for subsequent data quality detection and will be used to evaluate the quality indicators of the data to be detected.

[0034] Step 103: Detect the data quality of the data to be detected according to the initial detection rules, and obtain the quality detection result of the data to be detected.

[0035] Specifically, before formal quality testing, the system preprocesses the data to be tested to ensure that the data format meets the requirements of the testing rules. Preprocessing may include data cleaning and data format conversion operations, such as removing duplicate data and filling missing values. The purpose of preprocessing is to reduce the interference of data quality issues on the testing process and improve testing efficiency and accuracy. The system then tests the data quality of the data to be tested according to the initial testing rules. Each rule evaluates specific quality indicators of the data, such as completeness, consistency, uniqueness, accuracy, and timeliness, resulting in the quality test results. Completeness testing checks for missing values, such as whether fields are empty. Consistency testing checks whether the data conforms to predefined formats and constraints. Uniqueness testing checks for duplicate records. Accuracy testing checks whether the data conforms to business logic and data standards. Timeliness testing checks whether the data is within a valid time frame. The test results for each rule are recorded, including the testing indicators, test results, and problem descriptions. Finally, the overall quality of the data is comprehensively evaluated based on the test results. This comprehensive evaluation may consider quality scores from multiple dimensions and generate a comprehensive quality score through weighted averaging or other statistical methods. The overall quality score reflects the overall quality level of the data, providing a reference for subsequent data quality improvement. A quality report can also be generated based on the test results and comprehensive evaluation. This report includes the overall data quality score, detailed test results for each quality indicator, including rules that passed and failed, descriptions of the problems and impact scope of the failed rules, and improvement suggestions and optimization directions. Therefore, the quality test results for the data to be tested include the test results and overall quality score for each quality indicator, quality problems in the data to be tested, and detailed descriptions and impact scope of these quality problems.

[0036] Optionally, there may be multiple initial detection rules, and step 103 can be implemented through steps 1031 to 1032.

[0037] Step 1031: Detect the data quality of the data to be detected according to multiple initial detection rules, and obtain the data quality detection results of multiple detection dimensions of the data to be detected.

[0038] Each initial detection rule corresponds to a detection dimension.

[0039] Specifically, multiple initial detection rules are determined from a pre-defined quality detection rule library based on the data type of the data to be tested. These rules match the data type and are used to assess the data's quality indicators such as completeness, consistency, uniqueness, accuracy, and timeliness. Each initial detection rule corresponds to a detection dimension, ensuring that data quality is evaluated from multiple dimensions. For example, the completeness rule detects whether the data contains missing values, the consistency rule detects whether the data conforms to predefined formats and constraints, the uniqueness rule detects whether the data contains duplicate records, the accuracy rule detects whether the data conforms to business logic and data standards, and the timeliness rule detects whether the data is within a valid time frame. Based on the data type of the data to be tested, initial detection rules matching the data type are selected from the pre-defined quality detection rule library. By using multiple initial detection rules to perform multi-dimensional detection on the data, with each rule evaluating a specific quality indicator, the data quality detection results for the data to be tested across multiple dimensions can be obtained.

[0040] For example, the data quality of the data to be tested is assessed according to integrity rules, consistency rules, uniqueness rules, accuracy rules, and timeliness rules. The integrity test results are: integrity score, list of missing fields, and scope of impact. The consistency test results are: consistency score, list of fields that do not conform to the format, and scope of impact. The uniqueness test results are: uniqueness score, list of duplicate records, and scope of impact. The accuracy test results are: accuracy score, list of fields that do not conform to business logic, and scope of impact. The timeliness test results are: timeliness score, list of fields that are outside the time frame, and scope of impact. This ensures a comprehensive assessment of data quality from multiple dimensions, providing detailed information for subsequent updates to the data quality testing rules.

[0041] Step 1032: Determine the quality detection result of the data to be detected based on the data quality detection results of multiple detection dimensions of the data to be detected.

[0042] Specifically, the quality inspection results for each inspection dimension can include scores for quality indicators such as completeness, consistency, uniqueness, accuracy, and timeliness. After obtaining the data quality inspection results for multiple inspection dimensions of the data to be inspected, the overall quality inspection result of the data is determined. For example, weights can be assigned to each inspection dimension based on business needs and the importance of the data. This weight allocation can be based on historical data, business logic, or user-defined priorities. For example, the weight for completeness is 0.25, for consistency it is 0.25, for uniqueness it is 0.2, for accuracy it is 0.2, and for timeliness it is 0.1. After weight allocation, the overall quality inspection result of the data to be inspected is determined based on the data quality inspection results for multiple inspection dimensions and the corresponding weight information for each dimension. The method for determining the quality inspection result of the data to be inspected can be to calculate the overall quality score using a weighted average method based on the quality inspection scores for each dimension. Furthermore, the problem descriptions and impact scope of all inspection dimensions are summarized to generate a problem summary table, which allows users to quickly understand the distribution of data quality problems. Therefore, the obtained quality inspection result can include an overall quality score, a problem summary table, and the impact scope. The overall quality score reflects the overall level of data quality. The issue summary table lists all issues identified across all data quality inspection dimensions. The impact scope indicates the overall impact of all issues, providing visualization of the overall effect. The quality inspection results may also include suggestions for improving the quality inspection rules, making them more applicable to the data being inspected. Based on the data quality inspection results from multiple dimensions, a comprehensive quality inspection result is determined for the data being inspected, ensuring a holistic assessment of data quality from multiple dimensions. This provides users with an overall data quality inspection result, facilitating their decision-making and optimizing the data quality management process.

[0043] Step 104: Input the quality inspection results into the pre-trained data analysis model to obtain the data analysis results output by the data analysis model.

[0044] The data analysis results include, but are not limited to, the adjustment method of the first rule.

[0045] Specifically, a data analysis model is built based on machine learning or deep learning algorithms to analyze quality inspection results. The output data analysis results include patterns and trends of data quality problems and provide optimization suggestions for quality inspection rules. That is, the model's goal is to identify patterns and trends of data quality problems through quality inspection results and propose suggestions for improving quality inspection rules. The model's input includes quality inspection results, such as the results of indicators like completeness, consistency, uniqueness, accuracy, and timeliness. The model can be trained using historical inspection results and historical rule adjustment records. The model's output is the patterns and trends of data quality problems, as well as rule adjustment suggestions, i.e., the first rule adjustment method, which can include adding rules, modifying existing rules, or deleting invalid rules. For example, if the model finds that a certain field has frequent format errors, it may suggest modifying the validation rules for that field, adding stricter format checks. The data analysis results are stored in the data storage and algorithm layer for subsequent optimization and improvement. By inputting the quality inspection results into a pre-trained data analysis model, the data analysis results output by the data analysis model are obtained. These data analysis results include rule adjustment suggestions, thereby optimizing the data quality inspection rules.

[0046] Optionally, a data quality assessment report can be generated based on the quality inspection results, abnormal data, and data quality change trends.

[0047] Specifically, the quality inspection results can include the quality indicator score for each piece of data to be inspected, a description of the detected problems, and the scope of the problem's impact. Abnormal data refers to data points that do not conform to normal patterns or regularities, identified through anomaly detection algorithms. The labeling information for abnormal data includes the type of anomaly, its cause, and the scope of its impact. The data quality change trend is the output obtained by analyzing historical and current quality inspection results through a time-series prediction model. The data quality change trend includes predicted values ​​of quality indicators, rising or falling quality trends, and potential anomalies. Based on the quality inspection results, abnormal data, and data quality change trends, a data quality assessment report can be generated. This report includes data source information, the inspection time range, data quality inspection results, detected abnormal data, and predicted data quality change trends. Analyzing the impact of abnormal data on overall data quality allows for the assessment of its potential risks to business processes. Furthermore, the data quality assessment report can include processing suggestions for abnormal data, such as data correction or data cleaning. The data quality assessment report provides predictions of future data quality change trends, including predicted values ​​of quality indicators and trend directions, and analyzes the trend prediction results to identify potential data quality anomalies and corrective measures. Data quality assessment reports can also be exported as electronic documents, making it easy for staff to share and archive them.

[0048] Optionally, the data quality assessment report can be sent to the display module so that the display module can display the data quality assessment report.

[0049] Specifically, the data quality assessment report format may include text, charts, and other visual elements to facilitate reading and understanding by staff. If the generated data quality assessment report is in a format unsuitable for display, the report content will be formatted into a display-appropriate format, such as PDF or the internal visualization format of the electronic device executing this embodiment, ensuring the report content is clear and easy for staff to read and understand. The formatted data quality assessment report is then sent to the display module, which, upon receiving the report, displays it. The display module can be a user interface or other visualization tool. For example, the display method could be directly showing the report content on a web interface or mobile application. This ensures that users can promptly obtain and understand data quality information to take corresponding improvement measures and optimize the data quality detection process.

[0050] Optionally, after performing step 104, steps 41 to 42 may also be performed.

[0051] Step 41: Identify the abnormal data corresponding to the quality inspection results according to the preset anomaly identification algorithm.

[0052] Specifically, an anomaly detection algorithm is a pre-defined method used to detect data that does not conform to normal patterns or regularities. In data quality inspection, anomalous data may include erroneous data, outliers, and data that does not conform to business logic. Common anomaly detection algorithms include statistical methods, machine learning-based methods, and deep learning-based methods. Anomaly detection algorithms corresponding to different data types can be pre-defined based on the characteristics of the data and business needs. For example, for structured data, statistical methods can be used to detect outliers to identify anomalous data; for complex data, machine learning or deep learning-based methods, such as Isolation Forest and One-Class SVM, can be used. The quality inspection results are input into the anomaly detection algorithm. These results include the quality index score and problem description for each data record. The anomaly detection algorithm identifies anomalous data based on the input quality inspection results. For example, the algorithm analyzes the quality index scores of data records to determine if there are data points that do not conform to normal patterns. After identifying the anomalous data corresponding to the quality inspection results, the anomalous data is marked, and the type and cause of the anomaly are recorded. For example, the anomaly is marked as an integrity anomaly. By identifying anomalous data corresponding to the quality inspection results according to the pre-defined anomaly detection algorithm, anomaly detection of data quality problems is achieved, providing data support for data quality improvement.

[0053] Step 42: Determine the second rule adjustment method based on the data type of the abnormal data.

[0054] Specifically, based on the data type of the abnormal data, the adjustment method for the rules corresponding to each data type of abnormal data is determined. For example, if the abnormal data is structured data and has integrity issues, the second rule adjustment method might include adding rules to fill in missing fields, such as using default values ​​or filling data through data inference. Alternatively, the non-null validation rules for fields might be adjusted, such as adding non-null requirements for specific fields. If the abnormal data has consistency issues, the second rule adjustment method might include adding or modifying format validation rules for field values, such as expanding or narrowing the allowed range of field values. In other words, the second rule adjustment method, based on business needs, determines adjustment methods that can make the screening of abnormal data more stringent or reduce the amount of abnormal data, depending on the data type of the abnormal data.

[0055] Optionally, after performing step 42, steps 43 to 45 may also be performed.

[0056] Step 43: Obtain historical quality inspection results.

[0057] Specifically, historical quality inspection results are the results generated during historical data quality inspection tasks. The information type of historical quality inspection results is consistent with the information type of the quality inspection results obtained in this application. For example, historical quality inspection results include the quality index score for each data record, a description of the detected problem, the scope of the problem's impact, and the corresponding detection rules. Historical quality inspection results can be obtained from upstream systems or from data stored and read by the electronic device executing this embodiment.

[0058] Step 44: Input the historical quality inspection results and the quality inspection results into the preset time series prediction model to obtain the data quality change trend output by the preset time series prediction model.

[0059] Specifically, the preset time-series prediction model is a model used to analyze and predict time-series data. For example, the preset time-series prediction model could be an autoregressive integral moving average model, a long short-term memory network, or other time-series prediction models. In data quality inspection, the time-series prediction model is used to analyze the time series of historical and current quality inspection results and predict future trends in data quality. The model's input includes historical quality inspection results and current quality inspection results. The time-series prediction model is a model trained on historical training data that can predict trends in data quality. The model outputs trends in data quality, which may include, for example, predicted values ​​of quality indicators at various future time points, upward or downward trends in quality indicators, and predictions of outliers. The prediction results can identify potential data quality problems in advance, providing data support for data quality improvement. After outputting the data quality trend, a trend analysis report can also be generated, detailing the prediction results. The trend analysis report can be displayed to users through the user interface layer, and users can view and manage the trend analysis report through the report center.

[0060] Step 45: Determine the adjustment method for the third rule based on the trend of data quality changes.

[0061] Specifically, the data quality change trend is obtained by analyzing historical quality inspection results and current quality inspection results using a pre-defined time-series prediction model. The trend can include predicted values ​​of quality indicators, predicted data quality trend direction, and potential outliers. Based on the data quality change trend, a third rule adjustment method is determined to address predicted potential data quality issues. This adjustment method can include adding rules, modifying existing rules, or deleting invalid rules. For example, based on the data quality change trend, the third rule adjustment method could be: if the trend indicates an increase in certain data quality issues, the third rule adjustment method would be to add a rule to address those issues; if an increase in integrity issues for a field is predicted, the third rule adjustment method would be to add a rule to fill in missing values; if the trend indicates that parameters of existing rules need adjustment, the third rule adjustment method would be to modify existing rules. For example, if a change in the value range of a field is predicted, the third rule adjustment method would be to adjust the value range validation rule for that field. Rule modifications may include adjusting field value ranges, adding or deleting validation conditions, and modifying rule priorities. If the trend of data quality changes shows that certain rules are no longer effective or applicable, the third rule adjustment method is to delete the rule.

[0062] Step 105: Adjust the initial detection rules according to the first rule adjustment method to obtain the adjusted detection rules, and detect the data to be detected according to the adjusted detection rules to obtain the quality detection update results.

[0063] Specifically, after obtaining the first rule adjustment method from the data analysis results output by the data analysis model, the initial detection rules are adjusted according to this method to obtain the adjusted detection rules. For example, if the first rule adjustment method suggests adding a new rule, a new rule is created accordingly. The definition of the new rule is configured based on the first rule adjustment method, which may include the rule's name, description, applicable data type, and detection logic. If the first rule adjustment method suggests modifying an existing rule, the rules' parameters or logic are adjusted accordingly. For example, the field value range may be adjusted, validation conditions may be added or deleted, or the rule's priority may be modified. If the first rule adjustment method suggests deleting invalid rules, invalid rules are removed from the rule base according to this method. After adjusting the initial detection rules, the adjusted detection rules are obtained.

[0064] After obtaining the adjusted detection rules, the data to be detected is re-detected using the adjusted rules. This detection process is similar to the initial detection process, but uses the adjusted detection rules. After the detection is completed, quality detection update results are generated. The obtained quality detection update results can reflect the detection results of the adjusted rules on data quality assessment, as well as the improvements made by the adjusted rules on data quality assessment.

[0065] In one possible implementation, after obtaining the adjusted detection rules, the adjusted detection rules are used as the new initial detection rules, and the execution returns to step 103 until a preset iteration termination condition is reached. The quality detection result of the data to be detected is then used as the final quality detection result. The preset iteration termination condition can be that the number of iterations reaches a preset number. Thus, the data quality detection rules are optimized multiple times during the iteration process, resulting in optimized rules, and the data quality of the data to be detected is tested according to the final rules.

[0066] Optionally, after performing steps 41 to 42, adjusting the initial detection rules according to the first rule adjustment method can be achieved through step 1051.

[0067] Step 1051: Adjust the initial detection rules according to the first rule adjustment method and the second rule adjustment method.

[0068] Specifically, the first rule adjustment method is derived from the overall analysis of quality inspection results by the data analysis model. For example, the data analysis model might suggest adding a format validation rule for a certain field. The second rule adjustment method is derived from anomaly analysis based on anomaly detection algorithms, focusing more on resolving specific anomaly issues. For example, anomaly detection algorithms might find that some data records have format errors or missing values; the second rule adjustment method is for adjusting the data in these anomaly records. After obtaining the first and second rule adjustment methods, they are merged, and the initial detection rules are adjusted. When merging the first and second rule adjustment methods, if the two methods conflict on the same rule, the priority of the first and second rule adjustment methods is used to determine the winner. For example, the first rule adjustment method has a higher priority than the second rule adjustment method. If the first rule adjustment method suggests deleting rule A, while the second rule adjustment method suggests modifying rule A, then rule A is deleted based on priority. After the rule adjustment is completed, historical data or test data are used to test the adjusted rules to ensure their correctness and effectiveness. If issues are found with the rules, they are further adjusted based on the verification results. After adjusting the initial detection rules according to the first and second rule adjustment methods, detailed information about the rule adjustments is recorded for easy review by staff. By adjusting the initial detection rules according to the first and second rule adjustment methods, the rules in the data quality detection process are optimized, thereby improving the accuracy and efficiency of data quality detection.

[0069] Optionally, after performing step 45, step 1051 can be implemented through step 1052.

[0070] Step 1052: Adjust the initial detection rules according to the first rule adjustment method, the second rule adjustment method, and the third rule adjustment method.

[0071] Specifically, the first rule adjustment method is derived from the overall analysis of quality inspection results by the data analysis model. The second rule adjustment method is derived from anomaly detection algorithms based on anomaly data analysis, focusing more on solving specific anomaly data problems. The third rule adjustment method is derived from the output of a time-series prediction model after predicting data quality change trends, focusing more on addressing potential future data quality issues. The three adjustment methods are combined, and when they conflict for the same rule, the priority strategy of the first, second, and third rule adjustment methods is used to determine the priority. For example, the first rule adjustment method has higher priority than the second, and the second has higher priority than the third. If the first rule adjustment method suggests deleting rule A, the second suggests modifying rule A, and the third suggests adding rule B, then rule A is deleted based on priority. After the rule adjustment is completed, historical data or test data are used to test the adjusted rules to ensure their correctness and effectiveness. If problems are found, the rules are further adjusted based on the verification results. After adjusting the initial detection rules according to the first, second, and third rule adjustment methods, detailed information on the rule adjustments is recorded for easy review by staff. By comprehensively considering the first, second, and third rule adjustment methods, the initial detection rules are optimized to further improve the accuracy and adaptability of data quality detection.

[0072] The solution of this application involves acquiring data to be tested and identifying its data type; determining an initial detection rule matching the data type from a pre-set quality detection rule library; detecting the data quality of the data to be tested according to the initial detection rule to obtain a quality detection result; inputting the quality detection result into a pre-trained data analysis model to obtain the data analysis result output by the data analysis model; wherein the data analysis result includes, but is not limited to, a first rule adjustment method; adjusting the initial detection rule according to the first rule adjustment method to obtain an adjusted detection rule; and detecting the data to be tested according to the adjusted detection rule to obtain an updated quality detection result. In other words, the solution of this application analyzes the data using the data quality detection result obtained after using the initial detection rule, and adjusts the initial detection rule based on the data analysis result, thereby avoiding the situation where data quality detection can only be performed using existing rules, improving the accuracy of data quality detection, and reducing the false positive and false negative rates of the quality detection results obtained after detecting data quality according to the detection rules.

[0073] Figure 2This is a schematic diagram of the training process of the data analysis model for the data quality detection method provided in this application. This embodiment... Figure 1 Based on the illustrated embodiments and various optional implementation schemes, the training steps of the data analysis model are described in detail. For example... Figure 2 As shown, the method may include the following steps:

[0074] Step 201: Obtain the training quality detection results and training data analysis results.

[0075] The training quality detection results and training data analysis results are a set of training data in a preset training database. The preset training database includes multiple sets of training data, and each set of training data includes a set of corresponding training quality detection results and training data analysis results.

[0076] Specifically, the preset training database stores multiple sets of training data, each set including a corresponding set of training quality detection results and training data analysis results. The training quality detection results and training data analysis results in the preset training database can be the quality detection results and data analysis results of historical detection data from the same source as the data to be detected in this application, thereby improving the output accuracy of the model obtained based on the training data when analyzing the data to be detected. The data format of the training quality detection results in the preset training database is the same as that of the quality detection results in any embodiment of this application, ensuring that the feature vectors during model training are consistent with the feature vector format in subsequent applications, thereby improving the accuracy of the model output results. The training data analysis results are the data analysis results corresponding to the training quality detection results.

[0077] Step 202: Input the training quality detection results into the model to be trained to obtain the training analysis results output by the model to be trained.

[0078] Specifically, the model to be trained can be any machine learning or deep learning architecture. By inputting the training quality detection results into the model to be trained, the training analysis results corresponding to the training quality detection results output by the model can be obtained.

[0079] Step 203: Based on the analysis results of the training data and the analysis results of the training data, adjust the target model parameters of the training model to obtain the training model with adjusted parameters.

[0080] Specifically, based on the results of the training analysis and the training data analysis, the target model parameters of the model to be trained are adjusted, such as adjusting the learning rate, to obtain the parameter-adjusted model to be trained. Optionally, the target model parameters include the learning rate and / or the length of the soft cue embedding. Choosing a smaller learning rate and a longer soft cue embedding can ensure the stability of the model training process while avoiding overfitting.

[0081] Step 204: Use the parameter-adjusted training model as the new training model, use the training quality detection result from an uninputted training model in the preset training database as the new training quality detection result, use the training data analysis result corresponding to the training quality detection result as the new training data analysis result, return to step 202, until the preset iteration end condition is reached, and use the parameter-adjusted training model as the data analysis model.

[0082] Specifically, the process involves taking the parameter-adjusted model to be trained as the new model, using a training quality detection result from a pre-defined training database that was not input into the model as the new training quality detection result, and using the corresponding training data analysis result as the new training data analysis result. The process then returns to the step of inputting the training quality detection result into the model to obtain the training analysis result output by the model, until a pre-defined iteration termination condition is reached. For example, the pre-defined iteration termination condition could be stopping training when a certain cutoff condition is reached, i.e., stopping the return steps. The cutoff condition could be reaching a pre-defined number of training epochs, or the training loss falling below a certain value, etc. The resulting parameter-adjusted model is then the data analysis model.

[0083] The solution of this application pre-sets that the data format of the quality detection results to be trained in the training database is the same as that of the quality detection results in any embodiment of this application, so that the feature vector during model training is consistent with the feature vector format in subsequent applications, thereby improving the accuracy of the model output results.

[0084] Figure 3 This is a schematic diagram of the data quality inspection apparatus provided in this application, which is suitable for performing the data quality inspection method provided in this application. Figure 3 As shown, the device may specifically include:

[0085] The acquisition module 301 is used to acquire the data to be detected and identify the data type of the data to be detected.

[0086] The determination module 302 is used to determine an initial detection rule that matches the data type from a preset quality detection rule library based on the data type to be detected.

[0087] The detection module 303 is used to detect the data quality of the data to be detected according to the initial detection rules, and obtain the quality detection result of the data to be detected.

[0088] The input module 304 is used to input the quality detection result into a pre-trained data analysis model to obtain the data analysis result output by the data analysis model; wherein, the data analysis result includes, but is not limited to, the first rule adjustment method.

[0089] The adjustment module 305 is used to adjust the initial detection rule according to the first rule adjustment method to obtain the adjusted detection rule, and to detect the data to be detected according to the adjusted detection rule to obtain the quality detection update result.

[0090] In one embodiment, the input module 304 is further configured to: input the quality detection result into a pre-trained data analysis model, obtain the data analysis result output by the data analysis model, identify the abnormal data corresponding to the quality detection result according to a preset anomaly identification algorithm; determine a second rule adjustment method according to the data type of the abnormal data; the adjustment module 305 is specifically configured to: adjust the initial detection rule according to the first rule adjustment method and the second rule adjustment method in terms of adjusting the initial detection rule.

[0091] In one embodiment, the input module 304 is further configured to: after determining the second rule adjustment method according to the data type of the abnormal data, obtain historical quality detection results; input the historical quality detection results and the quality detection results into a preset time series prediction model to obtain the data quality change trend output by the preset time series prediction model; determine a third rule adjustment method according to the data quality change trend; the adjustment module 305 is specifically configured to: adjust the initial detection rule according to the first rule adjustment method, the second rule adjustment method and the third rule adjustment method in terms of adjusting the initial detection rule according to the first rule adjustment method and the second rule adjustment method.

[0092] In one embodiment, the device further includes a report generation module, which is used to input the historical quality detection results and the quality detection results into a preset time series prediction model after the input module 304 inputs them into the preset time series prediction model to obtain the data quality change trend output by the preset time series prediction model, and then generates a data quality assessment report based on the quality detection results, the abnormal data, and the data quality change trend.

[0093] In one embodiment, the apparatus further includes a sending module, configured to send the data quality assessment report to the display module after the report generation module generates the data quality assessment report based on the quality inspection results, the abnormal data, and the data quality change trend, so that the display module can display the data quality assessment report.

[0094] In one embodiment, there are multiple initial detection rules, and the detection module 303 is specifically used to: detect the data quality of the data to be detected according to the multiple initial detection rules, and obtain data quality detection results of multiple detection dimensions of the data to be detected; wherein, each initial detection rule corresponds to a detection dimension; and determine the quality detection result of the data to be detected based on the data quality detection results of the multiple detection dimensions of the data to be detected.

[0095] In one embodiment, the device further includes: a training module, configured to acquire training quality detection results and training data analysis results; wherein the training quality detection results and the training data analysis results are a set of training data in a preset training database, the preset training database including multiple sets of data to be trained, each set of data to be trained including a set of corresponding training quality detection results and training data analysis results; inputting the training quality detection results into the model to be trained to obtain the training analysis results output by the model to be trained; adjusting the target model parameters of the model to be trained according to the training analysis results and the training data analysis results to obtain a parameter-adjusted model to be trained; using the parameter-adjusted model to be trained as a new model to be trained, using a training quality detection result from the preset training database that was not input into the model to be trained as a new training quality detection result, using the training data analysis results corresponding to the training quality detection result as a new training data analysis result, returning to execute the step of "inputting the training quality detection results into the model to be trained to obtain the training analysis results output by the model to be trained" until a preset iteration termination condition is reached, and using the parameter-adjusted model to be trained as the data analysis model.

[0096] The apparatus of this application acquires data to be tested and identifies the data type of the data to be tested; determines an initial detection rule matching the data type from a preset quality detection rule library based on the data type of the data to be tested; detects the data quality of the data to be tested according to the initial detection rule to obtain a quality detection result of the data to be tested; inputs the quality detection result into a pre-trained data analysis model to obtain a data analysis result output by the data analysis model; wherein, the data analysis result includes, but is not limited to, a first rule adjustment method; adjusts the initial detection rule according to the first rule adjustment method to obtain an adjusted detection rule, and detects the data to be tested according to the adjusted detection rule to obtain a quality detection update result. In other words, the solution of this application analyzes the data using the data quality detection result obtained after using the initial detection rule, and adjusts the initial detection rule according to the data analysis result, thereby avoiding the situation where data quality detection can only be performed using existing rules, improving the accuracy of data quality detection, and reducing the false alarm rate and false negative rate of the quality detection result obtained after detecting data quality according to the detection rule.

[0097] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data quality detection method provided in any of the above embodiments.

[0098] This application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data quality detection method provided in any of the above embodiments.

[0099] The following is for reference. Figure 4 It shows a schematic diagram of the structure of an electronic device 400 suitable for implementing the present application. Figure 4 The electronic device shown is merely an example and should not impose any limitations on the functionality and scope of this application.

[0100] like Figure 4 As shown, the electronic device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0101] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed.

[0102] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined above in the system of this application.

[0103] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The modules and / or units described in this application can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, it can be described as: a processor applied to a user equipment, including an operation information acquisition module, a determination module, a detection module, an input module, and an adjustment module. The names of these modules do not, in certain circumstances, constitute a limitation on the module itself.

[0106] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform the following operations:

[0107] The process involves: acquiring the data to be tested and identifying its data type; determining an initial detection rule matching the data type from a pre-defined quality detection rule library; detecting the data quality of the data to be tested according to the initial detection rule to obtain the quality detection result; inputting the quality detection result into a pre-trained data analysis model to obtain the data analysis result output by the model; wherein the data analysis result includes, but is not limited to, the first rule adjustment method; adjusting the initial detection rule according to the first rule adjustment method to obtain the adjusted detection rule; and detecting the data to be tested according to the adjusted detection rule to obtain the updated quality detection result.

[0108] According to the technical solution of this invention, data to be tested is acquired, and the data type of the data to be tested is identified; based on the data type of the data to be tested, an initial detection rule matching the data type is determined from a preset quality detection rule library; the data quality of the data to be tested is detected according to the initial detection rule to obtain a quality detection result of the data to be tested; the quality detection result is input into a pre-trained data analysis model to obtain a data analysis result output by the data analysis model; wherein, the data analysis result includes, but is not limited to, a first rule adjustment method; the initial detection rule is adjusted according to the first rule adjustment method to obtain an adjusted detection rule, and the data to be tested is detected according to the adjusted detection rule to obtain a quality detection update result. That is, the solution of this application analyzes the data using the data quality detection result obtained after using the initial detection rule, and adjusts the initial detection rule according to the data analysis result, thereby avoiding the situation where data quality detection can only be performed using existing rules, improving the accuracy of data quality detection, and reducing the false alarm rate and false negative rate of the quality detection result obtained after detecting data quality according to the detection rule.

[0109] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the data quality detection method provided in any embodiment of this application.

[0110] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data quality inspection method, characterized in that, The method includes: Acquire the data to be detected and identify the data type of the data to be detected; Based on the data type of the data to be detected, an initial detection rule matching the data type is determined from a preset quality detection rule base; The data quality of the data to be tested is detected according to the initial detection rules, and the quality detection result of the data to be tested is obtained. The quality inspection results are input into a pre-trained data analysis model to obtain the data analysis results output by the data analysis model; wherein, the data analysis results include a first rule adjustment method; The abnormal data corresponding to the quality inspection results are identified according to a preset anomaly identification algorithm; The second rule adjustment method is determined based on the data type of the abnormal data; Obtain historical quality inspection results; The historical quality detection results and the quality detection results are input into a preset time series prediction model to obtain the data quality change trend output by the preset time series prediction model. Based on the aforementioned data quality change trend, determine the adjustment method for the third rule; The initial detection rules are adjusted according to the first rule adjustment method, the second rule adjustment method, and the third rule adjustment method to obtain the adjusted detection rules. The data to be detected is then detected according to the adjusted detection rules to obtain the quality detection update result.

2. The method according to claim 1, characterized in that, After inputting the historical quality detection results and the quality detection results into a preset time series prediction model to obtain the data quality change trend output by the preset time series prediction model, the method further includes: A data quality assessment report is generated based on the quality inspection results, the abnormal data, and the data quality change trend.

3. The method according to claim 2, characterized in that, The method further includes: The data quality assessment report is sent to the display module so that the display module can display the data quality assessment report.

4. The method according to claim 1, characterized in that, The number of initial detection rules is multiple. The step of detecting the data quality of the data to be detected according to the initial detection rules to obtain the quality detection result of the data to be detected includes: The data quality of the data to be detected is detected according to multiple initial detection rules, and the data quality detection results of the data to be detected in multiple detection dimensions are obtained; wherein, each initial detection rule corresponds to one detection dimension; The quality detection result of the data to be detected is determined based on the data quality detection results of multiple detection dimensions of the data to be detected.

5. The method according to claim 1, characterized in that, The data analysis model was trained according to the following steps: Obtain training quality detection results and training data analysis results; wherein, the training quality detection results and the training data analysis results are a set of training data in a preset training database, the preset training database includes multiple sets of data to be trained, and each set of data to be trained includes a set of corresponding training quality detection results and training data analysis results; The training quality detection results are input into the model to be trained to obtain the training analysis results output by the model to be trained. Based on the analysis results of the model to be trained and the analysis results of the training data, the target model parameters of the model to be trained are adjusted to obtain the model to be trained after parameter adjustment. The model to be trained with the adjusted parameters is used as the new model to be trained. A training quality detection result from the preset training database that has not been input into the model to be trained is used as the new training quality detection result. The training data analysis result corresponding to the training quality detection result is used as the new training data analysis result. The process of "inputting the training quality detection result into the model to be trained to obtain the training analysis result output by the model to be trained" is repeated until the preset iteration end condition is met. The model to be trained with the adjusted parameters is then used as the data analysis model.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data quality detection method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data quality detection method as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data quality detection method as described in any one of claims 1 to 5.

Images