Data quality detection method and device, equipment and storage medium

By using data quality detection methods in the data warehouse system, monitoring targets are extracted and algorithm libraries are called to detect data quality, timely discovery and processing of data quality problems is solved, data quality monitoring efficiency is improved, and user experience is avoided.

CN119025514AInactive Publication Date: 2024-11-26BEIJING CHANGDONG TECH CO LTD

Patent Information

Application Number
CN202411211537.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When enterprises build a data warehouse system, massive data often encounters data quality problems during extraction, cleaning, conversion and processing. For example, data missing, empty values ​​or non-compliant format, and failure to discover and process in time will affect the accuracy of downstream data and Timeliness affects user experience.

Method used

Provide a data quality detection method, which can extract monitoring targets by responding to the monitoring needs of the user, call the data detection algorithm in the algorithm library, generate monitoring indicator values, and compare the thresholds set by the user to determine whether the data quality meets the preset Requirements: Generate inspection reports to promptly discover and solve data quality problems.

Benefits of technology

It improves the efficiency of data quality monitoring, ensures efficient completion of data quality detection, reduces the repeated development of user logarithmic warehouse data quality monitoring tasks, and avoids the impact of user experience caused by failure to promptly discover data quality problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025514A_ABST
    Figure CN119025514A_ABST
Patent Text Reader

Abstract

The invention provides a data quality detection method, device and equipment and a storage medium, and the method comprises the steps: in response to a data quality monitoring demand sent by a user side, extracting a monitoring target in the data quality monitoring demand, and according to the monitoring target, determining the data quality of the user side; calling a data detection algorithm corresponding to the monitoring target in an algorithm library to detect the data quality monitoring demand, and generating a monitoring index value; comparing the monitoring index value with a threshold value set by the user side, judging whether the currently detected data quality meets a preset requirement or not, and generating a judgment result; and generating a data quality detection report according to the judgment result. The method has the technical effects that the method is used for constructing a data quality monitoring system, and the situation that downstream data is affected and user experience is affected due to the fact that the data quality problem is not found and solved in time is avoided as much as possible.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data quality detection method, device, equipment and storage medium. Background Art

[0002] When enterprises build their own data warehouse systems, as the number of users continues to increase and the demand for business data processing increases, the data warehouse will accumulate massive amounts of data. The sources of the data may be various. During the data extraction, cleaning, conversion and processing, various data quality issues such as missing data, null data values ​​or non-compliant data formats are often encountered. If these data quality issues are not discovered and handled in a timely manner, they will affect the accuracy and timeliness of downstream data processing, thereby affecting the user experience of related data services.

[0003] Therefore, how to build an efficient data warehouse data quality monitoring system has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a data quality detection method, apparatus, device and storage medium for building a data quality monitoring system to avoid the situation where data quality problems are not discovered and resolved in a timely manner, which affects downstream data and thus affects user experience.

[0005] In a first aspect, the present application provides a data quality detection method, the method comprising: in response to a data quality monitoring requirement sent by a user terminal, extracting a monitoring target in the data quality monitoring requirement; According to the monitoring target, calling the data detection algorithm corresponding to the monitoring target in the algorithm library to detect the data quality monitoring requirements and generate a monitoring indicator value; comparing the monitoring indicator value with the threshold set by the user end, judging whether the quality of the currently detected data meets the preset requirements, and generating a judgment result; A data quality detection report is generated based on the judgment result.

[0006] By adopting the above technical solution, the monitoring targets in the data monitoring requirements are extracted according to the data quality monitoring requirements sent by the user end, and then the data detection algorithm corresponding to the monitoring target is called from the algorithm library to detect the data quality monitoring requirements, and the monitoring indicator value is generated. By comparing the monitoring indicator value with the threshold set by the user, it can be determined whether the current detected data quality meets the preset requirements. The above method can be configured according to user needs, and data quality problems can be discovered and solved in a timely manner, and try to avoid the situation where data quality problems are not discovered and solved in a timely manner, which affects downstream data and thus affects the user experience.

[0007] Optionally, the method also includes: obtaining the requirement characteristics of the data quality monitoring requirement, and judging whether there is a first monitoring template corresponding to the requirement characteristics in the template library based on the requirement characteristics; if the first monitoring template exists, directly calling the first monitoring template; obtaining the first monitoring template filled in by the user end, and generating a data quality monitoring task corresponding to the data quality monitoring requirement based on the first monitoring template.

[0008] By adopting the above technical solution, the corresponding monitoring template is called according to the demand characteristics of data quality monitoring needs, and can be configured according to user needs. Users can independently choose the monitoring shift that meets their needs, and then generate data quality monitoring tasks corresponding to data quality monitoring needs, reducing the user's repeated development of data warehouse data quality monitoring tasks, greatly improving the efficiency of data quality monitoring, and trying to avoid the subsequent failure to discover and solve data quality problems in a timely manner, which affects downstream data and thus affects user experience.

[0009] Optionally, according to the monitoring target, calling the data detection algorithm corresponding to the monitoring target in the algorithm library to detect the data quality monitoring requirements, including: calling the data detection algorithm corresponding to the monitoring target, integrating and adapting the data detection algorithm and the big data computing engine to generate an algorithm template; calling the algorithm template to detect the data quality monitoring requirements.

[0010] By adopting the above technical solution, the efficiency of monitoring data quality is greatly improved. Data quality detection calculation based on the big data computing engine improves the efficiency of quality detection of massive data, so that data quality detection can be completed efficiently.

[0011] Optionally, the judgment result includes qualified and unqualified, and the comparison of the monitoring indicator value and the threshold set by the user terminal is used to determine whether the quality of the currently detected data meets the preset requirements, and a judgment result is generated, including: if the monitoring indicator value is not equal to the threshold set by the user terminal, it is determined that the quality of the currently detected data does not meet the preset requirements, and the judgment result is unqualified; otherwise, the judgment result is qualified.

[0012] By adopting the above technical solution, it is possible to determine whether the quality of the currently detected data meets the preset requirements and generate a judgment result. Based on the judgment result, it can be determined whether there is a problem with the current data quality, thereby avoiding the situation where data quality problems are not discovered and resolved in a timely manner, which affects downstream data and thus affects the user experience.

[0013] Optionally, comparing the monitoring indicator value with the threshold set by the user terminal to determine whether the quality of the currently detected data meets the preset requirements includes: calling the data quality monitoring template corresponding to the data detection algorithm; obtaining the threshold set by the user terminal on the data quality monitoring template; comparing the monitoring indicator value with the threshold set by the user terminal on the data quality monitoring template, so as to determine whether the quality of the currently detected data meets the preset requirements.

[0014] By adopting the above technical solution, by comparing the monitoring indicator value with the threshold set by the user end on the data quality monitoring template, it can be determined whether the quality of the currently detected data meets the preset requirements.

[0015] Optionally, based on the monitoring target, calling the data detection algorithm corresponding to the monitoring target in the algorithm library to detect the data quality monitoring requirements, including: responding to the trigger condition set by the user end, when the trigger condition is detected, detecting the data quality monitoring requirements through the data detection algorithm.

[0016] By adopting the above technical solution, when a trigger condition is detected, the numerical quality monitoring requirements are directly detected through the data detection algorithm. Users can automatically generate executable tasks based on big data through corresponding configuration, thereby greatly reducing the subsequent data quality monitoring development work and making data quality monitoring changes easier and more efficient.

[0017] Optionally, after generating the data quality detection report according to the judgment result, it also includes: storing the data quality monitoring report, and sending the data quality monitoring report to data business personnel, so that the business personnel perform data processing according to the data quality monitoring report.

[0018] By adopting the above technical solution, after generating a data quality detection report based on the judgment results, the data quality report is stored and sent to business personnel so that business personnel can process data according to the quality monitoring report, and try to avoid failure to discover and solve data quality problems in time, which will affect downstream data and thus affect user experience.

[0019] In the second aspect, the present application provides a data quality detection device, which includes: a response module, a generation module, a comparison module and an output module; wherein the response module is used to respond to the data quality monitoring requirements sent by the user terminal, and extract the monitoring targets in the data quality monitoring requirements; the generation module is used to call the data detection algorithm corresponding to the monitoring target in the algorithm library according to the monitoring target to detect the data quality monitoring requirements and generate a monitoring indicator value; the comparison module is used to compare the monitoring indicator value with the threshold set by the user terminal, determine whether the data quality currently detected meets the preset requirements, and generate a judgment result; the output module is used to generate a data quality detection report based on the judgment result.

[0020] By adopting the above technical solution, the monitoring targets in the data monitoring requirements are extracted according to the data quality monitoring requirements sent by the user end, and then the data detection algorithm corresponding to the monitoring target is called from the algorithm library to detect the data quality monitoring requirements, and the monitoring indicator value is generated. By comparing the monitoring indicator value with the threshold set by the user, it can be determined whether the current detected data quality meets the preset requirements. The above method can be configured according to user needs, and data quality problems can be discovered and solved in a timely manner, and try to avoid the situation where data quality problems are not discovered and solved in a timely manner, which affects downstream data and thus affects the user experience.

[0021] In a third aspect, the present application provides an electronic device, which adopts the following technical solution: it includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes a computer program such as any one of the above-mentioned data quality detection methods.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned data quality detection methods.

[0023] In summary, the present application includes at least one of the following beneficial technical effects: 1. It can be configured according to user needs, and data quality problems can be discovered and solved in a timely manner, so as to avoid the situation where data quality problems are not discovered and solved in a timely manner, which may affect downstream data and thus affect user experience; 2. Greatly improve the efficiency of data quality monitoring. Data quality detection calculation based on the big data computing engine improves the efficiency of quality detection of massive data, so that data quality detection can be completed efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1It is a flow chart of a data quality detection method provided in an embodiment of the present application; Figure 2 It is a structural schematic diagram of a data quality detection device provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0025] Description of reference numerals: 1000, electronic device; 1001, processor; 1002, communication bus; 1003, user interface; 1004, network interface; 1005, memory. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of the present application, words such as "illustrative", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "illustrative", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "illustrative", "for example" or "for example" is intended to present related concepts in a concrete way.

[0028] In the era of big data, more and more online business data of enterprises rely on the data processing results of big data warehouses. Data quality determines the quality of online data-related services of enterprises. Therefore, data quality in data warehouses needs to be monitored and problems discovered in a timely manner. Quality issues such as timeliness, accuracy, completeness, and consistency of data in data warehouses need to be efficiently solved.

[0029] This application can be understood as analyzing and summarizing the data quality monitoring requirements proposed by users, extracting data detection algorithms or rules that achieve data quality monitoring goals, detecting the current data to be detected according to the data detection algorithms or rules, and obtaining detection results, and then comparing the detection results with the preset requirements set by the user to determine whether the current data to be detected meets the requirements, and generating and storing a real-time data quality monitoring report based on the results of the data to be detected; sending the report to the person in charge related to the data business in real time, and the person in charge of the data business processes the business data according to the content of the data quality report.

[0030] Figure 1 is a flow chart of a data quality detection method provided in an embodiment of the present application. It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0031] This application discloses a data quality detection method, such as Figure 1 As shown, the method includes S101-S104.

[0032] S101, in response to a data quality monitoring requirement sent by a user terminal, extracting a monitoring target in the data quality monitoring requirement.

[0033] In one example, for enterprises, as data continues to increase, data will continue to accumulate. During the extraction, cleaning, conversion and processing of a large amount of data, various data quality issues such as missing data, null data values ​​or non-compliant data formats will often be encountered. If these data quality issues are not discovered and handled in a timely manner, they will affect the accuracy and timeliness of downstream data processing, and even affect the user experience of related data services. Therefore, how to build an efficient data warehouse data quality monitoring system has become an urgent problem to be solved.

[0034] Responding to the data quality monitoring requirements sent by the user end can be understood as the user can choose to perform relevant operations on the data according to personal needs. For example, to monitor whether there is a null value, the user sends a message to the computer device to monitor whether there is a null value in any field. The computer device receives the monitoring request sent by the user and extracts the monitoring target in the monitoring request sent by the user; generally speaking, the monitoring target can be obtained based on the data quality monitoring request sent by the user. The monitoring target here is a null value. When the user sends a data quality monitoring request, the computer device will automatically extract the monitoring target in the request.

[0035] It should be noted that when the computer device extracts the data quality monitoring requirements sent by the user, if there is a situation where the extraction cannot be carried out, the user needs to consider whether the sent data quality monitoring requirements do not meet the preset requirements, or the computer device does not store the data quality monitoring requirements sent by the corresponding user. If the data quality monitoring requirements sent by the corresponding user are not stored, at this time, an application request can be sent to the developer or business personnel so that the developer or business personnel can add it.

[0036] S102, according to the monitoring target, calling the data detection algorithm corresponding to the monitoring target in the algorithm library to detect the data quality monitoring requirements and generate the monitoring indicator value.

[0037] In one example, when the monitoring target is obtained, it is necessary to call the data detection algorithm corresponding to the monitoring target in the algorithm library. Here, the monitoring target is a null value for illustration. The algorithm library stores a large number of data detection algorithms. When the data quality monitoring requirements sent by the user are detected, the corresponding data detection algorithm is called from the algorithm library for detection. The monitoring target here is a null value, so it is necessary to call the data detection algorithm for detecting null values ​​from the algorithm library for detection. After calling the data detection algorithm corresponding to the data quality monitoring requirements currently sent by the user, the relevant data is directly calculated to obtain the monitoring indicator value. The monitoring indicator value here can be understood as the detection result. If the detected target is a null value, then the monitoring indicator value is the null value rate.

[0038] Based on the above embodiment, as an optional implementation, S102, according to the monitoring target, calling the data detection algorithm corresponding to the monitoring target in the algorithm library to detect the data quality monitoring requirements specifically includes: Call the data detection algorithm corresponding to the monitoring target, integrate and adapt the data detection algorithm and the big data computing engine, and generate an algorithm template; call the algorithm template to detect data quality monitoring requirements.

[0039] In one example, the system first calls the corresponding data detection algorithm from the algorithm library according to the monitoring target provided by the user. These algorithms may include various types such as null value detection, outlier identification, consistency check, etc. However, simply calling these algorithms is not enough, because they may not be able to run efficiently directly in a large-scale data environment. Therefore, the system will integrate and adapt these algorithms with big data computing engines (such as Hadoop, Spark, etc.). This process involves converting the algorithm logic into a distributed computing model to fully utilize the parallel processing capabilities of the big data platform.

[0040] During the integration and adaptation process, the system will consider multiple factors. First, it is necessary to ensure that the algorithm can be correctly decomposed into tasks that can be executed in parallel to make full use of distributed computing resources. Secondly, it is necessary to optimize the storage and access methods of data to reduce network transmission and disk I / O overhead. Thirdly, it is also necessary to consider how to handle intermediate results and how to summarize and integrate the results. Through these optimizations, data quality detection tasks that may have taken hours or even days to complete may get results in minutes.

[0041] After integration and adaptation, the system will generate an algorithm template. This template not only contains the logic of the original algorithm, but also contains the interface and optimization strategy for interacting with the big data computing engine. Such algorithm templates are highly reusable and can be quickly called and parameters adjusted for different data sets and monitoring requirements.

[0042] When a specific data quality monitoring task needs to be performed, the system will call this algorithm template and set the corresponding parameters according to the actual monitoring needs. For example, for a task to detect outliers, the system may set parameters such as data source, target field, and anomaly determination threshold. Then, the system will start the big data computing engine and distribute the task to multiple nodes in the cluster for simultaneous execution.

[0043] After calling the data detection algorithm of the corresponding monitoring target, the data detection algorithm and the data computing engine need to be integrated and adapted. When using the data detection algorithm to detect the data quality monitoring requirements sent by the user, it is necessary to execute according to the user's needs. For example, the user needs to detect the null value rate of the corresponding data at a fixed time every day; after obtaining the data detection algorithm, the algorithm needs to be converted. For the spark big data computing engine, an algorithm library is built, and the algorithm library is converted into a class library code or sql template that can be run in spark, so that users can call it through configuration.

[0044] Spark big data computing engine is an open source big data computing engine, which provides efficient data processing and analysis functions. Spark is a high-performance, multi-functional, and fault-tolerant big data computing engine. It has advantages in processing large-scale data, can support various data processing and analysis tasks, and is one of the commonly used tools in the field of big data.

[0045] On the basis of the above embodiments, as an optional implementation, in S102, according to the monitoring target, the data detection algorithm corresponding to the monitoring target is called in the algorithm library to detect the data quality monitoring requirements, including: responding to the trigger condition set by the user end, when the trigger condition is detected, the data quality monitoring requirements are detected by the data detection algorithm.

[0046] In one example, the system first allows users to define trigger conditions when setting data quality monitoring requirements. These trigger conditions can be varied, such as time triggers (such as 2 a.m. every day), data volume triggers (such as when the amount of new data reaches 1 million), event triggers (such as after an important business process is completed), etc. Users can flexibly set these conditions based on their own business needs and data characteristics. For example, for transaction data with high real-time requirements, more frequent timed triggers may be set; while for basic data that changes slowly, you may choose to trigger detection after the data is updated.

[0047] The system will continue to monitor these trigger conditions. When a trigger condition is detected, the system will respond immediately and start the corresponding data quality monitoring process. This process includes calling the data detection algorithm corresponding to the monitoring target from the algorithm library, and then applying these algorithms to perform quality detection on the specified data set.

[0048] This trigger-based approach has several significant advantages. First, it improves the timeliness and pertinence of monitoring. By setting appropriate trigger conditions, the system can perform data quality checks at the most appropriate time, thereby quickly discovering and responding to data quality issues. For example, on an e-commerce platform, you can set up a quality check on order data immediately after each major promotion to ensure the accuracy and integrity of the large amount of data generated during business peak periods.

[0049] S103, comparing the monitoring indicator value with the threshold set by the user end, determining whether the quality of the currently detected data meets the preset requirements, and generating a determination result.

[0050] In one example, after obtaining the value of the monitoring indicator, it is necessary to compare the value of the monitoring indicator with the threshold set by the user, because different users have different quality requirements for the same data.

[0051] For example: User A needs to monitor whether there is a null value in the x field of table a. If it exists, a null value is issued and a report is sent. User B needs to monitor whether there is a null value in the y field of table b. If it exists and the total number of null values ​​is greater than 10, a report is sent. The common feature of users A and B is to detect whether there is a null value problem in the data in the table. This type of demand can be summarized as field null value detection. Similar ones include field abnormal value detection, field duplicate value detection, field length detection, etc. The corresponding detection algorithm is adapted to these detection types to generate an algorithm template library; for the monitoring indicators of users A and B, both are null values, and the judgment rules are similar. They are all judgments on the total number of null values. One condition is greater than 0, and the other condition is greater than 10. The final generated detection results are also similar. At that time, the final triggered thresholds are different. For A, null values ​​may not be allowed. If null values ​​exist, it is considered that the data of the current table to be inspected does not meet the preset requirements, and a report is sent; for B, only when the total number of null values ​​is greater than 10, the data of the current table to be inspected is considered to be inconsistent with the threshold. Therefore, when the same requirements are raised by users, although the data detection algorithm called is the same, the thresholds set by the users themselves are different. Different thresholds will eventually generate different results.

[0052] In one example, the system first obtains the value of the monitoring indicator calculated by the data detection algorithm. This value reflects a specific quality feature of the current data being detected. At the same time, the system also reads the threshold value pre-set by the user. This threshold value represents the user's expected standard for data quality. For example, when detecting data integrity, the user may set a maximum allowable missing rate threshold. Next, the system compares the monitoring indicator value with the threshold value set by the user. A strict judgment standard is adopted here: only when the monitoring indicator value is strictly equal to the threshold value set by the user, the data quality is judged to meet the preset requirements, and the judgment result is "qualified". On the contrary, if there is any deviation between the monitoring indicator value and the threshold, whether it is higher or lower than the threshold, the system will judge that the quality of the currently detected data does not meet the preset requirements, and the judgment result is "unqualified".

[0053] When calling the corresponding data detection algorithm to perform data quality monitoring needs, since different users have different expectations for the results, when obtaining the monitoring indicator value, it is necessary to compare the monitoring indicator value with the threshold set by the user. If the monitoring indicator data meets the threshold set by the user, the current detection data quality is judged to be qualified, otherwise, the result is judged to be unqualified.

[0054] Based on the above embodiment, as an optional implementation, in S103, comparing the monitoring indicator value with the threshold value set by the user end to determine whether the quality of the currently detected data meets the preset requirements specifically includes: Call the data quality monitoring template corresponding to the data detection algorithm; obtain the threshold set by the user end on the data quality monitoring template; compare the monitoring indicator value with the threshold set by the user end on the data quality monitoring template to determine whether the quality of the currently detected data meets the preset requirements.

[0055] In one example, when calling a data detection algorithm to execute data quality monitoring requirements, it is necessary to obtain the threshold set by the user. The threshold is used to determine whether the result of the data detection algorithm executing the data quality monitoring requirements meets the requirements. Generally speaking, a data detection algorithm corresponds to a data quality monitoring template, and the data quality monitoring template includes the threshold filled in by the user, the indicator judgment rule, the data quality report template, etc. Before executing the data quality monitoring requirements sent by the user, the user needs to set the threshold on the quality monitoring template, and then compare the threshold and the monitoring indicator value to determine whether the currently detected data meets the requirements. The indicator judgment rule here refers to the comparison method of the threshold and the monitoring indicator value, and finally the result is displayed through the data quality report template. The data quality report template is the content of the data quality report sent to the user, including text content and drawn monitoring icon content.

[0056] The system will first call the data quality monitoring template corresponding to the current data detection algorithm. These templates are pre-designed and contain various parameter settings, threshold ranges, and judgment logic required for specific types of data quality monitoring. For example, for a task to detect outliers in numerical data, the corresponding template may contain information such as the definition method of outliers (such as the standard deviation multiple method, the quartile method, etc.), the default threshold range, and how to handle boundary conditions.

[0057] After calling the template, the system will obtain the specific thresholds set by the user on the template. The threshold here may be a single value, or a range or a combination of multiple conditions, depending on the specific monitoring needs and data characteristics. For example, in a template for detecting abnormal order amounts, the user may set the minimum and maximum allowed values, as well as the allowed standard deviation multiples. This approach allows users to flexibly adjust monitoring standards based on their own business needs and data characteristics, without having to set all parameters from scratch.

[0058] After obtaining the thresholds set by the user, the system will compare the monitoring indicator values ​​with these thresholds. This comparison process may involve multiple steps and conditional judgments, depending on the judgment logic defined in the template. For example, for outlier detection, the system may first check whether the value is within the allowable range, and then determine whether it exceeds the multiple limit of the standard deviation. Through this multi-dimensional comparison, the system can more comprehensively and accurately determine whether the data quality meets the preset requirements.

[0059] S104: Generate a data quality detection report based on the judgment result.

[0060] In an example, user A needs to monitor the null value rate of field x in table a every day. If the null value rate is greater than 5% in the past week, a data quality detection report is generated. The monitoring indicator is the null value rate of field x in table a, and the threshold is 5%. Through detection, it is found that the total number of the table is 50 million on a certain day, the number of null values ​​in field x is 500,000, and the null value rate is 10%, which triggers the sending of a data quality report. For the data quality monitoring report content, the calculated monitoring data will be filled into the text content, and a monitoring chart of the null value rate of field x in table a for a week will be drawn.

[0061] This application also provides another feasible example, the method includes S201-S203: S201, obtaining demand characteristics of data quality monitoring requirements, and judging whether there is a first monitoring template corresponding to the demand characteristics in the template library according to the demand characteristics.

[0062] In one example, the system will first parse the data quality monitoring requirements submitted by the user and extract key features. These features may include the monitored data type (such as numeric type, text type), monitoring target (such as null value detection, outlier detection), monitoring scope (such as specific tables, fields), etc. For example, when a user submits a requirement such as "detect outliers in the product price field in the sales data table", the system will extract key features such as "sales data", "product price", and "outlier detection".

[0063] After obtaining these demand features, the system will match them in the pre-established template library. The template library stores standard templates for various common data quality monitoring scenarios, and each template has its corresponding feature label. The system will search for templates with similar features in the template library based on the extracted demand features. If a template with a high degree of match is found, the system will mark it as the first monitoring template.

[0064] S202: If the first monitoring template exists, directly call the first monitoring template.

[0065] S203, obtaining the first monitoring template filled by the user end, and generating a data quality monitoring task corresponding to the data quality monitoring requirement according to the first monitoring template.

[0066] In one example, when any user sends a data quality monitoring requirement, the computer device will generate an execution task to execute the data quality monitoring requirement sent by the user, and then store the task request so that when the same data quality monitoring requirement appears in the future, the corresponding execution task can be directly called to execute the data quality monitoring requirement sent by the user. After calling the corresponding execution task, the user only needs to set the threshold by himself or directly adopt the threshold; if there is no execution task that can be called, then consider generating the corresponding algorithm for execution.

[0067] When calling the same execution task, it is necessary to obtain the demand characteristics of the current data quality monitoring requirements. The demand characteristics here can be understood as the monitoring targets in the data quality monitoring requirements sent by the user. According to the monitoring targets, it can be determined whether there is a corresponding data detection algorithm in the current database, such as detecting null values. When there is an execution task for detecting null values ​​in the database, it can be called directly, and then the user can selectively set the threshold. The first monitoring template here is the execution task.

[0068] Based on the above embodiment, as an optional implementation, in S103, the judgment result includes qualified and unqualified, and the monitoring indicator value is compared with the threshold set by the user end to judge whether the quality of the currently detected data meets the preset requirements, and generate a judgment result, including: If the value of the monitoring indicator is not equal to the threshold set by the user, it is determined that the quality of the currently detected data does not meet the preset requirements and the result is judged to be unqualified; otherwise, the result is judged to be qualified.

[0069] After generating the data quality detection report according to the judgment result, the method further includes: storing the data quality monitoring report and sending the data quality monitoring report to data business personnel so that the business personnel can process the data according to the data quality monitoring report.

[0070] In a feasible example, after generating a data quality inspection report, the data quality inspection report can be stored for subsequent reference. At the same time, if the current data quality monitoring report does not meet the preset requirements, business personnel can also process relevant business data according to the data quality inspection report to avoid data quality problems not being discovered and handled in a timely manner, affecting the accuracy and timeliness of downstream data processing, thereby affecting the user's experience of related data services.

[0071] Based on the above method, the present application also discloses a data quality detection device, such as Figure 2 As shown, Figure 2 It is a structural schematic diagram of a data quality detection device provided in an embodiment of the present application.

[0072] A data quality monitoring device comprises: a response module, a generation module, a comparison module and an output module; wherein the response module is used to respond to the data quality monitoring requirements sent by the user end and extract the monitoring targets in the data quality monitoring requirements; the generation module is used to call the data detection algorithm corresponding to the monitoring target in the algorithm library according to the monitoring target to detect the data quality monitoring requirements and generate the monitoring index value; the comparison module is used to compare the monitoring index value with the threshold value set by the user end, judge whether the data quality currently detected meets the preset requirements, and generate the judgment result; the output module is used to generate the data quality detection report according to the judgment result.

[0073] In one example, the above-mentioned device is also used to obtain the demand characteristics of data quality monitoring requirements, and based on the demand characteristics, determine whether there is a first monitoring template corresponding to the demand characteristics in the template library; if the first monitoring template exists, directly call the first monitoring template; obtain the first monitoring template filled in by the user end, and generate a data quality monitoring task corresponding to the data quality monitoring requirements based on the first monitoring template.

[0074] In one example, the generation module is also used to call the data detection algorithm corresponding to the monitoring target, integrate and adapt the data detection algorithm and the big data computing engine, generate an algorithm template; and call the algorithm template to detect data quality monitoring requirements.

[0075] In one example, the comparison module is also used to determine that the quality of the currently detected data does not meet the preset requirements and the result is unqualified if the monitoring indicator value is not equal to the threshold set by the user end; otherwise, the result is qualified.

[0076] In one example, the comparison module is also used to call the data quality monitoring template corresponding to the data detection algorithm; obtain the threshold set by the user end on the data quality monitoring template; compare the monitoring indicator value with the threshold set by the user end on the data quality monitoring template, so as to determine whether the quality of the currently detected data meets the preset requirements.

[0077] In one example, the generation module is further configured to respond to a trigger condition set by a user terminal, and when the trigger condition is detected, detect the data quality monitoring requirement through a data detection algorithm.

[0078] In one example, the above device is also used to store a data quality monitoring report, and send the data quality monitoring report to a data business person, so that the business person performs data processing according to the data quality monitoring report.

[0079] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0080] See also Figure 3 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0081] The communication bus 1002 is used to realize the connection and communication between these components.

[0082] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0083] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0084] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0085] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Figure 3 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a data quality detection method.

[0086] exist Figure 3In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call an application program storing a data quality detection method in the memory 1005. When executed by one or more processors, the electronic device executes one or more methods described in the above embodiments.

[0087] An electronic device readable storage medium stores instructions, which, when executed by one or more processors, enable the electronic device to execute one or more of the methods described in the above embodiments.

[0088] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0094] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A data quality detection method, characterized in that: The method comprises: In response to the data quality monitoring requirement sent by the user end, extracting the monitoring target in the data quality monitoring requirement; According to the monitoring target, calling the data detection algorithm corresponding to the monitoring target in the algorithm library to detect the data quality monitoring requirements and generate a monitoring indicator value; Compare the monitoring indicator value with the threshold value set by the user end, determine whether the quality of the currently detected data meets the preset requirements, and generate a judgment result; A data quality detection report is generated based on the judgment result.

2. The data quality detection method according to claim 1, characterized in that: The method further comprises: Obtaining the demand characteristics of the data quality monitoring demand, and judging whether there is a first monitoring template corresponding to the demand characteristics in the template library according to the demand characteristics; If the first monitoring template exists, directly call the first monitoring template; The first monitoring template filled by the user end is obtained, and a data quality monitoring task corresponding to the data quality monitoring requirement is generated according to the first monitoring template.

3. The data quality detection method according to claim 1, characterized in that: The step of calling a data detection algorithm corresponding to the monitoring target in an algorithm library to detect the data quality monitoring requirement according to the monitoring target includes: Calling a data detection algorithm corresponding to the monitoring target, integrating and adapting the data detection algorithm and the big data computing engine, and generating an algorithm template; The algorithm template is called to detect the data quality monitoring requirement.

4. The data quality detection method according to claim 1, characterized in that: The judgment result includes qualified and unqualified, and the comparison of the monitoring indicator value and the threshold value set by the user terminal to judge whether the quality of the currently detected data meets the preset requirements and generate the judgment result includes: If the monitoring indicator value is not equal to the threshold value set by the user end, it is determined that the quality of the currently detected data does not meet the preset requirements, and the judgment result is unqualified; otherwise, the judgment result is qualified.

5. The data quality detection method according to claim 1, characterized in that: The comparing the monitoring indicator value with the threshold value set by the user terminal to determine whether the quality of the currently detected data meets the preset requirements includes: Calling a data quality monitoring template corresponding to the data detection algorithm; Obtaining a threshold value set by the user terminal on the data quality monitoring template; The monitoring indicator value is compared with the threshold value set by the user terminal on the data quality monitoring template to determine whether the quality of the currently detected data meets the preset requirements.

6. The data quality detection method according to claim 1, characterized in that: The step of calling a data detection algorithm corresponding to the monitoring target in an algorithm library to detect the data quality monitoring requirement according to the monitoring target includes: In response to the trigger condition set by the user end, when the trigger condition is detected, the data quality monitoring requirement is detected by the data detection algorithm.

7. The data quality detection method according to claim 1, characterized in that: After generating the data quality detection report according to the judgment result, the method further includes: The data quality monitoring report is stored, and the data quality monitoring report is sent to a data service personnel, so that the service personnel performs data processing according to the data quality monitoring report.

8. A data quality detection device, characterized in that: The device comprises: a response module, a generation module, a comparison module and an output module; wherein, The response module is used to respond to the data quality monitoring requirement sent by the user end and extract the monitoring target in the data quality monitoring requirement; The generating module is used to call the data detection algorithm corresponding to the monitoring target in the algorithm library according to the monitoring target to detect the data quality monitoring requirements and generate the monitoring indicator value; The comparison module is used to compare the monitoring indicator value with the threshold value set by the user terminal, determine whether the quality of the currently detected data meets the preset requirements, and generate a judgment result; The output module is used to generate a data quality detection report according to the judgment result.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Data monitoring method, equipment and system

    CN106371984A

  • Data quality checking method and device, electronic equipment and storage medium

    CN112667612A

  • Data quality monitoring method and system and computer equipment

    CN113312341A

  • Data quality monitoring method and device and related equipment

    CN113342939A

  • Data quality monitoring method and device

    CN114328441A

Cited By

  • Multi-modal joint detection method and system for data quality defects

    CN120832621A

  • A multi-modal joint detection method and system for data quality defects

    CN120832621B