Data quality detection method and device based on service scene and medium

Abstract

The embodiment of the invention discloses a data quality detection method and device based on a service scene and a medium, and relates to the technical field of data quality detection.The method comprises the steps that current service scene information corresponding to a current data quality detection requirement is determined, and a data quality detection strategy is determined based on the service scene information; according to the data quality detection strategy, quality detection task configuration is carried out to determine a corresponding data quality detection task set which comprises a plurality of data quality detection sub-tasks; submitting the data quality detection task set to a Griffin task scheduling system, performing task distribution on a plurality of data quality detection sub-tasks through the Griffin task scheduling system, and determining a working node corresponding to each data quality detection sub-task; and executing the data quality detection subtask through each working node, generating a quality detection result corresponding to each working node, integrating the quality detection results according to the current service scene information, and determining a data quality report.

Description

Technical Field

[0001] The present invention relates to the technical field of data quality detection, and in particular to a data quality detection method, device and medium based on a business scenario. Background Art

[0002] Data quality refers to the degree to which data has characteristics such as applicability, accuracy, completeness, consistency, timeliness and reliability in meeting business needs and decision support. It covers the entire process from data collection, processing, storage to application, ensuring that data can reflect real-world conditions and provide effective information support in various scenarios. With the substantial increase in the number, quality and frequency of digital services in various fields, such as social networks and e-commerce, enterprises need to collect and process large amounts of data to provide personalized services, optimize product experience and enhance market competitiveness. When the amount of data processed reaches a certain level (1 billion or 100GB) and the complexity of the system increases (the number of reports continues to increase), traditional data quality detection of large amounts of data usually uses Hive SQL to execute MapReduce tasks. The startup and operation of MapReduce jobs take a long time, resulting in low efficiency in discovering and processing data quality issues; secondly, due to the limited flexibility of SQL queries, complex data quality rules are difficult to express and implement effectively.

[0003] The traditional Griffin framework cannot cover the complex data quality detection requirements in all industries or specific business scenarios. When users face some special or highly customized data quality inspection tasks, they need to spend extra time and energy to customize quality rules by designing and writing corresponding Scala code, which raises the technical threshold of quality detection. Therefore, the current quality detection method is relatively separated from the business scenario, requiring users to customize quality rules, which reduces detection efficiency. Summary of the invention

[0004] One or more embodiments of this specification provide a data quality detection method, device and medium based on business scenarios, which are used to solve the following technical problems: the current quality detection method is relatively separated from the business scenario, requires users to customize quality rules, and reduces detection efficiency.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of the present specification provide a data quality detection method based on business scenarios, the method comprising: determining current business scenario information corresponding to current data quality detection requirements, and determining a data quality detection strategy based on the business scenario information, wherein the data quality detection strategy comprises data source connection request information of at least one data source corresponding to the data to be detected and at least one data quality rule; performing quality detection task configuration through the data quality detection strategy to determine a corresponding data quality detection task set, wherein the data quality detection task set comprises a plurality of data quality detection subtasks; submitting the data quality detection task set to a Griffin task scheduling system, and performing task distribution on the plurality of data quality detection subtasks through the Griffin task scheduling system to determine a working node corresponding to each of the data quality detection subtasks; executing the data quality detection subtask through each of the working nodes to generate a quality detection result corresponding to each of the working nodes, and integrating the quality detection results according to the current business scenario information to determine a data quality report.

[0007] Furthermore, based on the business scenario information, a data quality detection strategy is determined, specifically including: obtaining business data usage information, business data source information and at least one preset key data item of multiple business data in the business scenario information; determining the data source type of the data source corresponding to each business data through the business data source information, so as to determine the data source connection request information of each data source based on the data source type; determining at least one data quality detection rule corresponding to each business data according to the business data usage information corresponding to each business data, wherein the data quality detection rule includes any one or more of an integrity measurement, a consistency measurement, an accuracy measurement and a validity measurement; setting a quality detection priority through the at least one key data item, and determining the detection priority information, so as to determine the data quality detection strategy based on the data source connection request information, the data quality detection rule and the detection priority information.

[0008] Furthermore, quality detection task configuration is performed through the data quality detection strategy to determine the corresponding data quality detection task set, specifically including: configuring task parameters according to the data quality detection strategy to create a corresponding data quality detection task set; parsing the data quality detection task set through the Griffin task scheduling system to decompose it into multiple data quality detection subtasks, wherein the multiple data quality detection subtasks are independently executable detection tasks, and each of the data quality detection subtasks includes data source connection request information, data quality rule ID and subtask priority information.

[0009] Further, through the Griffin task scheduling system, the multiple data quality detection subtasks are distributed, and the working node corresponding to each of the data quality detection subtasks is determined, which specifically includes: through the Griffin task scheduling system, the current resource usage information corresponding to each working node in the Spark big data computing engine is dynamically captured, wherein the current resource usage information includes current computing resource data, current storage resource data and current network bandwidth data; task feature analysis is performed on each of the data quality detection subtasks to determine the resource requirement information corresponding to each of the data quality detection subtasks and the task dependency relationship between the multiple data quality detection subtasks; according to the task dependency relationship between the multiple data quality detection subtasks and the subtask priority information of each of the data quality detection subtasks, a resource allocation order corresponding to the multiple data quality detection subtasks is generated; according to the resource allocation order, according to the resource requirement information corresponding to each of the data quality detection subtasks and the current resource usage of each of the working nodes, each of the data quality detection subtasks is sequentially allocated nodes to determine the working node corresponding to each of the data quality detection subtasks.

[0010] Furthermore, the data quality detection subtask is executed through each of the working nodes to generate a quality detection result corresponding to each of the working nodes, specifically including: determining the data source connection request information, data quality rule ID and subtask priority information corresponding to each of the data quality detection subtasks; through a preset data reading intermediate layer, according to the data source connection request information, the data source is connected to obtain the data to be detected, wherein the data source includes a relational data source and a big data storage data source; according to each of the working nodes, according to the data quality rule ID, the corresponding quality detection algorithm is matched to perform quality detection on the data to be detected, and a quality detection result corresponding to each of the working nodes is generated.

[0011] Furthermore, after executing the data quality detection subtask through each of the working nodes and generating the quality detection results corresponding to each of the working nodes, the method also includes: monitoring the subtask execution status of each of the working nodes through a preset real-time monitoring interface, and collecting the real-time task execution data of each of the working nodes, wherein the real-time task specified data includes real-time node resource usage information and node task execution information; determining the real-time node task execution status corresponding to each of the working nodes according to the real-time node resource usage information and the node task execution information of each of the working nodes; and adjusting the task distribution based on the real-time node task execution status corresponding to each of the working nodes.

[0012] Furthermore, according to the current business scenario information, the quality detection results are integrated to determine a data quality report, which specifically includes: collecting the quality detection results of each of the working nodes to summarize the data quality based on multiple quality detection results, determine the summary result data, and determine the abnormal type of the abnormal data items in the summary result data; determine the corresponding quality result display type through the current business scenario information and the abnormal type; render the summary result data according to the quality result display type, and construct a data quality problem view to determine the data quality report.

[0013] Furthermore, determining the current business scenario information corresponding to the current data quality detection requirement specifically includes: under the triggering of the user's quality detection operation, sending a business scenario form corresponding to the quality detection requirement to the user end, wherein the business scenario form includes business data type items, business data usage items, business data source items and custom key data items; according to the business scenario form, determining the current business scenario information corresponding to the current data quality detection requirement, wherein the current business scenario information includes business data usage information of multiple business data, business data source information and at least one preset key data item.

[0014] One or more embodiments of this specification provide a data quality detection device based on a business scenario, including:

[0015] at least one processor; and,

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0018] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0019] At least one of the above-mentioned technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the above-mentioned technical solutions, by clarifying the current business scenario information, the data scope and data quality rules that need to be detected can be determined more accurately, thereby avoiding the waste of resources caused by indiscriminate comprehensive detection. The data quality rules customized for specific business scenarios can effectively identify and solve common data quality problems in the scenario, and improve the accuracy and reliability of the data; through the Griffin task scheduling system, the detection tasks are efficiently distributed and executed, and the quality detection results can be quickly generated; based on the data quality detection strategy customized for the business scenario, the detection resources can be allocated more reasonably to avoid unnecessary waste; for data quality tasks that require real-time detection, Griffin can quickly respond and distribute tasks to the corresponding work nodes to achieve near-real-time data quality monitoring; through the data quality rules defined by Griffin, the accuracy, completeness, timeliness, uniqueness, effectiveness, consistency and other dimensions of the data can be accurately measured to ensure comprehensive monitoring of data quality and improve the efficiency of data quality detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0021] Figure 1 A flowchart of a data quality detection method based on a business scenario provided in an embodiment of this specification;

[0022] Figure 2 A schematic diagram of the structure of a data quality detection device based on a business scenario provided in an embodiment of this specification. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0024] The embodiments of this specification provide a data quality detection method based on a business scenario. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 A flow chart of a data quality detection method based on a business scenario provided in an embodiment of this specification, such as Figure 1 As shown, it mainly includes the following steps:

[0025] Step S101, determining current business scenario information corresponding to current data quality detection requirements, and determining a data quality detection strategy based on the business scenario information.

[0026] The data quality detection strategy includes data source connection request information of at least one data source corresponding to the data to be detected and at least one data quality rule.

[0027] Determining the current business scenario information corresponding to the current data quality detection requirement specifically includes: under the triggering of the user's quality detection operation, sending a business scenario form corresponding to the quality detection requirement to the user end, wherein the business scenario form includes a business data type item, a business data usage item, a business data source item and a custom key data item; according to the business scenario form, determining the current business scenario information corresponding to the current data quality detection requirement, wherein the current business scenario information includes business data usage information of multiple business data, business data source information and at least one preset key data item.

[0028] In one embodiment of the present specification, when a user requests a quality inspection, the system first sends a business scenario form to the user terminal, and the form content generally includes but is not limited to business data type items, business data usage items, business data source items and custom key data items. Business data type items refer to the data types selected or input by the user to be quality inspected in the current business scenario, such as structured data (such as database tables), unstructured data (such as text files, images), etc. Business data usage items are used to identify the use of business data, such as financial transaction monitoring, user behavior analysis purposes, etc. Business data source items facilitate users to enter the data source of the business data to be inspected, such as internal systems, external suppliers or other methods. Custom key data items are used to specify which data items are important for the business scenario. The user fills in the form according to the actual business situation and submits it to the system. Before submission, the system can provide some verification mechanisms to ensure that the information entered by the user is accurate and complete. After the system receives the form submitted by the user, it parses the form content to determine the current business scenario information corresponding to the current data quality inspection requirements, including business data usage information, business data source information and at least one preset key data item for multiple business data.

[0029] Based on the business scenario information, a data quality detection strategy is determined, specifically including: obtaining business data usage information, business data source information and at least one preset key data item of multiple business data in the business scenario information; determining the data source type of the data source corresponding to each business data through the business data source information, and determining the data source connection request information of each data source based on the data source type; determining at least one data quality detection rule corresponding to each business data according to the business data usage information corresponding to each business data, wherein the data quality detection rule includes any one or more of an integrity measurement, a consistency measurement, an accuracy measurement and a validity measurement; setting a quality detection priority through the at least one key data item, and determining the detection priority information, so as to determine the data quality detection strategy based on the data source connection request information, the data quality detection rule and the detection priority information.

[0030] In one embodiment of the present specification, the data source type (such as Hive table, relational database, NoSQL database, file system, etc.) corresponding to each business data is identified according to the business data source information. For each data source type, the corresponding data source connection request information is obtained, including authentication information, connection address, port number, etc., so that the data source can be successfully connected to the data source during the data quality detection process. Various types of data sources can be selected, including but not limited to Hive tables, relational databases, file systems, streaming message queues such as Kafka, and other big data storage formats that support Spark reading. Specifically, the database name, table name and possible partition information need to be provided for the Hive table, and the relational database requires the database connection URL, user name, password, table name, etc., and the Kafka topic name, consumer group, cluster address, etc. are configured.

[0031] According to the information on the use of business data, analyze the characteristics and quality requirements of each type of business data, and determine the applicable data quality detection rules for each type of business data. The data quality rules that can be selected mainly include integrity measurement, such as checking whether there are missing values ​​or empty values ​​in the record. Integrity verification can be performed by configuring specific fields to have non-empty values. It is generally used for the primary key and non-empty business fields of the table. Consistency measurement compares whether the data between different data sources are consistent, such as comparing whether the key fields of two tables match through JOIN operations. It is generally used for the foreign keys of the table. The two data tables are related through this field; accuracy measurement, such as setting business logic rules, such as age range restrictions, unique identifier checks, etc., to verify whether the data accurately reflects the actual situation. The field type is generally numeric; timeliness measurement, for example, for time series data, you can configure the data update frequency requirements to ensure that the data is timely and valid. It is generally used for date or time types. For example, in a health care system, integrity measurement ensures that fields such as name, age, and disease diagnosis in patient records are complete and not empty; consistency measurement verifies the consistency of patient records in different systems, for example, basic patient information should be consistent in the electronic medical record system and the drug management system, and the patient ID and diagnosis information fields are compared through JOIN operations; accuracy measurement ensures the accuracy of data by setting business rules, for example, the age field must be within a reasonable range, and the disease diagnosis must be consistent with the experimental results; for patient health monitoring data, ensure that the data can be updated within the specified time range, that is, effectiveness measurement, such as real-time monitoring of heart rate data so that medical measures can be taken in a timely manner.

[0032] Through the preset key data items, we can identify which data items have the greatest impact on business decisions, and set a higher quality detection priority for these key data items. Other non-critical data items are given corresponding priorities based on business needs. We can formulate a complete data quality detection strategy based on the data source connection request information, data quality detection rules, and detection priority information to ensure that the strategy covers all important business data and performs quality detection in order of priority.

[0033] Through the above technical solution, by formulating detailed data quality detection rules, including completeness, consistency, accuracy and effectiveness metrics, it is possible to systematically check data quality, ensure that the data meets business requirements and quality standards, and enhance the relevance of data quality detection rules and business scenarios; by presetting key data items and setting quality detection priorities, it is possible to prioritize data items that have the greatest impact on business decisions, help optimize resource allocation, ensure that key data is timely and effectively quality-checked, and reduce unnecessary investment in non-critical data.

[0034] Step S102: Perform quality detection task configuration through a data quality detection strategy to determine a corresponding data quality detection task set.

[0035] The data quality detection task set includes multiple data quality detection subtasks.

[0036] Through the data quality detection strategy, quality detection task configuration is performed to determine the corresponding data quality detection task set, specifically including: according to the data quality detection strategy, task parameters are configured to create a corresponding data quality detection task set; through the Griffin task scheduling system, the data quality detection task set is parsed to decompose it into multiple data quality detection subtasks, wherein the multiple data quality detection subtasks are independently executable detection tasks, and each of the data quality detection subtasks includes data source connection request information, data quality rule ID and subtask priority information.

[0037] In one embodiment of the present specification, task configuration is performed in the data center. Creating a data quality detection task set requires configuring a task set name, a data source list, a data quality rule list, and a priority setting. The task set name is used to identify the entire task set. The data source list contains the connection information of all data sources that need to be detected, such as the database URL, user name, password, etc. The data quality rule list specifies the data quality rule ID list that needs to be applied to each data source. Priority can also be set for the entire task set or a single data source / rule.

[0038] Use the API or interface provided by Griffin to create a data quality detection task set according to the above parameters. When creating, the system should verify the validity of the data source connection information and check whether the rule ID exists in the system. A parser should be implemented inside the Griffin task scheduling system to parse the task set into multiple independently executable data quality detection subtasks. Each subtask contains data source connection request information, data quality rule ID and subtask priority information. The data source connection request information is extracted from the task set and used to connect to the corresponding data source. The data quality rule ID is used to specify the data quality rule that needs to be applied to the subtask. It can be set according to the priority of the task set or the priority set for the subtask individually.

[0039] Step S103: submitting the data quality detection task set to the Griffin task scheduling system, distributing the multiple data quality detection subtasks through the Griffin task scheduling system, and determining the working node corresponding to each data quality detection subtask.

[0040] In one embodiment of the present specification, the task configured in the data center is submitted to the Griffin task scheduling system, and the system will encapsulate the task into a format suitable for Griffin task scheduler to recognize and process. When submitting, you can set resource requirements, that is, specify the computing resources (such as CPU, memory) required for the task to ensure that the task has sufficient resources under high load; you can also set alarms and configure alarm mechanisms after task execution failure or timeout to ensure timely response and problem handling. In addition, you can also set the execution frequency (such as cron expressions for scheduled tasks), dependencies, etc. Through the above steps, the task submission of data quality detection tasks is realized.

[0041] The Griffin task scheduler will first parse the submitted tasks and identify the basic attributes of the tasks (such as task ID, priority, execution mode, etc.) and specific quality detection rules. After that, the scheduler will decide how to distribute the tasks to different nodes in the Spark cluster based on the analysis of the current Spark cluster resource usage, including but not limited to computing resources (CPU, memory), storage resources, and network bandwidth. For example, during the shopping festival activities of the e-commerce system, the automatic expansion mechanism of the cloud platform (automatically increasing Spark cluster nodes) can be used to improve the system processing capacity. At the same time, when submitting data quality tasks, the metadata of the task priority can be set, and the system will allocate resources to tasks with high priority. The scheduler will further encapsulate the tasks into task descriptions suitable for Spark execution and submit them to Spark's resource management module. The Spark resource manager will then schedule these tasks to various worker nodes for execution.

[0042] The Griffin task scheduling system is used to distribute tasks to the multiple data quality detection subtasks and determine the working node corresponding to each of the data quality detection subtasks, specifically including: dynamically capturing the current resource usage information corresponding to each working node in the Spark big data computing engine through the Griffin task scheduling system, wherein the current resource usage information includes current computing resource data, current storage resource data and current network bandwidth data; performing task feature analysis on each of the data quality detection subtasks to determine the resource demand information corresponding to each of the data quality detection subtasks and the task dependency relationship between the multiple data quality detection subtasks; generating a resource allocation sequence corresponding to the multiple data quality detection subtasks according to the task dependency relationship between the multiple data quality detection subtasks and the subtask priority information of each of the data quality detection subtasks; and performing node allocation on each of the data quality detection subtasks in turn according to the resource allocation sequence and the resource demand information corresponding to each of the data quality detection subtasks and the current resource usage of each of the working nodes, and determining the working node corresponding to each of the data quality detection subtasks.

[0043] In one embodiment of the present specification, first, ensure that the Griffin system can be integrated with the monitoring system of the Spark cluster to collect and display the resource usage of each working node in the Spark cluster in real time. The Griffin system can obtain current computing resources (such as CPU usage, memory usage), storage resources (such as disk I / O, HDFS usage) and network bandwidth data by regularly calling the API interface of the monitoring tool. Analyze each data quality detection subtask to determine its computational complexity, memory requirements, disk I / O requirements and network bandwidth requirements, which can be estimated through historical data, task configuration or user input. Convert the analysis results into specific resource requirement information, including the minimum number of CPU cores, minimum / maximum memory, disk I / O rate and network bandwidth requirements. By analyzing the data flow and processing order between the data quality detection subtasks, the dependencies between the data quality detection subtasks are determined, and directed graphs can be used to represent these dependencies. Combined with the priority information of each subtask and the task dependencies, a resource allocation order is generated. The resource allocation order takes into account both the priority of the task and the dependencies between the tasks. According to the resource allocation order, the Griffin system traverses each subtask to be assigned. For each subtask, the matching algorithm is used to select the most suitable node for allocation based on its resource demand information and the resource usage of the current working node. The matching algorithm here can be the best fit algorithm or the first fit algorithm. Submit the task through YARN and schedule each subtask to be executed on the assigned working node.

[0044] The Griffin task scheduling system can process multiple data quality detection subtasks in parallel, fully utilize computing resources, significantly shorten the overall detection time, and intelligently allocate tasks according to the load of the working nodes to ensure that the resources of each node are evenly utilized, avoiding the situation where some nodes are overloaded while other nodes are idle. For data quality tasks that require real-time detection, Griffin can quickly respond and distribute tasks to the corresponding working nodes to achieve near-real-time data quality monitoring. Through the data quality rules defined by Griffin, the accuracy, completeness, timeliness, uniqueness, validity, consistency and other dimensions of data can be accurately measured to ensure comprehensive monitoring of data quality. All data quality detection tasks are centrally managed in the Griffin task scheduling system for unified monitoring and scheduling. By configuring scheduled tasks, data quality detection can be automatically executed, reducing manual intervention and lowering the error rate. Griffin supports a variety of data sources and data processing methods, and can be easily expanded to meet new data quality detection needs.

[0045] Step S104, through each working node, execute the data quality detection subtask, generate the quality detection result corresponding to each working node, integrate the quality detection results according to the current business scenario information, and determine the data quality report.

[0046] Through each of the working nodes, the data quality detection subtask is executed to generate a quality detection result corresponding to each of the working nodes, specifically including: determining the data source connection request information, data quality rule ID and subtask priority information corresponding to each of the data quality detection subtasks; through a preset data reading intermediate layer, according to the data source connection request information, the data source is connected to obtain the data to be detected, wherein the data source includes a relational data source and a big data storage data source; according to each of the working nodes, according to the data quality rule ID, the corresponding quality detection algorithm is matched to perform quality detection on the data to be detected, and the quality detection result corresponding to each of the working nodes is generated.

[0047] In one embodiment of the present specification, in the task scheduling system, each data quality detection subtask first needs to clarify its corresponding data source connection request information. This information usually includes the type of data source (such as MySQL, Oracle, Hadoop HDFS, Hive, etc.), connection address (IP address or domain name), port number, database name, authentication information, etc. Through the data reading middle layer, adapt the relational data source connection and the big data storage system. The middle layer service is responsible for identifying the data source type according to the request parameters, selecting the appropriate adapter for data reading or writing operations, and providing API or service endpoints to allow the caller to access different types of data sources in a unified manner. Each subtask also needs to specify its corresponding data quality rule ID, which is used to reference predefined quality detection rules (such as data integrity check, consistency verification, range check, etc.) during the quality detection process. At the same time, the priority information of the subtask also needs to be clear so that more critical or more urgent tasks can be given priority when resources are limited. The data reading middle layer selects the appropriate adapter to connect to different data sources based on the data source connection request information, hiding the complexity of the underlying data source and providing a unified and standardized data access interface (API or service endpoint) for the upper-layer application. The Griffin task scheduling system can implement reading and writing operations on various types of data without having to care about the specific data source type or connection method. The data reading middle layer obtains the data to be tested, and matches the corresponding quality detection algorithm according to the data quality rule ID specified on each working node, including various built-in or custom inspection logic, which is used to evaluate the accuracy, completeness, consistency and other aspects of the data. After the quality detection algorithm is executed, the quality detection results corresponding to each working node will be generated for data analysis, report generation or triggering the corresponding processing flow (such as data cleaning, correction, etc.).

[0048] Through the above technical solutions, efficient docking and flexible management of different types of data sources can be achieved, while ensuring the accuracy and reliability of the data quality detection process. The application of the data reading middle layer greatly simplifies the complexity of data access and improves the scalability and maintainability of the entire system. The task distribution and execution mechanism based on work nodes and data quality rule IDs ensures the efficient execution of quality detection tasks and the rational use of resources.

[0049] After executing the data quality detection subtask through each working node and generating the quality detection result corresponding to each working node, the method also includes: monitoring the subtask execution status of each working node through a preset real-time monitoring interface, and collecting the real-time task execution data of each working node, wherein the real-time task specified data includes real-time node resource usage information and node task execution information; determining the real-time node task execution status corresponding to each working node according to the real-time node resource usage information and the node task execution information of each working node; and adjusting the task distribution based on the real-time node task execution status corresponding to each working node.

[0050] In one embodiment of this specification, by connecting to the real-time monitoring interface provided by Spark, the system can dynamically capture and display the usage of various key resources during the operation of data quality tasks, including but not limited to server-level CPU utilization, memory consumption, and resource allocation status on the YARN cluster, as well as disk I / O performance indicators and network traffic statistics, so that managers can fully understand and track resource bottlenecks and efficiency issues during task execution. On this basis, the API interface of Griffin is further expanded to allow fine-grained resource monitoring at the user level and project level by specifying task IDs and project IDs.

[0051] Collect the real-time node resource usage information and node task execution information of each working node. The node resource usage information includes the usage of key resources such as CPU usage, memory usage, disk I / O, network bandwidth, etc. The node task execution information refers to the list of tasks currently running on the node, the execution progress of each task, the amount of resources required for the task, and the priority of the task.

[0052] According to the collected resource usage information, the resource load of each node is evaluated to determine whether the node is in a high load, low load or moderate load state. The high load, low load or moderate load here can be determined by the load ratio of the current load to the total load, for example, 0-30% is low load, 30%-60% is medium load, and more than 60% is high load. Combined with the task execution information, analyze the execution efficiency of each task and whether it encounters resource bottlenecks or dependency problems. Combine the resource usage and task execution status to determine a real-time task execution status for each working node. The status can include "busy" (high load and task execution), "idle" (low load and short task queue), "overload" (resources are close to or exceed the threshold), etc. For overloaded nodes, consider migrating some tasks to idle or moderately loaded nodes to balance the load and avoid performance bottlenecks. According to the urgency of the task and the resource requirements, adjust the execution priority of the task on the node. High-priority tasks can be executed first, while low-priority tasks can be executed when resources are sufficient. According to the real-time node resource usage information and node task execution information of each working node, the task execution status of the node is dynamically determined, and effective task distribution adjustment is performed, thereby improving the overall performance and resource utilization of the system.

[0053] According to the current business scenario information, the quality detection results are integrated to determine the data quality report, which specifically includes: collecting the quality detection results of each working node to summarize the data quality based on multiple quality detection results, determine the summary result data, and determine the abnormal type of the abnormal data items in the summary result data; determine the corresponding quality result display type through the current business scenario information and the abnormal type; render the summary result data according to the quality result display type, and construct a data quality problem view to determine the data quality report.

[0054] In one embodiment of the present specification, after the tasks on each node complete their respective quality inspections, the API interface provided by Griffin is called to regularly obtain the results of the completed data quality inspection tasks. Griffin will integrate the results of all subtasks to form a complete data quality report. After collecting the quality inspection results of all working nodes, the quality inspection results of all working nodes are summarized. If multiple nodes have detected the same data item, it is necessary to remove duplicate results; classify the data according to the different test results (such as pass, fail, warning, etc.), and count the number of various types of test results, such as the total number of test items, the number of passed items, the number of failed items, etc. In the summary results, determine the data items that have not passed the quality inspection, that is, the abnormal data items. For each abnormal data item, it is necessary to determine its abnormal type, which is usually determined based on the rule ID or abnormal description violated during the detection. The abnormal type may include but is not limited to: missing data, wrong data type, wrong data format, inconsistent data, data out of range, etc.

[0055] After determining the abnormal data items and their abnormal types, the system needs to determine the appropriate quality result display type based on the current business scenario information and abnormal type. Different business scenarios and abnormal types may require different display methods. The corresponding quality result display type is determined by the current business scenario information and abnormal type. The quality result display types here include problem network diagrams, time trend diagrams, and quality score radar diagrams. For example, when you need to monitor data quality changes, such as tracking long-term changes in indicators such as data accuracy, completeness or consistency, identifying trend problems or evaluating improvement effects, observing the effects of data quality improvement measures after implementation, and analyzing whether the improvement of data quality is sustained, you can use time trend diagrams; when you need to comprehensively evaluate the data quality of different business modules to find areas that need improvement, use radar diagrams; when looking for the source of data inconsistency or integrity problems, you need to show how data flows and transforms in the system, and use network diagrams at this time. Parse and transform the structured data obtained from the Griffin API and adapt it to a data structure that can be recognized by the G6 graph visualization engine. Use the G6 graph visualization engine to render the processed result data, build an intuitive and easy-to-understand data quality problem view, and display it in the data middle platform as a data quality report. The data center determines whether the test results exceed the preset range. If so, it automatically sends a notification to the relevant responsible person to facilitate timely response to data quality issues. For the problems found, the rectification process can also be tracked until the problem is resolved, forming a complete closed-loop management mechanism.

[0056] Through the above technical solution, by clarifying the current business scenario information, the data scope and data quality rules that need to be tested can be determined more accurately, thereby avoiding the waste of resources caused by indiscriminate comprehensive testing. The data quality rules customized for specific business scenarios can effectively identify and solve common data quality problems in the scenario, and improve the accuracy and reliability of the data; through the Griffin task scheduling system, the detection tasks can be efficiently distributed and executed, and the quality detection results can be quickly generated; based on the data quality detection strategy customized for the business scenario, the detection resources can be allocated more reasonably to avoid unnecessary waste; for data quality tasks that require real-time detection, Griffin can quickly respond and distribute tasks to the corresponding work nodes to achieve near-real-time data quality monitoring; through the data quality rules defined by Griffin, the accuracy, completeness, timeliness, uniqueness, validity, consistency and other dimensions of the data can be accurately measured to ensure comprehensive monitoring of data quality and improve the efficiency of data quality detection.

[0057] The embodiment of this specification also provides a data quality detection device based on a business scenario, such as Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0058] The embodiments of the present specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0059] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0060] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0061] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0062] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0063] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0066] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0067] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0068] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0069] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0070] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A data quality detection method based on business scenarios, characterized in that: The method comprises: Determine current business scenario information corresponding to the current data quality detection requirement, and determine a data quality detection strategy based on the business scenario information, wherein the data quality detection strategy includes data source connection request information of at least one data source corresponding to the data to be detected and at least one data quality rule; Performing quality detection task configuration through the data quality detection strategy to determine a corresponding data quality detection task set, wherein the data quality detection task set includes a plurality of data quality detection subtasks; Submitting the data quality detection task set to the Griffin task scheduling system, distributing the multiple data quality detection subtasks through the Griffin task scheduling system, and determining the working node corresponding to each of the data quality detection subtasks; The data quality detection subtask is executed through each of the working nodes to generate a quality detection result corresponding to each of the working nodes. The quality detection results are integrated according to the current business scenario information to determine a data quality report.

2. The data quality detection method based on business scenarios according to claim 1 is characterized in that: Based on the business scenario information, determine the data quality detection strategy, which specifically includes: Acquire business data usage information, business data source information and at least one preset key data item of multiple business data in the business scenario information; Determine the data source type of the data source corresponding to each type of business data through the business data source information, and determine the data source connection request information of each data source based on the data source type; Determine at least one data quality detection rule corresponding to each type of business data according to the business data usage information corresponding to each type of business data, wherein the data quality detection rule includes any one or more of integrity measurement, consistency measurement, accuracy measurement and effectiveness measurement; The quality detection priority is set through the at least one key data item, and the detection priority information is determined to determine the data quality detection strategy based on the data source connection request information, data quality detection rules and detection priority information.

3. The data quality detection method based on business scenarios according to claim 1 is characterized in that: Through the data quality detection strategy, quality detection task configuration is performed to determine the corresponding data quality detection task set, which specifically includes: According to the data quality detection strategy, configure task parameters to create a corresponding data quality detection task set; The data quality detection task set is parsed through the Griffin task scheduling system to be decomposed into multiple data quality detection subtasks, wherein the multiple data quality detection subtasks are independently executable detection tasks, and each of the data quality detection subtasks includes data source connection request information, data quality rule ID and subtask priority information.

4. The data quality detection method based on business scenarios according to claim 1 is characterized in that: By using the Griffin task scheduling system, the multiple data quality detection subtasks are distributed, and a working node corresponding to each of the data quality detection subtasks is determined, specifically including: Dynamically capture the current resource usage information corresponding to each working node in the Spark big data computing engine through the Griffin task scheduling system, wherein the current resource usage information includes current computing resource data, current storage resource data, and current network bandwidth data; Performing task feature analysis on each of the data quality detection subtasks to determine resource requirement information corresponding to each of the data quality detection subtasks and task dependencies between the multiple data quality detection subtasks; Generate a resource allocation sequence corresponding to the multiple data quality detection subtasks according to the task dependency relationship between the multiple data quality detection subtasks and the subtask priority information of each data quality detection subtask; According to the resource allocation order, based on the resource requirement information corresponding to each data quality detection subtask and the current resource usage of each working node, each data quality detection subtask is allocated nodes in turn to determine the working node corresponding to each data quality detection subtask.

5. The data quality detection method based on business scenarios according to claim 1 is characterized in that: By each of the working nodes, the data quality detection subtask is executed to generate a quality detection result corresponding to each of the working nodes, specifically including: Determine the data source connection request information, data quality rule ID and subtask priority information corresponding to each of the data quality detection subtasks; Through a preset data reading middle layer, data source docking is performed according to the data source connection request information to obtain the data to be detected, wherein the data source includes a relational data source and a big data storage data source; According to each of the working nodes, according to the data quality rule ID, a corresponding quality detection algorithm is matched to perform quality detection on the data to be detected, and a quality detection result corresponding to each of the working nodes is generated.

6. The data quality detection method based on business scenarios according to claim 1 is characterized in that: After executing the data quality detection subtask through each of the working nodes and generating a quality detection result corresponding to each of the working nodes, the method further includes: Through a preset real-time monitoring interface, the subtask execution status of each of the working nodes is monitored, and the real-time task execution data of each of the working nodes is collected, wherein the real-time task specified data includes real-time node resource usage information and node task execution information; Determine the real-time node task execution status corresponding to each of the working nodes according to the real-time node resource usage information and the node task execution information of each of the working nodes; Task distribution adjustment is performed based on the real-time node task execution status corresponding to each of the working nodes.

7. The data quality detection method based on business scenarios according to claim 1 is characterized in that: According to the current business scenario information, the quality detection results are integrated to determine a data quality report, which specifically includes: Collect the quality detection result of each of the working nodes, perform data quality aggregation based on the multiple quality detection results, determine the aggregated result data, and determine the abnormal type of the abnormal data item in the aggregated result data; Determine the corresponding quality result display type according to the current business scenario information and the exception type; The summary result data is rendered according to the quality result display type, and a data quality problem view is constructed to determine the data quality report.

8. The data quality detection method based on business scenarios according to claim 1 is characterized in that: Determine the current business scenario information corresponding to the current data quality detection requirements, including: When triggered by the user's quality inspection operation, a business scenario form corresponding to the quality inspection requirement is sent to the user terminal, wherein the business scenario form includes a business data type item, a business data usage item, a business data source item, and a custom key data item; According to the business scenario form, current business scenario information corresponding to the current data quality detection requirements is determined, wherein the current business scenario information includes business data usage information of multiple business data, business data source information and at least one preset key data item.

9. A data quality detection device based on business scenarios, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images