Data quality detection method and system

Abstract

The application provides a data quality detection method and system, wherein the data quality detection method comprises the following steps: receiving a plurality of quality detection tasks input by a user, wherein the quality detection task comprises a quality detection rule and data element information; assigning the plurality of quality detection tasks to at least one data acquisition unit according to the data element information of the plurality of quality detection tasks, wherein the data element information in the quality detection task is used for the data acquisition unit to acquire to-be-processed data, the quality detection rule in the quality detection task is used for processing the to-be-processed data to generate to-be-verified data of the quality detection task, and the to-be-verified data is stored in a database; acquiring corresponding target to-be-verified data from the database according to the data element information in a first quality detection task, wherein the first quality detection task is any one of the plurality of quality detection tasks; and verifying the target to-be-verified data according to the quality detection rule in the first quality detection task to obtain a data quality detection result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data quality inspection method. This application also relates to a data quality inspection system, a data quality inspection device, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of computer technology, the types and sources of data are becoming increasingly diverse. The quality of data, such as timeliness and consistency, directly affects whether statistical analysis can draw correct conclusions. Data analysis and data mining cannot be separated from high-quality data, and data quality detection technology has emerged as a result.

[0003] In existing technologies, data quality inspection mostly targets a single data quality check rule. A specific task is created using a real-time computing framework to consume designated data, output the consumption results, and finally perform data quality checks and alerts based on these results. However, when there is a large amount of data and data quality check rules, this method requires creating a large number of tasks, leading to excessive processing load and wasted resources. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data quality detection method. This application also relates to a data quality detection system, a data quality detection device, a computing device, and a computer-readable storage medium, to address the technical shortcomings of existing technologies such as excessive processing volume and resource waste.

[0005] According to a first aspect of the embodiments of this application, a data quality detection method is provided, applied to a visualization processing unit, including:

[0006] Receive multiple quality inspection tasks input by the user, wherein the quality inspection tasks include quality inspection rules and data element information;

[0007] Based on the data metadata of the multiple quality inspection tasks, the multiple quality inspection tasks are assigned to at least one data acquisition unit. The data metadata of the quality inspection tasks is used by the data acquisition unit to acquire data to be processed. The quality inspection rules of the quality inspection tasks are used to process the data to be processed to generate the data to be verified for the quality inspection tasks. The data to be verified is stored in a database.

[0008] Based on the data element information in the first quality inspection task, the corresponding target data to be verified is obtained from the database, wherein the first quality inspection task is any one of the plurality of quality inspection tasks;

[0009] According to the quality inspection rules in the first quality inspection task, the target data to be inspected is inspected to obtain the data quality inspection result.

[0010] Optionally, the data metadata includes a data identifier;

[0011] The step of assigning the multiple quality inspection tasks to at least one data acquisition unit based on the data element information of the multiple quality inspection tasks includes:

[0012] The quality inspection tasks with the same data identifier are grouped into the same task group, and the data level corresponding to the data identifier of each task group is queried.

[0013] Each task group is assigned to at least one data acquisition unit based on the data level corresponding to each task group.

[0014] Optionally, assigning each task group to at least one data acquisition unit according to the data level corresponding to each task group includes:

[0015] The first task group with a data level higher than a set threshold is assigned to the first data acquisition unit. The data identifier corresponding to the first task group is used by the first data acquisition unit to acquire the data to be processed. The quality detection rules of each quality detection task in the first task group are used to process the data to be processed and generate the data to be verified for each quality detection task in the first task group.

[0016] At least one second task group whose data level is lower than or equal to the set threshold is assigned to the second data acquisition unit, wherein the data identifier corresponding to each second task group is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

[0017] Optionally, before retrieving the corresponding target data to be verified from the database based on the data metadata in the first quality inspection task, the method further includes:

[0018] The quality inspection rules for the second quality inspection task are sent to the database. The second quality inspection task is any quality inspection task in any second task group. The quality inspection rules for the second quality inspection task are used by the database to process the data to be processed corresponding to the second quality inspection task, generate and store the data to be verified for the second quality inspection task, and the data to be processed corresponding to the second quality inspection task is sent by the second data acquisition unit.

[0019] Optionally, the method further includes:

[0020] Receive a first change instruction for the quality inspection rules in the first task group, wherein the first change instruction carries the changed quality inspection rules;

[0021] The modified quality inspection rules are sent to the first data acquisition unit, wherein the modified quality inspection rules are used by the first data acquisition unit to process the data to be processed and generate the data to be verified for each quality inspection task in the first task group.

[0022] Optionally, the method further includes:

[0023] Receive a second change instruction for data element information in any second task group, wherein the second change instruction carries the changed data element information;

[0024] The modified data metadata is sent to the second data acquisition unit, wherein the data identifier contained in the modified data metadata is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

[0025] Optionally, obtaining the corresponding target data to be verified from the database based on the data element information in the first quality inspection task includes:

[0026] Based on the data element information in the first quality inspection task, a read request for the data to be verified corresponding to the first quality inspection task is generated.

[0027] The read request is sent to the proxy service unit, wherein the read request is used by the proxy service unit to determine the target database, obtain the target data to be verified corresponding to the read request from the target database, and feed it back to the visualization processing unit.

[0028] According to a second aspect of the embodiments of this application, a data quality inspection system is provided, comprising:

[0029] A visualization processing unit is configured to receive multiple quality inspection tasks input by a user, wherein the quality inspection tasks include quality inspection rules and data element information; and to assign the multiple quality inspection tasks to at least one data acquisition unit according to the data element information of the multiple quality inspection tasks.

[0030] The data acquisition unit is used to acquire data to be processed based on the data element information in the quality inspection task, wherein the quality inspection rules in the quality inspection task are used to process the data to be processed to generate the data to be verified for the quality inspection task, and the data to be verified is stored in the database.

[0031] The visualization processing unit is further configured to retrieve the corresponding target data to be verified from the database based on the data element information in the first quality detection task, wherein the first quality detection task is any one of the plurality of quality detection tasks; and to verify the target data to be verified according to the quality detection rules in the first quality detection task to obtain the data quality detection result.

[0032] Optionally, the data metadata includes data identifiers and parsing strategies;

[0033] The data acquisition unit is further configured to acquire an initial data stream based on the data identifier in the quality inspection task; and to parse the initial data stream according to the parsing strategy in the quality inspection task to obtain the data to be processed corresponding to the quality inspection task.

[0034] Optionally, the data element information includes a data identifier, and the at least one data acquisition unit includes a first data acquisition unit and a second data acquisition unit;

[0035] The visualization processing unit is further configured to divide the quality inspection tasks with the same data identifier into the same task group, and query the data level corresponding to the data identifier of each task group; assign the first task group with the data level higher than the set threshold to the first data acquisition unit, and assign at least one second task group with the data level lower than or equal to the set threshold to the second data acquisition unit.

[0036] The first data acquisition unit is used to acquire data to be processed according to the data identifier corresponding to the first task group; and to process the data to be processed according to the quality detection rules of each quality detection task in the first task group to generate the verification data of each quality detection task in the first task group.

[0037] The second data acquisition unit is used to acquire the data to be processed corresponding to each second task group according to the data identifier corresponding to each second task group.

[0038] Optionally, the system further includes multiple databases and agent service units;

[0039] The data acquisition unit is used to generate a write request and send it to the proxy service unit. When the data acquisition unit is a first data acquisition unit, the write request carries data to be verified. When the data acquisition unit is a second data acquisition unit, the write request carries data to be processed.

[0040] The proxy service unit is used to determine the target database from the plurality of databases according to the write request and the set routing algorithm, and forward the write request to the target database;

[0041] The target database is used to store the data to be verified when the write request carries data to be verified, or to process the data to be processed according to the data processing rules corresponding to the data to be processed sent by the visualization processing unit when the write request carries data to be processed, thereby generating and storing the data to be verified.

[0042] Optionally, the write request may also carry data metadata.

[0043] The system also includes multiple database clusters, each database cluster including multiple databases and database management units;

[0044] The proxy service unit is also used to determine the target database cluster from the multiple database clusters based on the set routing algorithm, the load of each database cluster and the data element information carried by the write request, and forward the write request to the database management unit in the target database cluster.

[0045] The database management unit is configured to determine the target database from among the multiple databases included in the target database set based on the data element information carried in the write request; and forward the write request to the target database.

[0046] According to a third aspect of the embodiments of this application, a data quality detection apparatus is provided, applied to a visualization processing unit, comprising:

[0047] The receiving module is configured to receive multiple quality detection tasks input by the user, wherein the quality detection tasks include quality detection rules and data element information;

[0048] The allocation module is configured to allocate the multiple quality inspection tasks to at least one data acquisition unit based on the data metadata of the multiple quality inspection tasks, wherein the data metadata in the quality inspection tasks is used by the data acquisition unit to acquire data to be processed, the quality inspection rules in the quality inspection tasks are used to process the data to be processed to generate the data to be verified for the quality inspection tasks, and the data to be verified is stored in a database.

[0049] The acquisition module is configured to acquire the corresponding target data to be verified from the database based on the data element information in the first quality inspection task, wherein the first quality inspection task is any one of the plurality of quality inspection tasks;

[0050] The verification module is configured to verify the target data to be verified according to the quality detection rules in the first quality detection task, and obtain the data quality detection result.

[0051] According to a fourth aspect of the present application, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps of the data quality detection method.

[0052] According to a fifth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the data quality detection method.

[0053] The data quality inspection method provided in this application receives multiple quality inspection tasks input by a user, wherein each quality inspection task includes quality inspection rules and data metadata. Based on the data metadata of the multiple quality inspection tasks, the methods allocate these tasks to at least one data acquisition unit. The data metadata of each quality inspection task is used by the data acquisition unit to acquire data to be processed, and the quality inspection rules of each quality inspection task are used to process the data to be processed to generate data to be verified for the quality inspection task. This data to be verified is stored in a database. Based on the data metadata of a first quality inspection task, the method retrieves corresponding target data to be verified from the database, wherein the first quality inspection task is any one of the multiple quality inspection tasks. Based on the quality inspection rules of the first quality inspection task, the method verifies the target data to be verified to obtain a data quality inspection result. By allocating multiple quality inspection tasks to at least one data acquisition unit, allowing one data acquisition unit to acquire multiple data to be processed or to process data to be processed according to multiple data quality rules, the method achieves computational resource reuse, improves resource utilization, avoids re-competition for resources, and improves system availability. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the structure of a data quality inspection system provided in one embodiment of this application;

[0055] Figure 2A This is a flowchart illustrating the overall process of a data quality detection method provided in one embodiment of this application.

[0056] Figure 2B This is a flowchart illustrating the processing performed by the second data acquisition unit in a data quality detection method provided in an embodiment of this application;

[0057] Figure 2C This is a flowchart illustrating the processing performed by the first data acquisition unit in a data quality detection method provided in an embodiment of this application;

[0058] Figure 2DThis is a flowchart illustrating the data acquisition unit acquiring data in a data quality detection method according to an embodiment of this application;

[0059] Figure 2E This is a flowchart illustrating the process of a proxy service unit determining a target database cluster in a data quality detection method according to an embodiment of this application;

[0060] Figure 2F This is a flowchart illustrating the process of a database management unit determining a target database in a data quality detection method according to an embodiment of this application;

[0061] Figure 3 This is a flowchart of a data quality detection method provided in an embodiment of this application;

[0062] Figure 4 This is a schematic diagram of the structure of a data quality detection device provided in one embodiment of this application;

[0063] Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this application. Detailed Implementation

[0064] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0065] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” used in one or more embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” used in one or more embodiments of this application refers to any or all possible combinations including one or more of the associated listed items.

[0066] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this application, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0067] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0068] Data quality inspection refers to the inspection of at least one of the following quality standards: data integrity, uniqueness, timeliness, accuracy, and consistency, and the provision of early warnings.

[0069] InfluxDB, an open-source time-series database, is a database focused on high-performance reading, high-performance writing, efficient storage, and real-time analysis of massive time-series data. It is widely used in DevOps monitoring, IoT monitoring, real-time analysis, and other scenarios.

[0070] Continuous Queries (CQ): These queries can run automatically and periodically on real-time data while storing the results.

[0071] Data identifier: also known as data topic, such as message topic, is a common representation of real-time data resources, usually KafkaTopic.

[0072] Consumption refers to reading data from a Topic and performing certain specific data processing operations.

[0073] Hot update: refers to the ability to sense changes in the operating environment while the task is running and dynamically adjust the task's running information without restarting.

[0074] Next, a brief description of the data quality testing method provided in this application will be given.

[0075] With the continuous development of computer technology, the types and sources of data are becoming increasingly diverse. The quality of data, such as timeliness and consistency, directly affects whether statistical analysis can draw correct conclusions. Data analysis and data mining cannot be separated from high-quality data, and data quality detection technology has emerged as a result.

[0076] In existing technologies, data quality inspection mostly targets a single data quality check rule. A specific task is created using a real-time computing framework to consume the specified data, output the consumption results, and finally perform data quality checks and alerts based on these results. For example, performing data quality checks on specified data requires creating a separate task to consume that data. However, task resources cannot be reused, and excessive data can strain cluster resources. Furthermore, creating multiple data quality check rules for the same data requires creating multiple tasks, each consuming the data and executing different business logic, leading to duplicate data consumption and significant resource waste. In short, with a large amount of data and data quality check rules, a large number of tasks need to be created, resulting in excessive processing load and resource waste. In addition, each update to a task requires restarting it, and each restart involves resource contention; failure to compete for resources can cause task startup failure, impacting system availability.

[0077] Therefore, this application provides a data quality monitoring method that receives multiple quality detection tasks input by a user, wherein each quality detection task includes quality detection rules and data metadata; based on the data metadata of the multiple quality detection tasks, the multiple quality detection tasks are assigned to at least one data acquisition unit, wherein the data metadata in the quality detection tasks is used by the data acquisition unit to acquire data to be processed, and the quality detection rules in the quality detection tasks are used to process the data to be processed to generate data to be verified for the quality detection tasks, and the data to be verified is stored in a database; based on the data metadata in a first quality detection task, the corresponding target data to be verified is retrieved from the database, wherein the first quality detection task is any one of the multiple quality detection tasks; based on the quality detection rules in the first quality detection task, the target data to be verified is verified to obtain a data quality detection result. By assigning multiple quality detection tasks to at least one data acquisition unit, allowing one data acquisition unit to acquire multiple data to be processed or to process data to be processed according to multiple data quality rules, computing resources are reused, resource utilization is improved, resource competition is avoided, and system availability is improved.

[0078] This application provides a data quality inspection method, and also relates to a data quality inspection system, a data quality inspection device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0079] Figure 1 This application provides a schematic diagram of the structure of a data quality inspection system according to an embodiment, which specifically includes:

[0080] The visualization processing unit 102 is used to receive multiple quality inspection tasks input by the user, wherein the quality inspection tasks include quality inspection rules and data element information; and to assign the multiple quality inspection tasks to at least one data acquisition unit 104 according to the data element information of the multiple quality inspection tasks.

[0081] The data acquisition unit 104 is used to acquire data to be processed based on the data element information in the quality inspection task, wherein the quality inspection rules in the quality inspection task are used to process the data to be processed to generate the data to be verified for the quality inspection task, and the data to be verified is stored in the database 106.

[0082] The visualization processing unit 102 is further configured to obtain the corresponding target data to be verified from the database 106 according to the data element information in the first quality detection task, wherein the first quality detection task is any one of the plurality of quality detection tasks; and to verify the target data to be verified according to the quality detection rules in the first quality detection task to obtain the data quality detection result.

[0083] Specifically, a visualization processing unit refers to a functional unit that can provide a visual interface and perform data processing. A quality inspection task refers to a task created to inspect data for at least one of the following quality standards: integrity, uniqueness, timeliness, accuracy, and consistency. Quality inspection rules, also known as data quality inspection rules, are rules designed for at least one of the following quality standards: integrity, uniqueness, timeliness, accuracy, and consistency. Data metadata refers to the metadata of the data being inspected. A data acquisition unit refers to a functional unit that acquires the data to be inspected, i.e., a data consumption task or unit. Data to be verified refers to the processing result obtained after processing the data to be processed. A database is an ordered collection of structured information or data; it can be a relational database, such as a Structured Query Language (SQL) database, or a time-series database, such as InfluxDB. Data quality inspection results refer to whether the data conforms to the quality inspection rules.

[0084] In practical applications, users set up and submit multiple quality inspection tasks through the visualization page provided by the visualization processing unit. Correspondingly, the visualization processing unit receives multiple quality inspection tasks. These tasks can be performed on the same data using different quality standards, such as two tasks: uniqueness inspection and timeliness inspection of data A. Alternatively, they can be performed on different data using the same or different quality standards, such as three tasks: uniqueness inspection of data A, timeliness inspection of data B, and uniqueness inspection of data C. Other methods are also possible, and this specification does not limit the specific tasks.

[0085] Furthermore, the visualization page determines the data acquisition unit corresponding to each quality inspection task based on the data metadata of each task, and then assigns each quality inspection task to its corresponding data acquisition unit. It should be noted that different quality inspection tasks may correspond to the same or different data acquisition units. For example, if there are five quality inspection tasks, tasks 1-4 may be assigned to the same data acquisition unit, while task 5 may be assigned to a different data acquisition unit.

[0086] For any data acquisition unit that receives a quality inspection task, the data acquisition unit acquires the data to be processed corresponding to each quality inspection task based on the data metadata received from each quality inspection task. For example, if a data acquisition unit receives three quality inspection tasks, and the data metadata of quality inspection task one and quality inspection task three are both metadata of data to be processed 1, then data to be processed 1 is acquired based on the metadata of data to be processed 1. If the data metadata of quality inspection task three is metadata of data to be processed 2, then data to be processed 2 is acquired based on the metadata of data to be processed 2.

[0087] Furthermore, the data to be processed can be generated based on the quality inspection rules in the quality inspection task. Continuing with the previous example, data to be processed 1 is processed according to the quality inspection rules of quality inspection task one to obtain data to be verified 1; data to be processed 1 is processed according to the quality inspection rules of quality inspection task two to obtain data to be verified 2; and data to be processed 2 is processed according to the quality inspection rules of quality inspection task three to obtain data to be verified 3.

[0088] The visualization processing unit, for each of multiple quality inspection tasks, can retrieve the target inspection data for that task from the database based on the task's metadata. Then, it compares the quality inspection rules for that task with the target inspection data to obtain the data quality inspection result. Continuing with the previous example, based on the quality inspection rules for quality inspection task one, it retrieves the data to be verified from the database. Then, it compares the data to be verified with the quality inspection standards in the quality inspection rules. If the data to be verified meets the quality inspection standards, the data quality inspection result is qualified; if the data to be verified does not meet the quality inspection standards, the data quality inspection result is unqualified.

[0089] In an optional example of this specification, the data to be processed needs to be parsed. In this case, the data metadata includes a data identifier and a parsing strategy; the data acquisition unit 104 is further configured to acquire an initial data stream based on the data identifier in the quality inspection task; and parse the initial data stream according to the parsing strategy in the quality inspection task to obtain the data to be processed corresponding to the quality inspection task.

[0090] Specifically, data identification refers to the identifier corresponding to the data to be processed, such as the topic and name of the data, i.e., the message topic. Parsing strategy refers to the methods, approaches, and means of parsing the data. Initial data stream refers to the unparsed data to be processed.

[0091] In practical applications, for any received quality inspection task, the data acquisition unit obtains the initial data stream corresponding to the data identifier contained in the data metadata of the quality inspection task. Then, according to the parsing strategy contained in the data metadata of the quality inspection task, the initial data stream is parsed to obtain the data to be processed corresponding to the quality inspection task. Thus, obtaining the initial data stream based on the data identifier in the data metadata and parsing the initial data stream based on the parsing strategy contained in the data metadata not only improves parsing efficiency, thereby improving the efficiency of obtaining the data to be processed, but also improves the reliability of the data to be processed, further enhancing the accuracy of data quality inspection.

[0092] In another optional example of this specification, the data to be processed is directly obtainable and does not require parsing. In this case, the data metadata includes a data identifier; the data acquisition unit 104 is further configured to acquire the data to be processed corresponding to the quality inspection task based on the data identifier in the quality inspection task.

[0093] In one or more optional embodiments of this specification, the visualization processing unit 102 is also used to assign quality inspection tasks with the same data metadata to the same data acquisition unit. Since the quality inspection tasks with the same data metadata perform the same data inspection, assigning these tasks to the same data acquisition unit allows the data acquisition unit to be a dedicated acquisition unit for a specific data, avoiding the resource waste of assigning quality inspection tasks with the same data identifier to different data acquisition units, where each data acquisition unit would need to acquire that data.

[0094] In one or more optional instances of this specification, the data element information includes a data identifier; the visualization processing unit 102 is further configured to divide quality inspection tasks with the same data identifier into the same task group, and query the data level corresponding to the data identifier of each task group; and assign each task group to at least one data acquisition unit according to the data level corresponding to each task group.

[0095] Specifically, a task group refers to a collection of at least one quality inspection task. A data level refers to a grade or level defined based on the amount or volume of data.

[0096] In practical applications, the visualization processing unit can group quality inspection tasks with the same data identifier into the same task group. Then, based on the data identifier corresponding to each task group, it queries the data volume or data flow corresponding to that data identifier, i.e., the data volume or data flow of the data to be inspected. The data level corresponding to this data volume or data flow is then used as the data level for that task group. Furthermore, based on the data level of each task group, each task group is assigned to at least one data acquisition unit. That is, quality inspection tasks within the same task group are assigned to the same data acquisition unit. Different task groups can be assigned to different data acquisition units according to the size of their data levels, or they can be assigned to the same data acquisition unit. Since the data identifiers of all quality inspection tasks within the same task group are the same, meaning that the data being inspected by each quality inspection task within the same task group is the same, assigning all quality inspection tasks within the same task group to the same data acquisition unit avoids the resource waste of assigning quality inspection tasks with the same data identifier to different data acquisition units, thus preventing each data acquisition unit from needing to acquire the same data. In addition, based on data level, several task groups with lower data levels can be assigned to the same data acquisition unit. While ensuring the data acquisition unit's data acquisition pressure, using the same data acquisition unit to acquire the data to be processed for multiple task groups can reduce the number of data acquisition units used and reduce resource waste.

[0097] For example, to better coordinate resource usage, data is categorized into different levels based on the amount or volume of data identified, and different data quality inspection strategies are used for topics at different levels.

[0098] Data volume or data flow Data Level <40K / s small 40K / s~60K / s medium >60K / s Large

[0099] Optionally, the data element information includes a data identifier, and the at least one data acquisition unit 104 includes a first data acquisition unit and a second data acquisition unit;

[0100] The visualization processing unit 102 is further configured to divide the quality inspection tasks with the same data identifier into the same task group, and query the data level corresponding to the data identifier of each task group; assign the first task group whose data level is higher than a set threshold to the first data acquisition unit, and assign at least one second task group whose data level is lower than or equal to the set threshold to the second data acquisition unit.

[0101] The first data acquisition unit is used to acquire data to be processed according to the data identifier corresponding to the first task group; and to process the data to be processed according to the quality detection rules of each quality detection task in the first task group to generate the verification data of each quality detection task in the first task group.

[0102] The second data acquisition unit is used to acquire the data to be processed corresponding to each second task group according to the data identifier corresponding to each second task group.

[0103] Specifically, setting a threshold refers to setting a data level.

[0104] In practical applications, the visualization processing unit can group quality inspection tasks with the same data identifier into the same task group. Then, based on the data identifier corresponding to each task group, it queries the data volume or data flow corresponding to the data identifier, i.e., the data volume or data flow of the data to be inspected. The data level corresponding to this data volume or data flow is then used as the data level for that task group. Furthermore, the first task group with a data level higher than a set threshold is assigned to the first data acquisition unit, meaning each first data acquisition unit is assigned one first task group; at least one second task group with a data level lower than or equal to the set threshold is assigned to the second data acquisition unit, meaning multiple second task groups are assigned to the second data acquisition unit.

[0105] Furthermore, for any first data acquisition unit, the first data acquisition unit acquires the data to be processed according to the data identifier corresponding to the first task group it receives, and then processes the data to be processed according to the quality detection rules of each quality detection task in the first task group to generate the data to be verified for each quality detection task in the first task group.

[0106] For any second data acquisition unit, for each second task group it receives, the second data acquisition unit acquires the data to be processed corresponding to the second task group according to the data identifier corresponding to the second task group.

[0107] Therefore, by assigning at least one second task group with a data level lower than or equal to a set threshold to the same second data acquisition unit, and using the same data acquisition unit to acquire the data to be processed for multiple task groups, the number of data acquisition units used can be reduced, thus minimizing resource waste. Assigning a first task group with a data level higher than the set threshold to the second data acquisition unit can prevent the second data acquisition unit from experiencing excessive data processing load, which could cause it to crash.

[0108] Optionally, the system further includes multiple databases 106 and proxy service units;

[0109] The data acquisition unit 104 is used to generate a write request and send it to the proxy service unit. When the data acquisition unit 104 is a first data acquisition unit, the write request carries data to be verified. When the data acquisition unit 104 is a second data acquisition unit, the write request carries data to be processed.

[0110] The proxy service unit is used to determine the target database from the plurality of databases 106 according to the write request and the set routing algorithm, and forward the write request to the target database;

[0111] The target database is used to store the data to be verified when the write request carries data to be verified, or to process the data to be processed according to the data processing rules corresponding to the data to be processed sent by the visualization processing unit 102 when the write request carries data to be processed, thereby generating and storing the data to be verified.

[0112] Specifically, the proxy service unit refers to the functional module of the proxy service, which acts as the gateway for all databases and provides management services.

[0113] In practical applications, after generating the data to be tested, the first data acquisition unit generates a write request for the data to be tested based on the data to be tested and sends the write request to the proxy service unit; after the second data acquisition unit obtains the data to be processed, it generates a write request for the data to be processed based on the data to be processed and sends the write request to the proxy service unit.

[0114] Furthermore, after receiving a write request, the proxy service unit will match the target database from multiple databases based on the write request and the set routing algorithm, and then forward the write request to the target database.

[0115] Then, if a write request for the data to be verified is received, the target database can directly store the data to be verified. If a write request for the data to be processed is received, the target database will process the data to be processed according to the data processing rules corresponding to the data to be processed sent by the visualization processing unit, generate the data to be verified corresponding to the data to be processed, and store it.

[0116] Thus, for high-level data quality inspection tasks, due to the large amount or flow of data to be processed, in order to avoid reaching the database write bottleneck (i.e., the data volume is too large and cannot be written to the database), the amount or flow of the data to be verified after processing is smaller, which can be easily written to the database, thus improving the writing efficiency and thereby improving the data quality inspection efficiency.

[0117] Optionally, the write request also carries data element information; the system also includes multiple database clusters, each database cluster including multiple databases 106 and a database management unit;

[0118] The proxy service unit is also used to determine the target database cluster from the multiple database clusters based on the set routing algorithm, the load of each database cluster and the data element information carried by the write request, and forward the write request to the database management unit in the target database cluster.

[0119] The database management unit is configured to determine the target database from the multiple databases 106 included in the target database set based on the data element information carried in the write request; and forward the write request to the target database.

[0120] Specifically, a data management unit refers to the functional unit in a database cluster that manages the databases. Load refers to the amount of data stored or the amount of remaining space.

[0121] In practical applications, after receiving a write request, the proxy service unit determines the target database cluster to receive the write request from the multiple database clusters based on the data level corresponding to the data element information carried in the write request, the set routing algorithm, and the load of each database cluster. Then, it forwards the write request to the target database cluster, where the database management unit receives the write request.

[0122] Furthermore, the database management unit extracts the data metadata carried in the write request, which includes setting database information (such as dqc2_ori) and setting table identifier. The setting table identifier can be obtained by converting the data identifier according to the set conversion rules. Then, the setting database information and setting table identifier are hashed to determine the target database, and the write request is forwarded to the target database, where the target database processes the data in the write request.

[0123] In this way, the proxy service unit determines the target database cluster based on the configured routing algorithm, load, and data metadata, and the database management unit determines the target database based on the data metadata, which can improve the accuracy of determining the target data. Furthermore, it can improve data security and cluster usability.

[0124] Optionally, the visualization processing unit 102 is further configured to send the quality inspection rules of the second quality inspection task to the database, wherein the second quality inspection task is any quality inspection task in any second task group;

[0125] The database 106 is also used to process the data to be processed corresponding to the second quality inspection task according to the quality inspection rules of the second quality inspection task, generate and store the data to be verified for the second quality inspection task, and the data to be processed corresponding to the second quality inspection task is sent by the second data acquisition unit.

[0126] In practical applications, the visualization processing unit sends the quality detection rules for each quality detection task in each second task group to the database. This can be done directly to the database, through a proxy service unit, or a combination of the proxy service unit and the database management unit. Upon receiving the quality detection rules for a specific quality detection task and the corresponding pending data from the second data acquisition unit, the database processes the pending data according to the quality detection rules, generating and storing the corresponding verification data. Because the data level of the quality detection tasks in the second task group is relatively low, there may be multiple second task groups assigned to the second data acquisition unit, resulting in a large number of quality detection tasks assigned to it. If the second data acquisition unit were to determine the verification data, it would not only need to acquire multiple sets of pending data but also determine the verification data for each quality detection task, increasing its data processing burden. Therefore, having the database determine the verification data for each quality detection task in the second task group reduces the data processing burden on the second data acquisition unit, prevents it from crashing, improves write efficiency, and ultimately improves data quality detection efficiency.

[0127] Optionally, the visualization processing unit 102 is further configured to receive a first change instruction for the quality inspection rules in the first task group, wherein the first change instruction carries the changed quality inspection rules; and send the changed quality inspection rules to the first data acquisition unit;

[0128] The first data acquisition unit is further configured to process the data to be processed according to the modified quality inspection rules, and generate the data to be verified for each quality inspection task in the first task group.

[0129] Specifically, the first change instruction can modify, delete, or add quality inspection rules. The changed quality inspection rules refer to all quality inspection rules in the first task group after the change. For example, if the first task group contains quality inspection rules 1-3, and the first change instruction is to delete quality inspection rule 1, then the changed quality inspection rules will include quality inspection rules 2 and 3.

[0130] In practical applications, users modify the quality inspection rules in the first task group through the visualization processing unit, or the visualization processing unit periodically and automatically modifies the quality inspection rules in the first task group. Accordingly, the visualization processing unit receives a first change instruction for the quality inspection rules in the first task group. Then, the modified quality inspection rules are sent to the first data acquisition unit.

[0131] Accordingly, the first data acquisition unit receives the modified quality inspection rules through a listener; that is, if a change in the quality inspection rules is detected, the modified quality inspection rules are loaded and synchronized into memory. Based on the data identifiers in the data identifier list (Topic list) (the acquisition unit adds the data identifiers from the received quality inspection tasks to the data identifier list), the raw data stream is acquired. The raw data stream is parsed according to the data metadata in memory to obtain the data to be processed. Then, according to each quality inspection rule in the modified quality inspection rules, the data to be processed is processed separately to obtain the data to be verified corresponding to each quality inspection rule in the modified quality inspection rules. In this way, the quality inspection rules can be dynamically retrieved, enabling a single first data acquisition unit to acquire data for data quality tasks and generate data to be verified, thus improving efficiency.

[0132] Optionally, the visualization processing unit 102 is further configured to receive a second change instruction for data element information in any second task group, wherein the second change instruction carries the changed data element information; and send the changed data element information to the second data acquisition unit;

[0133] The second data acquisition unit is further configured to acquire the data to be processed corresponding to each second task group based on the data identifier contained in the modified data element information.

[0134] Specifically, the second change instruction can modify, delete, or add data element information, i.e., change the data to be processed. The changed data element information refers to all the changed data element information in the second task group after the quality inspection rules are changed.

[0135] In practical applications, users modify the data element information in the second task group through the visualization processing unit, or the visualization processing unit periodically and automatically modifies the data element information in the second task group. Accordingly, the visualization processing unit receives a first change instruction for the data element information in the second task group. Then, it sends the modified data element information to the second data acquisition unit.

[0136] Accordingly, the second data acquisition unit receives the changed data metadata information through a listener. This data metadata includes data identifiers; if a change in a data identifier is detected, the changed data metadata is loaded and synchronized into memory. The data identifier list (Topic list) is updated based on the changed data identifiers. Specifically, the acquisition unit adds the data identifiers received from the quality inspection tasks to the data identifier list. Based on the data identifiers in the list, the raw data stream is acquired. The raw data stream is then parsed according to the data metadata information in memory to obtain the data to be processed. This allows for dynamic retrieval of data identifiers, enabling a single second data acquisition unit to acquire multiple sets of data to be processed, thus improving quality inspection efficiency.

[0137] Optionally, the visualization processing unit 102 is further configured to generate a read request for the data to be verified corresponding to the first quality inspection task based on the data element information in the first quality inspection task; and send the read request to the proxy service unit.

[0138] The proxy service unit is further configured to determine the target database based on the read request, obtain the target data to be verified corresponding to the read request from the target database, and feed it back to the visualization processing unit.

[0139] In practical applications, the visualization processing unit can also generate a read request for the data to be inspected based on the data metadata in the first quality inspection task, and send the read request to the proxy service unit. Further, upon receiving a write request, the proxy service unit will match the target database from multiple databases based on the write request and a set routing algorithm, and then forward the write request to the target database. Then, the target database retrieves the data to be verified corresponding to the first quality inspection task according to the read request and feeds it back to the visualization processing unit through the proxy service unit.

[0140] It should be noted that after receiving a read request, the proxy service unit can determine the target database cluster to receive the read request from the multiple database clusters based on the data level corresponding to the data element information carried in the read request, the set routing algorithm, and the load of each database cluster. Then, it forwards the read request to the target database cluster, where the database management unit receives the read request.

[0141] Furthermore, to avoid redundant calculations and reduce data processing volume, the proxy service unit, upon first receiving a write request for data (data to be verified or processed) corresponding to a specific data element or data identifier, records the target database cluster after determining it. Subsequent write or read requests for the same data element or data identifier will directly determine the target cluster based on the record, without requiring further calculations.

[0142] See Figure 2A , Figure 2A Here is a flowchart of a data quality detection method provided in one embodiment of this application:

[0143] 1. Add a quality inspection task: Users add a data quality inspection task through DQCORE (visual processing unit). 2. Assign task: DQCORE dynamically assigns the data quality inspection task to designated consumer tasks (data acquisition units) for processing based on the data metadata in the task. 3. Consumption: Small and medium-sized consumer tasks (second data acquisition unit) retrieve data to be processed from the Kafka Cluster based on the data metadata; large consumer tasks (first data acquisition unit) retrieve data to be processed from the Kafka Cluster based on the data metadata and process the data according to the data inspection rules in the data quality inspection task to obtain data to be verified. 4. Write data: Small and medium-sized consumer tasks send write requests for the data to be processed to InfluxdbProxyCluster (proxy service unit); large consumer tasks send write requests for the data to be verified to InfluxdbProxyCluster. 5. Read / Write: InfluxdbProxyCluster writes the data to be verified or processed to InfluxDB; or, when DQCORE reads the data to be verified, InfluxdbProxyCluster reads the data to be verified from InfluxDB based on the received read request. 6. InfluxDB processes the data to be processed using the CQ corresponding to the data quality check rules, such as data aggregation, to obtain the data to be verified and store it. 7. Read the data to be verified: DQCORE reads the data to be verified from the database. 8. Quality check alarm: DQCORE determines the data quality check result based on the data to be verified and the quality check rules, and issues an alarm.

[0144] exist Figure 2A Based on this, see Figure 2B , Figure 2B This is a flowchart illustrating the processing performed by the second data acquisition unit in a data quality detection method provided in an embodiment of this application:

[0145] 1. Consumption: For task groups with data levels below or equal to a set threshold (i.e., small to medium-sized topics), a merged consumption approach is adopted. A single small to medium-sized consumption task (second data acquisition unit) consumes multiple topics (e.g., Topic 1-Topic 6). This involves retrieving data to be processed from the Kafka Cluster based on data metadata, thus reusing resources. 2. Writing data to be processed 3. Writing: Due to the relatively small volume of this data to be processed, it is written directly to the original data ORI database in InfluxDB via InfluxdbProxyCluster. Simultaneously, quality control rules are pushed down to InfluxDB. CQ (Data Quality Queuing) is used to process the data to be processed according to these rules, obtaining the data to be inspected and storing it in the aggregation result CQ database. Subsequent additions, modifications, and deletions of data quality rules only require synchronous operations on the CQ in InfluxDB, achieving lightweight operation without affecting the consumption tasks. The TopicListener is used to monitor changes in multiple topics.

[0146] exist Figure 2A Based on this, see Figure 2C , Figure 2C This is a flowchart illustrating the processing performed by the first data acquisition unit in a data quality detection method provided in an embodiment of this application:

[0147] 1. Consumption: For task groups with data levels below or equal to a set threshold (i.e., large topics), merging consumption could lead to significant data backlog and false positives in quality checks due to the large data volume. Therefore, a single-task, single-topic approach is adopted. One large consumption task (the first data acquisition unit) consumes one topic, retrieving data to be processed from the Kafka Cluster based on data metadata. 2. Writing data to be verified. 3. Writing: To avoid reaching the InfluxDB write bottleneck, after acquiring the data to be processed, the large consumption task processes the data according to multiple data quality rules in the task group (i.e., the rule list), resulting in data to be verified. This data to be verified does not need to pass through the original ORI database; it is directly stored in the InfluxDB aggregation result CQ database. Within a single consumption task, data to be processed according to multiple quality check rules is processed, achieving resource reuse.

[0148] In small to medium-sized consumption tasks, a single consumption task consumes multiple topics; in large-scale consumption tasks, a single consumption task processes multiple quality inspection rules. These two consumption methods achieve resource reuse and improve resource utilization. Since consumption tasks need to request resources from the cluster (Kubernetes) to start, resource contention is inevitable in resource-constrained clusters. Failure to meet this contention can lead to task startup failure. Therefore, when adding or removing topics or quality inspection rules that a consumption task needs to consume, it can do so dynamically, avoiding the need to restart the task every time, reducing resource contention, and achieving a "start once, no more restarts" scenario.

[0149] exist Figure 2A Based on this, see Figure 2D , Figure 2D This is a flowchart illustrating the data acquisition unit acquiring data in a data quality detection method according to an embodiment of this application:

[0150] When data identifiers or quality control rules are added or removed (e.g., different task groups correspond to different data identifiers, and data identifiers can also serve as topics), the listener embedded in the consumer task will detect the change (Topic / RuleListener). 1. Refresh Metadata: Upon detecting a change, the Listener first loads the latest Topic / RuleMeta bytecode file using the Java class loader and refreshes it into the task's memory. 2. Notify of Partition Changes: If a Topic change is detected, after refreshing it into the task's memory, it notifies the partition discoverer, DynamicPartitionDiscoverer. 3. Refresh the Topic List: DynamicPartitionDiscoverer updates the list of Topics to be pulled. 4. Pull Data: With the updated Topic list, DynamicKafkaConsumer can pull the latest raw data stream from the KafkaCluster. 5. Send Data: DynamicKafkaConsumer submits the obtained raw data stream to the data processing process for processing. 6. Retrieve corresponding Topic / Rule information and 7. Write data: For small to medium-sized consumption tasks, the data processing process reads the TopicMeta information in memory, parses the raw data stream according to the TopicMeta information, and writes the obtained data to be processed to InfluxDB through InfluxdbProxyCluster; For large consumption tasks, the data processing process reads all Meta information in memory, first parses the raw data stream according to the TopicMeta information to obtain the data to be processed, and at the same time processes the data to be processed according to the RuleMeta information, and writes the obtained data to be verified to InfluxDB through InfluxdbProxy Cluster.

[0151] exist Figure 2A Based on this, see Figure 2E , Figure 2EThis is a flowchart illustrating the process of a proxy service unit determining a target database cluster in a data quality detection method provided in one embodiment of this application: When writing data, if the number of task groups with data levels lower than or equal to a set threshold (i.e., too many small and medium-sized topics) is too large, in order to ensure that data is not lost and to also ensure the stability of downstream Influxdb, the proxy service unit (InfluxdbProxy Cluster) writes the data to be processed to a designated InfluxdbCluster based on the Topic size (data level), the load of each database cluster (InfluxdbCluster, such as Influxdb Cluster1 and InfluxdbCluster2), and the set routing algorithm.

[0152] exist Figure 2A Based on this, see Figure 2F , Figure 2F This is a flowchart illustrating the process of a database management unit determining a target database in a data quality detection method according to an embodiment of this application:

[0153] To ensure data accuracy and stability, four databases (Influxdb) were deployed. At the same time, a database management unit (InfluxdbProxy) was developed and deployed to manage multiple Influxdb instances, realizing a high-availability Influxdb cluster, and featuring functions such as dynamically adding or removing InfluxDB instances and data synchronization.

[0154] For each database cluster, two InfluxdbCircles (InfluxdbCircle1 and InfluxdbCircle2) are configured. Each InfluxdbCircle maintains complete data and contains two databases. However, each Influxdb instance in an InfluxdbCircle only maintains a portion of the data. In other words, the complete data consists of Influxdb1 + Influxdb2 in the InfluxdbCircle.

[0155] Through the principle of consistent hashing, it is guaranteed that an InfluxdbCirc is a logical consistent hashing ring, and all Influxdb instances in an InfluxdbCirc correspond to physical nodes of the consistent hashing ring.

[0156] Upon receiving a write request: After receiving the write request, InfluxdbProxy uses a consistent hashing algorithm based on the db and measurement in the request to calculate which Influxdb in each InfluxdbCircle should the data in the write request be written to, and then forwards the request to that Influxdb. Furthermore, if that Influxdb goes down or experiences a network failure, InfluxdbProxy will write the data to a local file, and rewrite it after the instance recovers.

[0157] Upon receiving a read request: After receiving a write request, InfluxdbProxy uses a consistent hashing algorithm to calculate the corresponding Influxdb in all InfluxdbCircles based on the db and measurement in the request, and selects the Influxdb that is running normally and does not have a corresponding file to be written to the local file for data query.

[0158] Exemplarily, a data quality detection method provided in one embodiment of this application will be described, including the following steps:

[0159] Visualization Unit (DQCORE): Receives quality inspection tasks input by the user, wherein the quality inspection tasks carry quality inspection rules and data metadata (Topic metadata), including data identifiers (Topic names) and parsing strategies; groups quality inspection tasks with the same data identifier into the same task group, and queries the data level corresponding to the data identifier of each task group; queries the data level (Topic type) corresponding to each data identifier; assigns the first task group whose data level is higher than a set threshold to the first data acquisition unit, and assigns at least one second task group whose data level is lower than or equal to the set threshold to the second data acquisition unit.

[0160] The second data acquisition unit monitors whether data identifiers have changed (i.e., the visualization unit sends newly added data identifiers or indicates the deletion of data identifiers); if a change in data identifiers is detected, the updated data metadata is loaded and synchronized to memory; the data identifier list (Topic list) is updated based on the updated data metadata, where the data identifier list stores the data identifiers to be pulled; the raw data stream is obtained according to the data identifiers in the data identifier list; the raw data stream is parsed according to the data metadata in memory to obtain the data to be processed; the data to be processed is sent to the proxy service unit (InfluxdbProxyCluster), and the quality detection rules are also sent to the proxy service unit;

[0161] The first data acquisition unit monitors whether the quality inspection rules have changed (i.e., the visualization unit sends new quality inspection rules or instructs the deletion of quality inspection rules); if a change in the quality inspection rules is detected, the updated rule metadata is loaded and synchronized to memory; based on the data identifiers in the data identifier list (Topic list), the raw data stream is obtained; the raw data stream is parsed according to the data metadata in memory to obtain the data to be processed, and the data to be processed is processed according to the rule metadata to obtain the data to be inspected; the data to be inspected is sent to the proxy service unit (Influxdb ProxyCluster).

[0162] The InfluxdbProxyCluster service unit, upon receiving data to be processed and quality inspection rules, determines the target database cluster from among the database clusters based on the data level (Topic type or Topic size), the load of each database cluster (Influxdb Cluster), and the set routing algorithm. It then generates a write request based on the data to be processed and the quality inspection rules, and sends the write request to the target database cluster. Similarly, upon receiving data to be inspected, it determines the target database cluster from among the database clusters based on the data level, the load of each database cluster, and the set routing algorithm. It then generates a write request based on the data to be inspected and sends the write request to the target database cluster.

[0163] Target database cluster: Receive write requests through the database management unit (InfluxdbProxyCluster), and perform hash calculations based on the database information (db) and table name (measurement) carried in the write request to determine the target database (Influxdb); send the data to be processed and quality inspection rules carried in the write request to the target database, or send the data to be inspected carried in the write request to the target database.

[0164] Target Database: If data to be processed and quality inspection rules are received, the data to be processed is stored in the initial storage area (original data ORI database) of the target database, and the continuous query unit (CQ) processes the data to be processed based on the quality inspection rules to obtain the data to be inspected; the data to be inspected is then stored in the target storage area (aggregated result CQ database). If data to be inspected is received, it is stored in the target storage area.

[0165] Visualization Boundary Unit (DQCORE): Based on the quality inspection task, it periodically generates read requests to obtain the data to be inspected and sends the read requests to the proxy service unit.

[0166] The proxy service unit determines the target database cluster from among the database clusters based on the data level carried in the read request, the load of each database cluster (InfluxdbCluster), and the set routing algorithm, and forwards the read request to the target database cluster.

[0167] Target database cluster: Receives read requests through the database management unit and performs hash calculations based on the database information (db) and table name (measurement) carried in the read request to determine the target database (Influxdb); sends the data to be processed and quality inspection rules carried in the read request to the target database, or sends the data to be inspected carried in the write request to the target database.

[0168] Target Database: Based on the read request, retrieve the target data to be inspected from the target storage area and feed the target data to be inspected back to the visualization boundary unit (DQCORE).

[0169] Visualized Qualified Cell (DQCORE): Based on the quality inspection rules, the target data to be inspected is verified to obtain the quality inspection results; the quality inspection results are fed back to the user, i.e., quality inspection alarms.

[0170] In this way, a notification mechanism can detect changes in the data identifiers or quality inspection rules that need to be consumed, and perform hot updates of these rules. This enables dynamic additions and removals of pending data and data quality rule lists, allowing a single data acquisition unit to process multiple pending data sets or perform multiple data quality rule checks. This achieves resource reuse and improves resource utilization. Furthermore, since the data acquisition unit itself performs these tasks dynamically, restarts are unnecessary, avoiding resource contention and improving system availability. Each time the second data acquisition unit acquires pending data, it writes the data to the database, while the visualization processing unit pushes down the specific quality inspection rules to the database. This achieves the effect of "one consumption, multiple rules shared," preventing multiple data acquisition units from repeatedly acquiring pending data and further improving resource utilization.

[0171] The data quality inspection system provided in this application assigns multiple quality inspection tasks to at least one data acquisition unit, which then acquires multiple data to be processed or processes the data to be processed according to multiple data quality rules. This enables the reuse of computing resources, improves resource utilization, avoids re-competition for resources, and enhances system availability.

[0172] Figure 3 A flowchart of a data quality detection method according to an embodiment of this application is shown. Applied to a visualization processing unit, the method specifically includes the following steps:

[0173] Step 302: Receive multiple quality inspection tasks input by the user, wherein the quality inspection tasks include quality inspection rules and data element information.

[0174] Step 304: Based on the data element information of the multiple quality inspection tasks, assign the multiple quality inspection tasks to at least one data acquisition unit, wherein the data element information in the quality inspection tasks is used by the data acquisition unit to acquire data to be processed, the quality inspection rules in the quality inspection tasks are used to process the data to be processed to generate the data to be verified for the quality inspection tasks, and the data to be verified is stored in the database.

[0175] Step 306: Based on the data element information in the first quality inspection task, obtain the corresponding target data to be verified from the database, wherein the first quality inspection task is any one of the plurality of quality inspection tasks.

[0176] Step 308: Verify the target data to be verified according to the quality inspection rules in the first quality inspection task to obtain the data quality inspection result.

[0177] Optionally, the data metadata includes a data identifier;

[0178] The step of assigning the multiple quality inspection tasks to at least one data acquisition unit based on the data element information of the multiple quality inspection tasks includes:

[0179] The quality inspection tasks with the same data identifier are grouped into the same task group, and the data level corresponding to the data identifier of each task group is queried.

[0180] Each task group is assigned to at least one data acquisition unit based on the data level corresponding to each task group.

[0181] Optionally, assigning each task group to at least one data acquisition unit according to the data level corresponding to each task group includes:

[0182] The first task group with a data level higher than a set threshold is assigned to the first data acquisition unit. The data identifier corresponding to the first task group is used by the first data acquisition unit to acquire the data to be processed. The quality detection rules of each quality detection task in the first task group are used to process the data to be processed and generate the data to be verified for each quality detection task in the first task group.

[0183] At least one second task group whose data level is lower than or equal to the set threshold is assigned to the second data acquisition unit, wherein the data identifier corresponding to each second task group is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

[0184] Optionally, before retrieving the corresponding target data to be verified from the database based on the data metadata in the first quality inspection task, the method further includes:

[0185] The quality inspection rules for the second quality inspection task are sent to the database. The second quality inspection task is any quality inspection task in any second task group. The quality inspection rules for the second quality inspection task are used by the database to process the data to be processed corresponding to the second quality inspection task, generate and store the data to be verified for the second quality inspection task, and the data to be processed corresponding to the second quality inspection task is sent by the second data acquisition unit.

[0186] Optionally, the method further includes:

[0187] Receive a first change instruction for the quality inspection rules in the first task group, wherein the first change instruction carries the changed quality inspection rules;

[0188] The modified quality inspection rules are sent to the first data acquisition unit, wherein the modified quality inspection rules are used by the first data acquisition unit to process the data to be processed and generate the data to be verified for each quality inspection task in the first task group.

[0189] Optionally, the method further includes:

[0190] Receive a second change instruction for data element information in any second task group, wherein the second change instruction carries the changed data element information;

[0191] The modified data metadata is sent to the second data acquisition unit, wherein the data identifier contained in the modified data metadata is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

[0192] Optionally, obtaining the corresponding target data to be verified from the database based on the data element information in the first quality inspection task includes:

[0193] Based on the data element information in the first quality inspection task, a read request for the data to be verified corresponding to the first quality inspection task is generated.

[0194] The read request is sent to the proxy service unit, wherein the read request is used by the proxy service unit to determine the target database, obtain the target data to be verified corresponding to the read request from the target database, and feed it back to the visualization processing unit.

[0195] The data quality detection method provided in this application assigns multiple quality detection tasks to at least one data acquisition unit, which then acquires multiple data to be processed or checks the data to be processed according to multiple data quality rules before processing. This enables the reuse of computing resources, improves resource utilization, avoids re-competition for resources, and enhances system availability.

[0196] The above is an illustrative scheme of a data quality detection method according to this embodiment. It should be noted that the technical solution of this data quality detection method and the technical solution of the data quality detection system described above belong to the same concept. For details not described in detail in the technical solution of the data quality detection method, please refer to the description of the technical solution of the data quality detection system described above.

[0197] Corresponding to the above method embodiments, this application also provides embodiments of a data quality detection device. Figure 4 A schematic diagram of a data quality detection device according to an embodiment of this application is shown. Figure 4 As shown, the device is applied to a visualization processing unit and includes:

[0198] The receiving module 402 is configured to receive multiple quality detection tasks input by the user, wherein the quality detection tasks include quality detection rules and data element information;

[0199] The allocation module 404 is configured to allocate the multiple quality inspection tasks to at least one data acquisition unit based on the data element information of the multiple quality inspection tasks, wherein the data element information in the quality inspection tasks is used by the data acquisition unit to acquire data to be processed, the quality inspection rules in the quality inspection tasks are used to process the data to be processed to generate the data to be verified for the quality inspection tasks, and the data to be verified is stored in a database.

[0200] The acquisition module 406 is configured to acquire the corresponding target data to be verified from the database based on the data element information in the first quality detection task, wherein the first quality detection task is any one of the plurality of quality detection tasks;

[0201] The verification module 408 is configured to verify the target data to be verified according to the quality detection rules in the first quality detection task, and obtain the data quality detection result.

[0202] Optionally, the data metadata includes a data identifier;

[0203] The allocation module 404 is further configured to:

[0204] The quality inspection tasks with the same data identifier are grouped into the same task group, and the data level corresponding to the data identifier of each task group is queried.

[0205] Each task group is assigned to at least one data acquisition unit based on the data level corresponding to each task group.

[0206] Optionally, the allocation module 404 is further configured to:

[0207] The first task group with a data level higher than a set threshold is assigned to the first data acquisition unit. The data identifier corresponding to the first task group is used by the first data acquisition unit to acquire the data to be processed. The quality detection rules of each quality detection task in the first task group are used to process the data to be processed and generate the data to be verified for each quality detection task in the first task group.

[0208] At least one second task group whose data level is lower than or equal to the set threshold is assigned to the second data acquisition unit, wherein the data identifier corresponding to each second task group is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

[0209] Optionally, the device further includes a first transmitting module configured to:

[0210] The quality inspection rules for the second quality inspection task are sent to the database. The second quality inspection task is any quality inspection task in any second task group. The quality inspection rules for the second quality inspection task are used by the database to process the data to be processed corresponding to the second quality inspection task, generate and store the data to be verified for the second quality inspection task, and the data to be processed corresponding to the second quality inspection task is sent by the second data acquisition unit.

[0211] Optionally, the device further includes a second transmitting module configured to:

[0212] Receive a first change instruction for the quality inspection rules in the first task group, wherein the first change instruction carries the changed quality inspection rules;

[0213] The modified quality inspection rules are sent to the first data acquisition unit, wherein the modified quality inspection rules are used by the first data acquisition unit to process the data to be processed and generate the data to be verified for each quality inspection task in the first task group.

[0214] Optionally, the device further includes a third transmitting module configured to:

[0215] Receive a second change instruction for data element information in any second task group, wherein the second change instruction carries the changed data element information;

[0216] The modified data metadata is sent to the second data acquisition unit, wherein the data identifier contained in the modified data metadata is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

[0217] Optionally, the verification module 408 is further configured to:

[0218] Based on the data element information in the first quality inspection task, a read request for the data to be verified corresponding to the first quality inspection task is generated.

[0219] The read request is sent to the proxy service unit, wherein the read request is used by the proxy service unit to determine the target database, obtain the target data to be verified corresponding to the read request from the target database, and feed it back to the visualization processing unit.

[0220] The data quality detection device provided in this application assigns multiple quality detection tasks to at least one data acquisition unit, which acquires multiple data to be processed or checks the data to be processed according to multiple data quality rules before processing. This enables the reuse of computing resources, improves resource utilization, avoids re-competition for resources, and improves system availability.

[0221] The above is a schematic scheme of a data quality detection device according to this embodiment. It should be noted that the technical solution of this data quality detection device and the technical solution of the data quality detection method described above belong to the same concept. For details not described in detail in the technical solution of the data quality detection device, please refer to the description of the technical solution of the data quality detection method described above.

[0222] Figure 5 A structural block diagram of a computing device according to an embodiment of this application is shown. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0223] The computing device 500 also includes an access device 540, which enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0224] In one embodiment of this application, the aforementioned components of the computing device 500 and Figure 5 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 5 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0225] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 500 can also be a mobile or stationary server.

[0226] The processor 520 implements the data quality detection method when executing the computer instructions.

[0227] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data quality detection method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data quality detection method described above.

[0228] An embodiment of this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the data quality detection method described above.

[0229] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data quality detection method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the data quality detection method described above.

[0230] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0231] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0232] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0233] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0234] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A data quality inspection method, characterized in that, Applied to the visualization processing unit, including: Receive multiple quality inspection tasks input by the user, wherein the quality inspection tasks include quality inspection rules and data element information; Based on the data metadata of the multiple quality inspection tasks, the multiple quality inspection tasks are assigned to at least one data acquisition unit. The data metadata of the quality inspection tasks is used by the data acquisition unit to acquire data to be processed. The quality inspection rules of the quality inspection tasks are used to process the data to be processed to generate the data to be verified for the quality inspection tasks. The data to be verified is stored in a database. Based on the data element information in the first quality inspection task, the corresponding target data to be verified is obtained from the database, wherein the first quality inspection task is any one of the plurality of quality inspection tasks; According to the quality inspection rules in the first quality inspection task, the target data to be inspected is inspected to obtain the data quality inspection result; The data metadata includes a data identifier; the step of allocating the multiple quality inspection tasks to at least one data acquisition unit based on the data metadata of the multiple quality inspection tasks includes: The quality inspection tasks with the same data identifier are grouped into the same task group, and the data level corresponding to the data identifier of each task group is queried. Based on the data level corresponding to each task group, each task group is assigned to at least one data acquisition unit. The step of assigning each task group to at least one data acquisition unit according to the data level corresponding to each task group includes: The first task group with a data level higher than a set threshold is assigned to the first data acquisition unit. The data identifier corresponding to the first task group is used by the first data acquisition unit to acquire the data to be processed. The quality detection rules of each quality detection task in the first task group are used to process the data to be processed and generate the data to be verified for each quality detection task in the first task group. At least one second task group whose data level is lower than or equal to the set threshold is assigned to the second data acquisition unit, wherein the data identifier corresponding to each second task group is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

2. The method of claim 1, wherein, Before retrieving the corresponding target data to be verified from the database based on the data element information in the first quality inspection task, the process further includes: The quality inspection rules for the second quality inspection task are sent to the database. The second quality inspection task is any quality inspection task in any second task group. The quality inspection rules for the second quality inspection task are used by the database to process the data to be processed corresponding to the second quality inspection task, generate and store the data to be verified for the second quality inspection task, and the data to be processed corresponding to the second quality inspection task is sent by the second data acquisition unit.

3. The method of claim 1, wherein, The method further includes: Receive a first change instruction for the quality inspection rules in the first task group, wherein the first change instruction carries the changed quality inspection rules; The modified quality inspection rules are sent to the first data acquisition unit, wherein the modified quality inspection rules are used by the first data acquisition unit to process the data to be processed and generate the data to be verified for each quality inspection task in the first task group.

4. The method of claim 1, wherein, The method further includes: Receive a second change instruction for data element information in any second task group, wherein the second change instruction carries the changed data element information; The modified data metadata is sent to the second data acquisition unit, wherein the data identifier contained in the modified data metadata is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

5. The method according to any one of claims 1 to 4, characterized in that, The step of retrieving the corresponding target data to be verified from the database based on the data element information in the first quality inspection task includes: Based on the data element information in the first quality inspection task, a read request for the data to be verified corresponding to the first quality inspection task is generated. The read request is sent to the proxy service unit, wherein the read request is used by the proxy service unit to determine the target database, obtain the target data to be verified corresponding to the read request from the target database, and feed it back to the visualization processing unit.

6. A data quality detection system characterized by, include: A visualization processing unit is configured to receive multiple quality inspection tasks input by a user, wherein the quality inspection tasks include quality inspection rules and data element information; and to assign the multiple quality inspection tasks to at least one data acquisition unit according to the data element information of the multiple quality inspection tasks. The data acquisition unit is used to acquire data to be processed based on the data element information in the quality inspection task, wherein the quality inspection rules in the quality inspection task are used to process the data to be processed to generate the data to be verified for the quality inspection task, and the data to be verified is stored in the database. The visualization processing unit is further configured to retrieve the corresponding target data to be verified from the database based on the data element information in the first quality detection task, wherein the first quality detection task is any one of the plurality of quality detection tasks; and to verify the target data to be verified according to the quality detection rules in the first quality detection task to obtain the data quality detection result. The data metadata includes a data identifier; the step of allocating the multiple quality inspection tasks to at least one data acquisition unit based on the data metadata of the multiple quality inspection tasks includes: The quality inspection tasks with the same data identifier are grouped into the same task group, and the data level corresponding to the data identifier of each task group is queried. Based on the data level corresponding to each task group, each task group is assigned to at least one data acquisition unit. The step of assigning each task group to at least one data acquisition unit according to the data level corresponding to each task group includes: The first task group with a data level higher than a set threshold is assigned to the first data acquisition unit. The data identifier corresponding to the first task group is used by the first data acquisition unit to acquire the data to be processed. The quality detection rules of each quality detection task in the first task group are used to process the data to be processed and generate the data to be verified for each quality detection task in the first task group. At least one second task group whose data level is lower than or equal to the set threshold is assigned to the second data acquisition unit, wherein the data identifier corresponding to each second task group is used by the second data acquisition unit to acquire the data to be processed corresponding to each second task group.

7. The system of claim 6, wherein, The data metadata includes data identifiers and parsing strategies; The data acquisition unit is further configured to acquire an initial data stream based on the data identifier in the quality inspection task; and to parse the initial data stream according to the parsing strategy in the quality inspection task to obtain the data to be processed corresponding to the quality inspection task.

8. The system of claim 6 or 7, wherein, The data element information includes a data identifier, and the at least one data acquisition unit includes a first data acquisition unit and a second data acquisition unit; The visualization processing unit is further configured to divide the quality inspection tasks with the same data identifier into the same task group, and query the data level corresponding to the data identifier of each task group; assign the first task group with the data level higher than the set threshold to the first data acquisition unit, and assign at least one second task group with the data level lower than or equal to the set threshold to the second data acquisition unit. The first data acquisition unit is used to acquire data to be processed according to the data identifier corresponding to the first task group; According to the quality inspection rules of each quality inspection task in the first task group, the data to be processed is processed respectively to generate the verification data of each quality inspection task in the first task group. The second data acquisition unit is used to acquire the data to be processed corresponding to each second task group according to the data identifier corresponding to each second task group.

9. The system of claim 8, wherein, The system also includes multiple databases and agent service units; The data acquisition unit is used to generate a write request and send it to the proxy service unit. When the data acquisition unit is a first data acquisition unit, the write request carries data to be verified. When the data acquisition unit is a second data acquisition unit, the write request carries data to be processed. The proxy service unit is used to determine the target database from the plurality of databases according to the write request and the set routing algorithm, and forward the write request to the target database; The target database is used to store the data to be verified when the write request carries data to be verified, or to process the data to be processed according to the data processing rules corresponding to the data to be processed sent by the visualization processing unit when the write request carries data to be processed, thereby generating and storing the data to be verified.

10. The system of claim 9, wherein, The write request also carries data element information; the system also includes multiple database clusters, each database cluster including multiple databases and database management units; The proxy service unit is also used to determine the target database cluster from the multiple database clusters based on the set routing algorithm, the load of each database cluster and the data element information carried by the write request, and forward the write request to the database management unit in the target database cluster. The database management unit is configured to determine the target database from among the multiple databases included in the target database set based on the data element information carried in the write request; and forward the write request to the target database.

11. A computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein, When the processor executes the computer instructions, it implements the steps of the method according to any one of claims 1-5.

12. A computer-readable storage medium storing computer instructions, wherein, When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1-5.

Images