Data processing method, medium, apparatus, and computing device

Abstract

Embodiments of the present disclosure provide a data processing method, relating to the field of data processing. The data processing method comprises: in response to receiving a trigger instruction for executing a data quality monitoring task of to-be-monitored data, executing the data quality monitoring task, and generating M first query statements; extracting a data source of each of the M first query statements; based on the data source, performing merging processing on the M first query statements to obtain a second query statement; and executing the second query statement to obtain an execution result, the execution result being used to determine the data quality of the to-be-monitored data. By merging the first query statements of the same data source to obtain the second query statement, the data quality of the monitoring data can be determined by executing the second query statement once, thereby significantly improving the efficiency of data quality monitoring and bringing a better experience for the user. In addition, the embodiments of the present disclosure provide a medium, an apparatus and a computing device.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of data processing, and more specifically, the embodiments of this disclosure relate to data processing methods, media, apparatus, and computing devices. Background Technology

[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] Data quality is the foundation, and the most important prerequisite and guarantee, for the validity and accuracy of data analysis conclusions. By defining data quality monitoring rules in advance, monitoring and controlling the data generation process during the process, and conducting data evaluation and problem tracing afterward, high-quality data can be controlled. Having high-quality data ensures that business decisions are not negatively affected by "bad" or "dirty" data.

[0004] In related technologies, data quality monitoring is typically achieved by executing queries to assess data quality and obtaining the results, which can then be used to determine the quality of the data to be monitored. However, the larger the volume of data to be monitored, the more queries are required, thus increasing the execution time and resulting in low efficiency in data quality monitoring.

[0005] Therefore, there is an urgent need for a solution that can improve the efficiency of data quality monitoring. Summary of the Invention

[0006] This disclosure provides a data processing method, medium, apparatus, and computing device to improve the efficiency of data quality monitoring.

[0007] In a first aspect of this disclosure, a data processing method is provided, comprising: responding to receiving a trigger instruction for performing a data quality monitoring task for data to be monitored, executing the data quality monitoring task, generating M first query statements, wherein the data quality monitoring task is used to generate query statements for monitoring the data quality of the data to be monitored, and M is a positive integer; extracting the data source for each of the M first query statements; merging the M first query statements based on the data source to obtain a second query statement; executing the second query statement to obtain an execution result, wherein the execution result is used to determine the data quality of the data to be monitored.

[0008] In one embodiment of this disclosure, the data processing method is applied to a server. Based on a data source, M first query statements are merged to obtain a second query statement. The method includes: classifying the M first query statements based on the data source to obtain N sets of first query statements, wherein the data sources of different first query statement sets in the N sets are different, N is less than M and N is a positive integer; and merging T first query statements in each of the N sets of first query statements to obtain a second query statement, wherein T is less than M and T is a positive integer.

[0009] In another embodiment of this disclosure, the T first query statements in the first query statement set are merged to obtain a second query statement, including: extracting the query conditions of the T first query statements, or extracting the query conditions and filtering conditions of the T first query statements; in response to the fact that none of the T first query statements have filtering conditions, the query conditions of the T first query statements are merged to obtain a second query statement.

[0010] In another embodiment of this disclosure, merging T first query statements in a first query statement set to obtain a second query statement includes: 1) merging the query conditions of the T first query statements, given that all T first query statements have filtering conditions and the filtering conditions are identical, to obtain a second query statement; 2) merging the query conditions of the T first query statements, given that all T first query statements have filtering conditions and the filtering conditions are different, using a preset branch function for each of the T first query statements, merging the filtering conditions of each first query statement into the query conditions of each first query statement, to obtain T third query statements; 3) merging the query conditions of the T third query statements to obtain a second query statement; and 4) merging the query conditions of the T third query statements, given that all T first query statements have filtering conditions and some of the T first query statements have different filtering conditions. Given the same filtering conditions, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement, resulting in T fourth query statements. The query conditions of the T fourth query statements are then merged to obtain a second query statement. Since all T first query statements have filtering conditions, and the filtering conditions of the T first query statements contain the same target sub-filtering condition, for each of the T first query statements, a preset branch function is used to merge the other sub-filtering conditions (excluding the target sub-filtering condition) of each first query statement into the query conditions of each first query statement, resulting in T fifth query statements. The query conditions of the T fifth query statements are then merged to obtain a second query statement.

[0011] In another embodiment of this disclosure, the T first query statements in the first query statement set are merged to obtain a second query statement, including: in response to the existence of filtering conditions in S first query statements among the T first query statements, for each of the S first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain S sixth query statements, where S < T and S is a positive integer; the query conditions of the T first query statements and the query conditions of the S sixth query statements are merged to obtain the second query statement.

[0012] In another embodiment of this disclosure, extracting the data source for each of the M first query statements includes: for each of the M first query statements, performing the following steps: parsing the first query statement to extract multiple query variables of the first query statement; and determining the data source based on the meaning of the variables represented by the multiple query variables.

[0013] In another embodiment of this disclosure, the data quality monitoring task includes M data quality monitoring rules, each of the M data quality monitoring rules includes a rule statement, and each of the M data quality monitoring rules corresponds one-to-one with each of the M first query statements; executing the data quality monitoring task to generate M first query statements includes: assigning values ​​to the target query variables in the M rule statements corresponding to the M data quality monitoring rules according to preset variable assignment rules to obtain M first query statements.

[0014] In another embodiment of this disclosure, after obtaining the execution result, the method further includes: splitting the execution result into multiple sub-results based on the fields in the execution result that correspond to the data quality monitoring rules, with each sub-result corresponding to a different data quality monitoring rule.

[0015] In another embodiment of this disclosure, each of the M data quality monitoring rules further includes the expected result corresponding to the rule statement; after obtaining multiple sub-results, the method further includes: verifying the multiple sub-results according to the expected result to determine the data quality of the data to be monitored.

[0016] In another embodiment of this disclosure, the method further includes: executing a data quality monitoring task, generating configuration information, the configuration information further including at least one of the following: snapshot information of the data quality monitoring task, start time and end time of executing the data quality monitoring task, triggering method of executing the data quality monitoring task, execution parameters of executing the data quality monitoring task, execution queue of executing the data quality monitoring task, computing resources and storage resources consumed when executing the data quality monitoring task; executing a second query statement to obtain an execution result, including: executing a second query statement based on the configuration information to obtain an execution result.

[0017] In another embodiment of this disclosure, the data quality monitoring task further includes at least one of the following: data source information of the data to be monitored, task information of the data quality monitoring task, and resource information required to perform the data quality monitoring task.

[0018] In another embodiment of this disclosure, the data processing method is applied to a client, which has a computing engine deployed for running query statements. Executing a second query statement and obtaining an execution result includes: using the computing engine to execute the second query statement and obtaining an execution result.

[0019] In a second aspect of the present disclosure, a medium is provided that stores computer execution instructions, which, when executed, are used to implement the data processing method of the first aspect.

[0020] In a third aspect of this disclosure, a data processing apparatus is provided, comprising: a generation module, configured to, in response to receiving a trigger instruction for performing a data quality monitoring task for data to be monitored, execute the data quality monitoring task and generate M first query statements, wherein the data quality monitoring task is used to generate query statements for monitoring the data quality of the data to be monitored, and M is a positive integer; an extraction module, configured to extract the data source of each of the M first query statements; a merging module, configured to, based on the data source, merge the M first query statements to obtain a second query statement; and an execution module, configured to execute the second query statement and obtain an execution result, wherein the execution result is used to determine the data quality of the data to be monitored.

[0021] In a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the data processing method as described in the first aspect.

[0022] According to the data processing method, medium, apparatus, and computing device of this disclosure, by executing a data quality monitoring task of the data to be monitored, multiple first query statements for monitoring the data quality of the data to be monitored can be generated. In order to reduce the number of times the query statements are executed, the data sources of the multiple first query statements can be determined, and the multiple first query statements can be merged according to the data sources to obtain a second query statement. In this way, the data quality of the data to be monitored can be determined by executing the second query statement. Therefore, it is not necessary to execute multiple first query statements one by one, thereby reducing the number of times the query statements are executed, and thus significantly improving the efficiency of data quality monitoring, bringing a better experience to the user. Attached Figure Description

[0023] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0024] Figure 1 A schematic diagram illustrating an application scenario of the data processing method provided in this embodiment;

[0025] Figure 2 A flowchart illustrating the data processing method provided in this embodiment of the disclosure;

[0026] Figure 3 A schematic diagram illustrating the merging of the first query statements provided in this disclosure;

[0027] Figure 4 This is a schematic diagram illustrating the execution of the second query statement provided in this disclosure;

[0028] Figure 5 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present disclosure;

[0029] Figure 6 This is a schematic diagram of the structure of the data processing apparatus according to an embodiment of the present disclosure;

[0030] Figure 7 This is a schematic diagram of the structure of a computing device according to an embodiment of the present disclosure.

[0031] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0032] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0033] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0034] According to embodiments of this disclosure, a data processing method, medium, apparatus, and computing device are proposed.

[0035] In this article, it is important to understand that the terms used have the following meanings:

[0036] Structured Query Language (SQL) is a database query and programming language used to access, query, update, and manage relational database systems.

[0037] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0038] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0039] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0041] The inventors have discovered that in related technologies, data quality monitoring typically involves executing a query statement to monitor the data quality of the data to be monitored, obtaining the execution result, which can then be used to determine the data quality of the monitored data. Specifically, when executing a query statement, it is necessary to first read the monitoring table data corresponding to the query statement, then filter it according to the WHERE condition in the query statement, and finally aggregate it according to the SELECT statement to obtain the execution result. However, the larger the amount of data to be monitored, the more queries are required. During the execution of the queries, the monitoring table data needs to be read multiple times, thus increasing the time consumed in executing the queries and resulting in low efficiency in data quality monitoring.

[0042] Therefore, this disclosure provides a data processing method. A data quality monitoring task includes multiple data quality monitoring rules. By executing the data quality monitoring task, multiple first query statements can be generated to monitor the data quality of the data to be monitored, with each first query statement corresponding to a data quality monitoring rule. However, directly executing multiple first query statements, while monitoring the data quality of the data to be monitored, requires executing each first query statement one by one, resulting in a long execution time and consequently, low efficiency in monitoring the data quality of the data to be monitored. Therefore, to reduce the execution time of the first query statements and improve the efficiency of monitoring the data quality of the data to be monitored, the data sources of multiple first query statements can be extracted. Then, first query statements with the same data source can be merged to obtain a second query statement. This transforms the execution of multiple first query statements into the execution of a single second query statement, requiring only one reading of the monitoring table data, thus reducing the execution time of the query statements and improving the monitoring efficiency of the data quality of the data to be monitored.

[0043] Furthermore, after executing the second query statement and obtaining the execution result, since the execution result is a merged result, it is necessary to split the execution result according to multiple data quality monitoring rules. This facilitates the verification of the corresponding split results according to each data quality monitoring rule, ensuring the accuracy of monitoring the data quality of the data to be monitored.

[0044] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0045] Application Scenarios Overview

[0046] First refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method provided in this disclosure. The data processing method provided in this disclosure can be applied to... Figure 1The data processing system shown may include a server and a client. The server includes a data quality monitoring task execution module, a query parsing module, a query merging module, an execution result splitting module, and an execution result verification module. The client includes a query execution module. Optionally, the server and client may be deployed on a single device or on different devices.

[0047] In the above application scenario, when the data quality monitoring task execution module on the server detects a trigger instruction for executing a data quality monitoring task for the data to be monitored, it executes a data quality monitoring task containing multiple data quality monitoring rules, generates multiple first query statements corresponding to the multiple data quality monitoring rules for monitoring the data quality of the data to be monitored, and sends the generated multiple first query statements to the query statement parsing module; after receiving the multiple first query statements, the query statement parsing module parses out the data sources of the multiple first query statements, and sends the multiple first query statements and the parsed data sources to the query statement merging module; after receiving the multiple first query statements and data sources, the query statement merging module merges the first query statements with the same data source to obtain a second query statement, and sends the second query statement to the query statement execution module on the client.

[0048] In the above application scenario, after receiving the second query statement, the query statement execution module in the client executes the second query statement. That is, it reads the corresponding data from the data source corresponding to the second query statement, obtains the execution result, and returns the execution result to the execution result splitting module in the server. Since only the second query statement is executed, and the second query statement is obtained by merging the first query statement, the second query statement is also used to monitor the data quality of the data to be monitored. This reduces the number of times the query statement is executed, thereby improving the execution efficiency of the query statement and further improving the efficiency of monitoring the data quality of the data to be monitored.

[0049] In the above application scenario, after obtaining the execution result, the execution result splitting module on the server side splits the execution result according to multiple data quality monitoring rules to obtain multiple sub-results, and sends multiple sub-results to the execution result verification module. Each sub-result can represent the result of quality monitoring of the data to be monitored according to the corresponding data quality monitoring rules. After receiving multiple sub-results, the execution result verification module verifies the corresponding sub-results according to multiple data quality monitoring rules to ensure the accuracy of the sub-results, thereby also ensuring the accuracy of the result of quality monitoring of the data to be monitored.

[0050] Exemplary methods

[0051] The following is combined Figure 1Application scenarios, refer to Figure 2 This document describes a data processing method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0052] Figure 2 This is a flowchart illustrating the data processing method provided in this embodiment of the disclosure. Figure 2 The data processing method includes the following steps:

[0053] S201: In response to receiving a trigger instruction for performing a data quality monitoring task on the data to be monitored, execute the data quality monitoring task and generate M first query statements, where M is a positive integer.

[0054] In this step, the data quality monitoring task is used to generate query statements to monitor the data quality of the data to be monitored. Therefore, by executing the data quality monitoring task of the data to be monitored, the generated M first query statements can be used to monitor the data quality of the data to be monitored.

[0055] Optionally, the trigger command can be generated by manually executing the data quality monitoring task, automatically generated by periodically triggering the execution of the data quality monitoring task, or automatically generated when the data quality monitoring task is generated.

[0056] S202: Extract the data source for each of the M first query statements.

[0057] In this step, the data source of the query statement can refer to the table where the data obtained when the query statement is executed. The data source can be represented in the query statement by the table name or a subquery.

[0058] In one alternative implementation, when executing a query statement, data needs to be retrieved from a data source. If M first query statements are executed, the data source needs to be read M times, resulting in a high number of executions and low efficiency. However, the same data source can be read only once to retrieve all the required data. Therefore, the data source of each of the M first query statements can be extracted to facilitate subsequent merging of the M first query statements based on the data source.

[0059] S203: Based on the data source, merge the M first query statements to obtain the second query statement.

[0060] In this step, the M first query statements can have at least one data source. Therefore, the first query statements with the same data source can be merged to obtain the second query statement.

[0061] Optionally, the second query statement is obtained by merging the first query statement; therefore, the number of second query statements is less than the number of first query statements. Merging the M first query statements is unnecessary only if the data sources of the M first query statements are all different.

[0062] S204: Execute the second query statement and obtain the execution result.

[0063] In this step, the execution results are used to determine the data quality of the data to be monitored.

[0064] Specifically, the first query statement is used to monitor the data quality of the data to be monitored, and the second query statement is obtained by merging the first query statement. Therefore, by executing the second query statement, the data quality of the data to be monitored can also be monitored. Furthermore, since the second query statement has fewer elements than the first query statement, the number of executions required for the second query statement is also less than the number of executions required for the first query statement, thus improving the efficiency of monitoring the data quality of the data to be monitored.

[0065] The data processing method disclosed herein generates multiple first query statements for monitoring the data quality of the data to be monitored by executing a data quality monitoring task. In order to reduce the number of times the query statements are executed, the data sources of the multiple first query statements can be determined, and the multiple first query statements are merged according to the data sources to obtain a second query statement. In this way, the data quality of the data to be monitored can be determined by executing the second query statement. Therefore, it is not necessary to execute multiple first query statements one by one, thereby reducing the number of times the query statements are executed and significantly improving the efficiency of data quality monitoring, thus bringing a better experience to users.

[0066] In one embodiment of this disclosure, the data processing method is applied to a server. Based on a data source, M first query statements are merged to obtain a second query statement. The method includes: classifying the M first query statements based on the data source to obtain N sets of first query statements, wherein the data sources of different first query statement sets in the N sets are different, N is less than M and N is a positive integer; and merging T first query statements in each of the N sets of first query statements to obtain a second query statement, wherein T is less than M and T is a positive integer.

[0067] In this embodiment, after obtaining the data source, the M first query statements can be classified according to the enumeration type of the data source. This will give us the first query statement corresponding to each enumeration type. By combining the first query statements corresponding to each enumeration type into a first query statement set, we can get N first query statement sets. Each of these N first query statement sets corresponds to an enumeration type. In other words, the data sources of different first query statement sets in the N first query statement sets are different.

[0068] In the above embodiments, after obtaining N sets of first query statements, the first query statements included in each set can be merged to obtain N second query statements. The number of first query statements contained in different first query statements can be different or the same.

[0069] Optionally, since N is less than M, meaning the number of second query statements is less than the number of first query statements, the number of executions required to execute the second query statement is less than the number of executions required to execute the first query statement, thereby improving execution efficiency. In this case, the monitoring efficiency of the data quality of the data to be monitored can be improved by executing the second query statement.

[0070] Optionally, if the number of second query statements is equal to the number of first query statements, that is, the data sources of the M first query statements are all different, then merging the first query statements can be disregarded, and the data quality of the data to be monitored can be monitored by executing each first query statement normally.

[0071] In another embodiment of this disclosure, the T first query statements in the first query statement set are merged to obtain a second query statement, including: extracting the query conditions of the T first query statements, or extracting the query conditions and filtering conditions of the T first query statements; in response to the fact that none of the T first query statements have filtering conditions, the query conditions of the T first query statements are merged to obtain a second query statement.

[0072] In this embodiment, a query statement can contain three parts: a select part, a from part, and a where part. The from part represents the data source, which can be a table name or a subquery. The select part represents the query conditions, which can be fields or aggregate values ​​from the table. The where part represents the filtering conditions. A query statement can contain all three parts, or it can contain only the select and from parts. Therefore, the select and where parts of the first query statement can be extracted. If the first query statement does not contain a where part, only the select part can be extracted.

[0073] In the above embodiments, for each of the N sets of first query statements, when merging the T first query statements in the first query statement set, there may be a situation where none of the T first query statements have filtering conditions. In this case, the query conditions of the T first query statements can be directly merged to obtain a second query statement. This eliminates the need to execute the T first query statements T times, and only requires executing the second query statement once to monitor the data quality of the data to be monitored, thereby greatly improving the efficiency of monitoring the data quality of the data to be monitored.

[0074] For example, consider the following two first query statements: `select count(id)as rule1 from students` and `select max(age)as rule 2 from students`. Neither of these first query statements contains a `where` clause. Therefore, we can directly merge `select count(id)as rule1` and `select max(age)as rule 2` to obtain the second query statement: `select count(id)as rule1, max(age)as rule 2 from students`.

[0075] In another embodiment of this disclosure, merging T first query statements in a first query statement set to obtain a second query statement includes: 1) merging the query conditions of the T first query statements, given that all T first query statements have filtering conditions and the filtering conditions are identical, to obtain a second query statement; 2) merging the query conditions of the T first query statements, given that all T first query statements have filtering conditions and the filtering conditions are different, using a preset branch function for each of the T first query statements, merging the filtering conditions of each first query statement into the query conditions of each first query statement, to obtain T third query statements; 3) merging the query conditions of the T third query statements to obtain a second query statement; and 4) merging the query conditions of the T third query statements, given that all T first query statements have filtering conditions and some of the T first query statements have different filtering conditions. Given the same filtering conditions, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement, resulting in T fourth query statements. The query conditions of the T fourth query statements are then merged to obtain a second query statement. Since all T first query statements have filtering conditions, and the filtering conditions of the T first query statements contain the same target sub-filtering condition, for each of the T first query statements, a preset branch function is used to merge the other sub-filtering conditions (excluding the target sub-filtering condition) of each first query statement into the query conditions of each first query statement, resulting in T fifth query statements. The query conditions of the T fifth query statements are then merged to obtain a second query statement.

[0076] In this embodiment, for each of the N sets of first query statements, when merging the T first query statements in the first query statement set, there is a situation where all T first query statements have filtering conditions. Since the filtering conditions may contain sub-filtering conditions, this situation can be further subdivided into the following cases:

[0077] In the first scenario, if all T filtering conditions of the T first query statements are the same, then the query conditions of the T first query statements can be directly merged, and the filtering conditions do not need to be adjusted in any way, thus obtaining the second query statement.

[0078] For example, consider the following two first query statements: `select count(id)as rule1 from students where gender=1` and `select max(age)as rule 2 from students where gender=1`. Both of these first query statements contain a `where` clause, and both `where` clauses have `gender=1`. Therefore, we can directly merge `select count(id)as rule1` and `select max(age)as rule2` to obtain the second query statement: `select count(id)as rule1, max(age)as rule 2 from students where gender=1`.

[0079] In the second scenario, if the T filtering conditions of the T first query statements are all different, a pre-defined branch function can be used to move the filtering condition of each first query statement to the query conditions of that first query statement, resulting in T third query statements. Then, the query conditions of these T third query statements are merged to obtain the second query statement. The pre-defined branch function can be an IF function or a CASE WHEN function.

[0080] For example, if the default branch function is an if function, it can be followed... Figure 3 The process shown merges the following two first query statements: select count(name) as rule1 from t_user where name is not null and select count(age) as rule 2 from t_user where age≤0.

[0081] like Figure 3As shown, parsing these two first query statements reveals the select, from, and where parts. In `select count(name) as rule1 from t_user where name is not null`, the select part is `count(name) as rule1`, the from part is `t_user`, and the where part is `name is not null`. Similarly, in `select count(age) as rule 2 from t_user where age≤0`, the select part is `count(age) as rule 2`, the from part is `t_user`, and the where part is `age≤0`. After analysis, it was found that the two first query statements have the same "from" part, both containing a "where" part, but the "where" parts are different. Therefore, the "if" function can be used to move the "where" part of "select count(name) as rule1 from t_user where name is not null" to the front of the "select" part, resulting in "select count(if(name is not null,name,null)) as rule1 from t_user". Similarly, the "if" function can be used to move the "where" part of "select count(age) as rule 2 from t_user where age≤0" to the front of the "select" part, resulting in "select count(if(age≤0,age,null)) as rule 2 from t_user". Then, "select count(if(name is not null,name,null)) as rule1" and "select count(if(age≤0,age,null)) as rule 2" are merged to obtain the second query statement: "select count(if(name is not null,name,null)) as rule1,count(if(age≤0,age,null)) as rule2 from t_user".

[0082] For another example, assuming the default branch function is `CASE WHEN`, consider the following two first query statements: `SELECT COUNT(name) AS rule1 FROM t_user WHERE name IS NOT NULL` and `SELECT COUNT(age) AS rule 2 FROM t_user WHERE age ≤ 0`. Both of these first query statements contain a `WHERE` clause, but the `WHERE` clauses are different. Therefore, we can use the `CASE WHEN` function to move the `WHERE` clause from `SELECT COUNT(name) AS rule1 FROM t_user WHERE name IS NOT NULL` to the `SELECT` clause, resulting in `SELECT COUNT(CASE WHEN name IS NOT NULL THEN name ELSE null END) AS rule1 FROM t_user`. Similarly, we can use the `CASE WHEN` function to move the `WHERE` clause from `SELECT COUNT(age) AS rule 2 FROM t_user WHERE age ≤ 0` to the `SELECT` clause, resulting in `SELECT COUNT(CASE WHEN age ≤ 0 THEN age ELSE null END) AS rule2 FROM t_user`. Then, the queries `select count(case when name is not null then name else null end) as rule1` and `select count(case when age≤0 then age else null end) as rule 2` are merged to obtain the second query statement: `select count(case when name is not null then name else null end) as rule1, count(case when age≤0 then age else null end) as rule 2 from t_user`.

[0083] In the third scenario, if some of the T first query statements share the same filtering conditions, then it's assumed that none of the T first query statements have completely identical filtering conditions. Therefore, a pre-defined branch function can be used to move the filtering conditions of each first query statement to the query conditions of that first query statement, resulting in T fourth query statements. The query conditions of these T fourth query statements are then merged to obtain the second query statement. The pre-defined branch function can be an IF function or a CASE WHEN function.

[0084] For example, the default branch function is illustrated by the IF function. Consider the following three first query statements: select count(id)as rule1 from students where gender=1, select max(age)as rule 2from students where gender=1, and select count(name)as rule3 from students where nameis not null. In these three first query statements, only the first two have the same filtering condition. Therefore, it is considered that these three first query statements do not have completely identical filtering conditions. Therefore, the IF function can be used to move the WHERE clause of `SELECT COUNT(id) AS rule1 FROM students WHERE gender=1` to the SELECT clause, resulting in `SELECT COUNT(IF(gender=1,id,null)) AS rule1 FROM students`; the IF function can be used to move the WHERE clause of `SELECT MAX(age) AS rule 2 FROM students WHERE gender=1` to the SELECT clause, resulting in `SELECT MAX(IF(gender=1,age,null)) AS rule 2 FROM students`; and the IF function can be used to move the WHERE clause of `SELECT COUNT(name) AS rule3 FROM students WHERE name is not null` to the SELECT clause, resulting in `SELECT COUNT(IF(name is not null,name,null)) AS rule3 FROM students`.Then, the queries `select count(if(gender=1,id,null))as rule1`, `select max(if(gender=1,age,null))as rule2`, and `select count(if(name is not null,name,null))as rule3` are merged to obtain the second query statement: `select count(if(gender=1,id,null))as rule1,select max(if(gender=1,age,null))as rule2,select count(if(name is not null,name,null))as rule3 from students`.

[0085] In the fourth scenario, some sub-filtering conditions are identical in the T filtering conditions of the T first query statements. These identical sub-filtering conditions are called target sub-filtering conditions. In this case, a preset branch function can be used to move all sub-filtering conditions except the target sub-filtering conditions from the filtering conditions of each first query statement to the query conditions of that first query statement. The target sub-filtering conditions are retained in the WHERE clause of the first query statement, resulting in T fifth query statements. Then, the query conditions of the T fifth query statements are merged to obtain the second query statement. The preset branch function can be an IF function or a CASE WHEN function.

[0086] For example, let's consider the IF function as a predefined branch function. Consider the following two first query statements: `select count(name) as rule1 from students where gender=1 and name is not null` and `select count(age) as rule 2 from students where gender=1 and age≤0`. Both of these first query statements contain a WHERE clause, and the WHERE clause contains the target sub-filter condition `gender=1`. Therefore, we can use the IF function to move the `name is not null` condition from `select count(name) as rule1 from students where gender=1 and name is not null` to the SELECT clause, resulting in `select count(if(name is not null,name,null)) as rule1 from students where gender=1`. Similarly, we can use the IF function to move the `age≤0` condition from `select count(age) as rule 2 from students where gender=1 and age≤0` to the SELECT clause, resulting in `select count(if(age≤0,age,null)) as rule 2 from students where gender=1`. Then, the query `select count(if(name is not null,name,null))as rule1` and `select count(if(age≤0,age,null))as rule2` are merged to obtain the second query statement: `select count(if(name is not null,name,null))as rule1,count(if(age≤0,age,null))as rule 2 from students where gender=1`.

[0087] Optionally, since the first query statement is used to monitor the data quality of the data to be monitored, the T first query statements are merged to obtain a second query statement, which is also used to monitor the data quality of the data to be monitored. Because the number of second query statements is less than the number of first query statements, when monitoring the data quality of the data to be monitored, it is not necessary to execute each of the T first query statements; only the second query statement needs to be executed once to achieve the monitoring of the data quality. That is, the number of executions required for the second query statement is less than the number of executions required for the first query statement, thus improving execution efficiency. Therefore, by executing the second query statement to monitor the data quality of the data to be monitored, the efficiency of data quality monitoring is improved.

[0088] In another embodiment of this disclosure, the T first query statements in the first query statement set are merged to obtain a second query statement, including: in response to the existence of filtering conditions in S first query statements among the T first query statements, for each of the S first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain S sixth query statements, where S < T and S is a positive integer; the query conditions of the T first query statements and the query conditions of the S sixth query statements are merged to obtain the second query statement.

[0089] In this embodiment, for each of the N sets of first query statements, when merging the T first query statements in each set, there may be cases where some of the T first query statements have filtering conditions, while others do not. That is, if S of the T first query statements have filtering conditions, then it is assumed that no two of the T first query statements have exactly the same filtering condition. Therefore, a preset branch function can be used to move the filtering conditions of each of the S first query statements to the query conditions of that first query statement, resulting in S sixth query statements. Then, the query conditions of the S sixth query statements are merged with the query conditions of the T first query statements to obtain the second query statement. The preset branch function can be an if function or a case when function.

[0090] For example, let's consider the IF function as a predefined branching function. Consider the following two first query statements: `select count(id) as rule1 from students` and `select max(age) as rule 2 from students where gender=1`. In these two first query statements, the first statement lacks a WHERE clause, while the second statement does. Therefore, we can use the IF function to move the WHERE clause from `select max(age) as rule2 from students where gender=1` to the SELECT clause, resulting in `select max(if(gender=1,age,null)) as rule 2 from students`. Then, we merge `select count(id) as rule1` and `select max(if(gender=1,age,null)) as rule 2` to obtain the second query statement: `select count(id) as rule1`, `select max(if(gender=1,age,null)) as rule 2 from students`.

[0091] Optionally, by merging T first query statements into a single second query statement, it is not necessary to execute the T first query statements T times. Instead, the second query statement can be executed only once to monitor the data quality of the data to be monitored, thereby greatly improving the efficiency of monitoring the data quality of the data to be monitored.

[0092] In another embodiment of this disclosure, extracting the data source for each of the M first query statements includes: for each of the M first query statements, performing the following steps: parsing the first query statement to extract multiple query variables of the first query statement; and determining the data source based on the meaning of the variables represented by the multiple query variables.

[0093] In this embodiment, a query statement includes a select part and a from part. More complex query statements may also include at least one of the following: a where part, a group by part, a having part, an order by part, and a limit part. Therefore, when extracting the data source of the first query statement, the first query statement can be parsed first to extract multiple query variables included in the first query statement, namely, select query variables, from query variables, where query variables, group by query variables, having query variables, order by query variables, and limit query variables. Then, based on the meaning of each query variable, the query variable representing the data source is determined, and thus the data source of the first query statement is determined based on the query variable representing the data source.

[0094] In one optional implementation, the `select` query variable represents the query conditions, the `from` query variable represents the data source, the `where` query variable represents the filtering conditions within the data source, the `group by` query variable represents the grouping and statistical conditions, the `having` query variable represents further filtering of the grouped and statistically analyzed results, the `order by` query variable represents the sorting conditions, and the `limit` query variable represents pagination of the sorted results. It can be seen that the `from` query variable represents the data source; therefore, the field name corresponding to the `from` variable in the first query statement is the table name of the data source.

[0095] Optionally, merging the first query statements from the same data source can yield a second query statement. Since the number of second query statements is less than the number of first query statements, the number of executions required to execute the second query statement is less than the number of executions required to execute the first query statement, thereby improving execution efficiency. In this way, the monitoring efficiency of the data quality of the data to be monitored can be improved by executing the second query statement.

[0096] In another embodiment of this disclosure, the data quality monitoring task includes M data quality monitoring rules, each of the M data quality monitoring rules includes a rule statement, and each of the M data quality monitoring rules corresponds one-to-one with each of the M first query statements; executing the data quality monitoring task to generate M first query statements includes: assigning values ​​to the target query variables in the M rule statements corresponding to the M data quality monitoring rules according to preset variable assignment rules to obtain M first query statements.

[0097] In this embodiment, the data quality monitoring rule includes rule statements and the expected range corresponding to those rule statements. The data quality monitoring task for the data to be monitored is to run the rule statements in the data quality monitoring rule and determine whether the running result is within the expected range, thereby determining the data quality of the data to be monitored. The data to be monitored can be data from a single table, data from multiple tables, or data corresponding to a specific field in a table.

[0098] In the above embodiments, when performing a data quality monitoring task, the target query variable in the M rule statements contained in the M data quality monitoring rules can be assigned values ​​according to the preset variable assignment rules, thereby generating M first query statements. This allows the M first query statements to be merged according to the data source to obtain a second query statement, thereby improving the monitoring efficiency of the data quality of the data to be monitored by executing the second query statement.

[0099] For example, a data quality monitoring task involves monitoring the data quality of a user information table (table name t_user). This table contains the following fields: username (name), ID number (id), age (age), and gender (gender). The requirements are that the username cannot be null and the gender must be female. This data quality monitoring task includes two data quality monitoring rules. The first rule counts the number of rows where the username is not null, with the corresponding rule statement being `SELECT COUNT(name) FROM t_user WHERE name_not_null`. The second rule counts the number of rows where the gender is female, with the corresponding rule statement being `SELECT COUNT(gender) FROM t_user WHERE gender_female`. The default variable assignment rules are: null for null, 1 for male, and 0 for female. The first query statement corresponding to Select count(name) From t_user Where name is not null is Select count(name) From t_user Where name is not null, and the first query statement corresponding to Select count(gender) From t_user Where gender is female is Select count(gender) From t_user Where gender = 0.

[0100] In another embodiment of this disclosure, after obtaining the execution result, the method further includes: splitting the execution result into multiple sub-results based on the fields in the execution result that correspond to the data quality monitoring rules, with each sub-result corresponding to a different data quality monitoring rule.

[0101] In this embodiment, the second query statement is obtained by merging multiple first query statements. Therefore, the execution result of the second query statement includes the execution results corresponding to multiple first query statements. Since this execution result is a composite result, it cannot be verified using the data quality monitoring rules included in the data quality monitoring task. To ensure the reliability of the data quality of the obtained data to be monitored, the execution result of the second query statement needs to be split. This allows us to obtain the execution result of each first query statement, so that the corresponding sub-results can be verified according to the data quality monitoring rules corresponding to each first query statement.

[0102] In the above embodiments, when splitting the execution result of the second query statement, since each first query statement corresponds to a data quality monitoring rule, and each data quality monitoring rule has a different field in the execution result of the second query statement, the execution result is split into multiple sub-results based on the field corresponding to the data quality monitoring rule in the execution result.

[0103] For example, the result of the second query is Table 1:

[0104] Table 1 shows the execution results of the second query statement.

[0105] rule1 Value1 rule2 Value2

[0106] After splitting Table 1, the resulting sub-results are Table 2 and Table 3:

[0107] Table 2 Sub-results after splitting

[0108] rule1 Value1

[0109] Table 3 Sub-results after splitting

[0110] rule2 Value2

[0111] In another embodiment of this disclosure, each of the M data quality monitoring rules further includes an expected result corresponding to the rule statement; after obtaining multiple sub-results, the data processing method further includes: verifying the multiple sub-results according to the expected result to determine the data quality of the data to be monitored.

[0112] In this embodiment, the data quality monitoring rule includes rule statements and the expected range corresponding to those rule statements. The data quality monitoring task for the data to be monitored is to run the rule statements in the data quality monitoring rule and determine whether the result is within the expected range. Therefore, it is necessary to split the execution result according to the data quality monitoring rule to obtain the sub-result corresponding to each data quality monitoring rule. This allows each data quality monitoring rule to be mapped to each sub-result, thereby improving the accuracy and convenience of verifying the corresponding sub-results. If the result is not split, the sub-result verified by the data quality monitoring rule may not be the sub-result corresponding to that data quality monitoring rule, leading to verification errors. Therefore, verifying the corresponding sub-results using the expected results contained in the data quality monitoring rule ensures the accuracy of monitoring the data quality of the data to be monitored.

[0113] In the above embodiment, after splitting the execution result of the second query statement into multiple sub-results, each sub-result can be verified according to the expected range contained in the data quality monitoring rules corresponding to the fields in each sub-result. This determines whether the data to be monitored is within the expected range. If it is within the expected range, the data quality of the data to be monitored can be considered excellent; if it is not within the expected range, the data quality of the data to be monitored can be considered poor. Therefore, after verifying the sub-results, the accuracy of the final determination of the data quality of the data to be monitored is high.

[0114] For example, a data quality monitoring task involves monitoring the data quality of a user information table (table name t_user). This table contains the following fields: username (name), ID number (id), age (age), and gender (gender). The requirements are that the username cannot be null and the gender must be female. This data quality monitoring task includes two data quality monitoring rules. The first rule counts the number of rows where the username is not null, with the corresponding rule statement being `Select count(name) From t_user Where name is not null`, and the expected range is: number of result rows / total number of rows in the table = 100%. The second rule counts the number of rows where the gender is female, with the corresponding rule statement being `Select count(gender) From t_user Where gender is female`, and the expected range is: number of result rows = 0. After splitting, the first sub-result of the first query statement corresponding to the first data quality monitoring rule is: number of result rows / total number of rows in the table = 98%. The second sub-result of the first query statement corresponding to the second data quality monitoring rule is: number of result rows = 0. Since only the second sub-result meets the expected range, while the first sub-result does not, it can be concluded that the data quality of the data to be monitored is poor, or that there are anomalies in the data to be monitored.

[0115] In another embodiment of this disclosure, the method further includes: executing a data quality monitoring task, generating configuration information, the configuration information further including at least one of the following: snapshot information of the data quality monitoring task, start time and end time of executing the data quality monitoring task, triggering method of executing the data quality monitoring task, execution parameters of executing the data quality monitoring task, execution queue of executing the data quality monitoring task, computing resources and storage resources consumed when executing the data quality monitoring task; executing a second query statement to obtain an execution result, including: executing a second query statement based on the configuration information to obtain an execution result.

[0116] In this embodiment, after executing the data quality monitoring task, in addition to generating multiple first query statements, configuration information can also be generated. This configuration information can be used to represent various configurations during the execution of the data quality monitoring task. Through this configuration information, the execution status of the data quality monitoring task can be monitored in real time, enabling timely detection of anomalies during execution. When anomalies are detected, appropriate strategies can be taken to address them, ensuring the normal execution of the data quality monitoring task and thus guaranteeing the generation of accurate first query statements. Furthermore, by merging accurate first query statements, the accuracy of the resulting second query statement can be improved. Therefore, after executing the second query statement, the accuracy of monitoring the data quality of the data to be monitored can be enhanced.

[0117] In another embodiment of this disclosure, the data quality monitoring task further includes at least one of the following: data source information of the data to be monitored, task information of the data quality monitoring task, and resource information required to perform the data quality monitoring task.

[0118] In this embodiment, the data quality monitoring task may include, in addition to several data quality monitoring rules, at least one of the following: data source information of the data to be monitored, including data source type, data source address, data source connection information, etc.; task information of the data quality monitoring task, including task name, task description, relevant personnel responsible for the task, and task ID; and resource information required to execute the data quality monitoring task, including memory resources, central processing unit (CPU) resources, thread resources, etc. Through this information, it can be determined whether the execution environment of the data quality monitoring task meets these requirements. If it does, the data quality monitoring task can be executed; if not, the execution environment of the data quality monitoring task needs to be adjusted to ensure its success rate.

[0119] In another embodiment of this disclosure, the data processing method is applied to a client, which has a computing engine deployed for running query statements. Executing a second query statement and obtaining an execution result includes: using the computing engine to execute the second query statement and obtaining an execution result.

[0120] In this embodiment, after receiving the second query statement, the server can send the second query statement to the client. The client uses a computing engine to execute the second query statement to improve the accuracy of the generated execution result and returns the execution result to the server. Since the second query statement is obtained by merging multiple first query statements, the data quality of the data to be monitored can be monitored by executing the second query statement only once, thereby greatly improving the efficiency of monitoring the data quality of the data to be monitored.

[0121] In the above embodiments, executing the second query statement can be as follows: Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the execution of the second query statement provided in this disclosure. Figure 4In the example of the second query statement, select count(if(name is not null,name,null))as rule1,count(if(age≤0,age,null))as rule 2from t_user, first reads the entire t_user table, filters name is not null and age≤0, aggregates count(name) and count(age), and finally obtains the execution result.

[0122] The data processing method disclosed herein merges the first query statement to obtain the second query statement. Since the second query statement has fewer elements than the first query statement, the number of executions required for the second query statement is also less than the number of executions required for the first query statement. This reduces the number of times the same data source is repeatedly read, thereby improving the monitoring efficiency of the data quality of the data to be monitored, as shown in Table 4.

[0123] Table 4 Comparison of execution time before and after query statement merging

[0124]

[0125]

[0126] Exemplary media

[0127] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 5 The storage medium of the exemplary embodiments of this disclosure will be described.

[0128] refer to Figure 5 As shown, the storage medium 50 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0129] The program product may employ any combination of one or more readable media. A readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0131] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0132] Exemplary device

[0133] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 6 The data processing apparatus of the exemplary embodiments of this disclosure will be described to implement the method in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0134] Figure 6 This is a schematic diagram of the structure of the data processing apparatus according to an embodiment of the present disclosure. Figure 6 In this context, the data processing device 600 includes:

[0135] The generation module 601 is used to respond to a trigger instruction received for performing a data quality monitoring task on the data to be monitored, execute the data quality monitoring task, and generate M first query statements. The data quality monitoring task is used to generate query statements for monitoring the data quality of the data to be monitored, where M is a positive integer.

[0136] Extraction module 602 is used to extract the data source of each of the M first query statements;

[0137] The merging module 603 is used to merge M first query statements based on the data source to obtain a second query statement;

[0138] Execution module 604 is used to execute the second query statement and obtain the execution result, which is used to determine the data quality of the data to be monitored.

[0139] Optionally, the data processing device 600 is applied to the server, and the merging module 603 is used to merge M first query statements based on the data source to obtain a second query statement in the following way: based on the data source, the M first query statements are classified to obtain N sets of first query statements, where the data sources of different first query statement sets in the N sets are different, N is less than M and N is a positive integer; for each of the N sets of first query statements, T first query statements in the first query statement set are merged to obtain a second query statement, where T is less than M and T is a positive integer.

[0140] Optionally, the merging module 603 is used to merge T first query statements in the first query statement set to obtain a second query statement by: extracting the query conditions of the T first query statements, or extracting the query conditions and filter conditions of the T first query statements; in response to the fact that none of the T first query statements have filter conditions, merging the query conditions of the T first query statements to obtain a second query statement.

[0141] Optionally, the merging module 603 is used to merge T first query statements in the first query statement set to obtain a second query statement in the following ways: In response to the fact that all T first query statements have filtering conditions and the filtering conditions are the same, the query conditions of the T first query statements are merged to obtain a second query statement; In response to the fact that all T first query statements have filtering conditions and the filtering conditions are different, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain T third query statements; The query conditions of the T third query statements are merged to obtain a second query statement; In response to the fact that all T first query statements have filtering conditions and some of the first query conditions of the T first query statements are different... Since the filtering conditions of the statements are the same, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement, resulting in T fourth query statements; the query conditions of the T fourth query statements are then merged to obtain a second query statement; in response to the fact that all T first query statements have filtering conditions, and the filtering conditions of the T first query statements contain the same target sub-filtering conditions, for each of the T first query statements, a preset branch function is used to merge the other sub-filtering conditions of each first query statement (excluding the target sub-filtering condition) into the query conditions of each first query statement, resulting in T fifth query statements; the query conditions of the T fifth query statements are then merged to obtain a second query statement.

[0142] Optionally, the merging module 603 is used to merge T first query statements in the first query statement set to obtain a second query statement in the following manner: in response to the existence of filtering conditions in S first query statements among the T first query statements, for each of the S first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain S sixth query statements, where S < T and S is a positive integer; the query conditions of the T first query statements and the query conditions of the S sixth query statements are merged to obtain the second query statement.

[0143] Optionally, the extraction module 602 is used to extract the data source of each of the M first query statements in the following manner: for each of the M first query statements, the following steps are performed: the first query statement is parsed and processed to extract multiple query variables of the first query statement; the data source is determined according to the meaning of the variables represented by the multiple query variables.

[0144] Optionally, the data quality monitoring task includes M data quality monitoring rules, each of the M data quality monitoring rules includes a rule statement, and each of the M data quality monitoring rules corresponds one-to-one with each of the M first query statements; the generation module 601 is used to execute the data quality monitoring task and generate M first query statements in the following way: according to the preset variable assignment rules, the target query variables in the M rule statements corresponding to the M data quality monitoring rules are assigned values ​​to obtain the M first query statements.

[0145] Optionally, the data processing device 600 further includes a splitting module (not shown), used to split the execution result after obtaining the execution result according to the fields in the execution result that correspond to the data quality monitoring rules, to obtain multiple sub-results, each sub-result corresponding to a different data quality monitoring rule.

[0146] Optionally, each of the M data quality monitoring rules also includes the expected result corresponding to the rule statement. The data processing device 600 further includes a verification module (not shown) for verifying the multiple sub-results according to the expected result after obtaining the multiple sub-results, so as to determine the data quality of the data to be monitored.

[0147] Optionally, the generation module 601 is also used to execute a data quality monitoring task and generate configuration information. The configuration information includes at least one of the following: snapshot information of the data quality monitoring task, start time and end time of the data quality monitoring task, triggering method of the data quality monitoring task, execution parameters of the data quality monitoring task, execution queue of the data quality monitoring task, and computing and storage resources consumed when executing the data quality monitoring task. The execution module 604 is used to execute the second query statement and obtain the execution result in the following way: based on the configuration information, execute the second query statement and obtain the execution result.

[0148] Optionally, the data quality monitoring task may also include at least one of the following: data source information of the data to be monitored, task information of the data quality monitoring task, and resource information required to perform the data quality monitoring task.

[0149] Optionally, the data processing device 600 is applied to a client, which has a computing engine deployed for running query statements. The execution module 604 is used to execute the second query statement and obtain the execution result by using the computing engine to execute the second query statement and obtain the execution result.

[0150] Exemplary computing device

[0151] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 7 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0152] Figure 7 The computing device 70 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0153] like Figure 7 As shown, the computing device 70 is presented in the form of a general-purpose computing device. The components of the computing device 70 may include, but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including the processing unit 701 and the storage unit 702). The at least one storage unit 702 stores computer-executable instructions; the at least one processing unit 701 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0154] The 703 bus includes a data bus, a control bus, and an address bus.

[0155] Storage unit 702 may include readable media in the form of volatile memory, such as random access memory (RAM) 7021 and / or cache memory 7022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 7023.

[0156] Storage unit 702 may also include a program / utility 7025 having a set (at least one) program module 7024, such program module 7024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0157] The computing device 70 can also communicate with one or more external devices 704 (e.g., keyboard, pointing device, etc.). This communication can be performed via an input / output interface 705, which may also be referred to as an I / O interface 705. Furthermore, the computing device 70 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter 706. Figure 7 As shown, network adapter 706 communicates with other modules of computing device 70 via bus 703. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 70, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] It should be noted that although several units / modules or sub-units / modules of the data processing apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0159] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0160] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data processing method, comprising: In response to receiving a trigger instruction for performing a data quality monitoring task on the data to be monitored, the data quality monitoring task is executed, generating M first query statements. The data quality monitoring task is used to generate query statements for monitoring the data quality of the data to be monitored, where M is a positive integer. Extract the data source for each of the M first query statements; Based on the data source, the M first query statements are classified to obtain N sets of first query statements. The data sources of different first query statement sets in the N sets are different, and N is less than M and N is a positive integer. For each of the N sets of first query statements, the T first query statements in the first query statement set are merged to obtain the second query statement, where T is less than M and T is a positive integer; The second query statement is executed to obtain the execution result, which is used to determine the data quality of the data to be monitored. The step of merging T first query statements in the first query statement set to obtain the second query statement includes: extracting the query conditions of the T first query statements, or extracting the query conditions and filtering conditions of the T first query statements. In response to the presence of filtering conditions in S of the T first query statements, a preset branch function is used for each of the S first query statements to merge the filtering conditions of each first query statement into the query conditions of each first query statement, resulting in S sixth query statements, where S < T and S is a positive integer. The query conditions of the TS first query statements and the query conditions of the S sixth query statements are merged to obtain the second query statement.

2. The data processing method according to claim 1, wherein merging T first query statements in the first query statement set to obtain the second query statement includes: Since none of the T first query statements have filtering conditions, the query conditions of the T first query statements are merged to obtain the second query statement.

3. The data processing method according to claim 1, wherein merging the T first query statements in the first query statement set to obtain the second query statement further includes: Since all T first query statements have filtering conditions and the filtering conditions are the same, the query conditions of the T first query statements are merged to obtain the second query statement; Since all T first query statements have filtering conditions and the filtering conditions are different, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain T third query statements. The query conditions of the T third query statements are merged to obtain the second query statement; In response to the fact that all T first query statements have filtering conditions, and that some of the first query statements have the same filtering conditions, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain T fourth query statements; the query conditions of the T fourth query statements are then merged to obtain the second query statement. In response to the fact that all T first query statements have filtering conditions, and that the filtering conditions of the T first query statements contain the same target sub-filtering condition, for each of the T first query statements, a preset branch function is used to merge the other sub-filtering conditions (excluding the target sub-filtering condition) in the filtering conditions of each first query statement into the query conditions of each first query statement, resulting in T fifth query statements; the query conditions of the T fifth query statements are then merged to obtain the second query statement.

4. The data processing method according to any one of claims 1-3, wherein extracting the data source for each of the M first query statements comprises: For each of the M first query statements, perform the following steps: The first query statement is parsed and processed to extract multiple query variables from the first query statement; The data source is determined based on the meanings of the variables represented by the multiple query variables.

5. The data processing method according to any one of claims 1-3, wherein the data quality monitoring task includes M data quality monitoring rules, each of the M data quality monitoring rules includes a rule statement, and each of the M data quality monitoring rules corresponds one-to-one with each of the M first query statements; The execution of the data quality monitoring task generates M first query statements, including: According to the preset variable assignment rules, the target query variables in the M rule statements corresponding to the M data quality monitoring rules are assigned values ​​to obtain M first query statements.

6. The data processing method according to claim 5, further comprising, after obtaining the execution result: Based on the fields in the execution result that correspond to the data quality monitoring rules, the execution result is split into multiple sub-results, each with a different data quality monitoring rule.

7. The data processing method according to claim 6, wherein each of the M data quality monitoring rules further includes the expected result corresponding to the rule statement; after obtaining multiple sub-results, it further includes: Based on the expected result, the multiple sub-results are verified to determine the data quality of the data to be monitored.

8. The data processing method according to any one of claims 1-3, further comprising: Execute the data quality monitoring task and generate configuration information, which includes at least one of the following: snapshot information of the data quality monitoring task, start time and end time of the data quality monitoring task, triggering method of the data quality monitoring task, execution parameters of the data quality monitoring task, execution queue of the data quality monitoring task, and computing and storage resources consumed when executing the data quality monitoring task. The execution of the second query statement yields the following results: Based on the configuration information, the second query statement is executed to obtain the execution result.

9. The data processing method according to claim 8, wherein the data quality monitoring task further includes at least one of the following: data source information of the data to be monitored, task information of the data quality monitoring task, and resource information required to perform the data quality monitoring task.

10. The data processing method according to claim 1, applied to a client, wherein the client is equipped with a computing engine for running query statements, and the execution of the second query statement to obtain the execution result includes: The second query statement is executed using the computing engine to obtain the execution result.

11. A medium storing computer-executable instructions, which, when executed, are used to implement the data processing method as described in any one of claims 1 to 10.

12. A data processing apparatus, comprising: The generation module is used to respond to a trigger instruction received for performing a data quality monitoring task on the data to be monitored, execute the data quality monitoring task, and generate M first query statements. The data quality monitoring task is used to generate query statements for monitoring the data quality of the data to be monitored, where M is a positive integer. The extraction module is used to extract the data source for each of the M first query statements; The merging module is used to merge the M first query statements based on the data source to obtain a second query statement; The execution module is used to execute the second query statement and obtain the execution result, which is used to determine the data quality of the data to be monitored. Applied to the server side, the merging module is used to merge the M first query statements based on the data source to obtain a second query statement in the following way: Based on the data source, the M first query statements are classified to obtain N sets of first query statements. The data sources of different first query statement sets in the N sets are different, and N is less than M and N is a positive integer. For each of the N sets of first query statements, the T first query statements in the first query statement set are merged to obtain the second query statement, where T is less than M and T is a positive integer; The merging module is used to merge T first query statements in the first query statement set to obtain the second query statement in the following manner: Extract the query conditions of the T first query statements, or extract the query conditions and filter conditions of the T first query statements; In response to the presence of filtering conditions in S of the T first query statements, a preset branch function is used for each of the S first query statements to merge the filtering conditions of each first query statement into the query conditions of each first query statement, resulting in S sixth query statements, where S < T and S is a positive integer. The query conditions of the TS first query statements and the query conditions of the S sixth query statements are merged to obtain the second query statement.

13. The data processing apparatus according to claim 12, wherein the merging module is configured to merge T first query statements in the first query statement set to obtain the second query statement in the following manner: Since none of the T first query statements have filtering conditions, the query conditions of the T first query statements are merged to obtain the second query statement.

14. The data processing apparatus according to claim 12, wherein the merging module is configured to merge T first query statements in the first query statement set to obtain the second query statement in the following manner: Since all T first query statements have filtering conditions and the filtering conditions are the same, the query conditions of the T first query statements are merged to obtain the second query statement; Since all T first query statements have filtering conditions and the filtering conditions are different, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain T third query statements. The query conditions of the T third query statements are merged to obtain the second query statement; In response to the fact that all T first query statements have filtering conditions, and that some of the first query statements have the same filtering conditions, for each of the T first query statements, a preset branch function is used to merge the filtering conditions of each first query statement into the query conditions of each first query statement to obtain T fourth query statements; the query conditions of the T fourth query statements are then merged to obtain the second query statement. In response to the fact that all T first query statements have filtering conditions, and that the filtering conditions of the T first query statements contain the same target sub-filtering condition, for each of the T first query statements, a preset branch function is used to merge the other sub-filtering conditions (excluding the target sub-filtering condition) in the filtering conditions of each first query statement into the query conditions of each first query statement, resulting in T fifth query statements; the query conditions of the T fifth query statements are then merged to obtain the second query statement.

15. The data processing apparatus according to any one of claims 12-14, wherein the extraction module is configured to extract the data source of each of the M first query statements in the following manner: For each of the M first query statements, perform the following steps: The first query statement is parsed and processed to extract multiple query variables from the first query statement; The data source is determined based on the meanings of the variables represented by the multiple query variables.

16. The data processing apparatus according to any one of claims 12-14, wherein the data quality monitoring task includes M data quality monitoring rules, each of the M data quality monitoring rules includes a rule statement, and each of the M data quality monitoring rules corresponds one-to-one with each of the M first query statements; The generation module is used to execute the data quality monitoring task and generate M first query statements in the following manner: According to the preset variable assignment rules, the target query variables in the M rule statements corresponding to the M data quality monitoring rules are assigned values ​​to obtain M first query statements.

17. The data processing apparatus according to claim 16, further comprising: The splitting module is used to split the execution result after obtaining the execution result according to the fields in the execution result that correspond to the data quality monitoring rules, so as to obtain multiple sub-results, each of which corresponds to different data quality monitoring rules.

18. The data processing apparatus according to claim 17, wherein each of the M data quality monitoring rules further includes the expected result corresponding to the rule statement, and further includes: The verification module is used to verify the multiple sub-results according to the expected result after obtaining multiple sub-results, so as to determine the data quality of the data to be monitored.

19. The data processing apparatus according to any one of claims 12-14, wherein the generation module is further configured to execute the data quality monitoring task and generate configuration information, the configuration information further comprising at least one of the following: snapshot information of the data quality monitoring task, start time and end time of executing the data quality monitoring task, triggering method of executing the data quality monitoring task, execution parameters of executing the data quality monitoring task, execution queue of executing the data quality monitoring task, and computing resources and storage resources consumed when executing the data quality monitoring task; The execution module is used to execute the second query statement and obtain the execution result in the following manner: Based on the configuration information, the second query statement is executed to obtain the execution result.

20. The data processing apparatus according to claim 19, wherein the data quality monitoring task further includes at least one of the following: data source information of the data to be monitored, task information of the data quality monitoring task, and resource information required to perform the data quality monitoring task.

21. The data processing apparatus according to claim 12, applied to a client, wherein the client is equipped with a computing engine for running query statements, and the execution module is configured to execute the second query statement and obtain an execution result in the following manner: The second query statement is executed using the computing engine to obtain the execution result.

22. A computing device, comprising: A processor, and a memory connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the data processing method as described in any one of claims 1 to 10.

Images