Data quality inspection method and device supporting natural language processing, equipment and medium

Abstract

The invention relates to a data quality inspection method and device supporting natural language processing, equipment and a medium. The data quality inspection method supporting natural language processing comprises the following steps: receiving a data quality inspection requirement statement described by a user in a natural language; analyzing the data quality inspection requirement statement, and converting the data quality inspection requirement statement into a quality inspection rule statement described by a computer language; combining the quality inspection rule statements into a data quality inspection task; and executing the data quality inspection task to obtain a data quality inspection result. According to the method, the data quality inspection requirement statement described by the natural language is converted into the quality inspection rule statement described by the computer language, so that a user can send out the quality inspection instruction of the data quality inspection rule and the task operation requirement through simple and understandable natural language input, the technical threshold of data quality inspection is reduced, the method is simple and easy to use, and the user experience is improved. And the universality of the data quality inspection scheme is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data quality inspection method, device, equipment and medium supporting natural language processing. Background Art

[0002] Nowadays, data has become a new type of asset for enterprises. Effective data can support the analysis and decision-making of enterprises, while wrong data may have negative effects. Poor data quality will bring various problems, including low data credibility, affecting the accuracy of data analysis and data mining, and may lead to wrong decisions.

[0003] As the work at the data development level increases and the links become longer, if the corresponding checks are not configured at some key nodes, it will be difficult to locate the data errors once they occur. Therefore, data quality monitoring is the top priority of data development work. Doing a good job of data quality inspection can improve the credibility of data, timely discover data errors, better locate problems and improve work efficiency. At present, the amount of data that needs to be quality checked is huge. For business personnel who do not have sufficient technical knowledge, the development tools for data quality inspection are difficult to use, the technical threshold is high, and the automation level of the quality inspection plan is low, which brings difficulties to data quality inspection work. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data quality inspection method, device, equipment and medium supporting natural language processing.

[0005] In a first aspect, the present disclosure provides a data quality inspection method supporting natural language processing, comprising:

[0006] Receive data quality inspection requirement statements described by users in natural language;

[0007] Parse data quality inspection requirement statements and convert them into quality inspection rule statements described in computer language;

[0008] Combine quality inspection rule statements into data quality inspection tasks;

[0009] Perform data quality inspection tasks and obtain data quality inspection results.

[0010] Optionally, parsing the data quality inspection requirement statement and converting the data quality inspection requirement statement into a quality inspection rule statement described in a computer language includes:

[0011] Convert data quality inspection requirement statements into SQL statements;

[0012] Match the corresponding data quality inspection rules according to the SQL statement, and fill the input parameters in the SQL statement into the data quality inspection rules to obtain the quality inspection rule statement.

[0013] Optionally, the data quality inspection requirement statement is parsed and converted into an SQL statement, including:

[0014] Perform semantic analysis on data quality inspection requirement statements to obtain semantic analysis results;

[0015] According to the semantic analysis results, the sentence structure of the data quality inspection requirement statement is mapped to the structure of the SQL statement, and the words in the data quality inspection requirement statement are replaced with SQL symbols to obtain the SQL statement converted from the data quality inspection requirement statement.

[0016] Optionally, before matching the corresponding data quality inspection rules according to the SQL statement, the following is also included:

[0017] Bind the pre-configured data quality inspection rules with the instruction words that express the intent in the SQL language to obtain the corresponding relationship between the data quality inspection rules and the SQL instruction words;

[0018] Among them, each data quality inspection rule corresponds to a data quality inspection indicator;

[0019] According to the SQL statement, the corresponding data quality inspection rules are matched, including:

[0020] Parse the keywords that express the intent in the SQL statement and determine the target keywords as SQL instruction words;

[0021] Determine the target data quality inspection rules corresponding to the target keywords based on the corresponding relationship;

[0022] Use the target data quality inspection rules as the data quality inspection rules corresponding to the SQL statement.

[0023] Optionally, the data quality inspection rule includes parameter input items, and the parameter input items at least include data source input items, statistical value parameters, comparison value parameters, and parameters related to result judgment;

[0024] The data source input item is used to define the data range that needs to be quality checked;

[0025] The statistical value parameter is used to define the calculation method of the statistical value. The statistical value is the value obtained after performing the corresponding quality inspection operation on the data that needs quality inspection.

[0026] The comparison value parameter is used to define the calculation method or source of the comparison value. The comparison value is the value used as the comparison target of the statistical value;

[0027] The parameters related to result judgment are used to define the method of judging whether the quality inspection data is abnormal.

[0028] Optionally, perform a data quality inspection task to obtain data quality inspection results, including:

[0029] Send the data quality inspection task to the execution component, and use the execution component to convert the data quality inspection task into the program parameters required by the data quality inspection component;

[0030] The program parameters are transmitted to the data quality check component, and the data quality check component verifies the corresponding data according to the program parameters to obtain a data verification result, which includes a statistical value and a comparison value of the corresponding data;

[0031] The data quality inspection results are determined based on the relevant parameters and data verification results.

[0032] Optionally, the parameters related to the result judgment include an operation formula, a comparison operator, and a user-defined threshold value;

[0033] The data quality inspection results are determined based on the relevant parameters and data verification results, including:

[0034] Fill the statistical value and comparison value in the data verification result into the calculation formula to obtain the calculation result;

[0035] If the comparison result between the operation result and the threshold value meets the comparison method defined by the comparison symbol, the data quality inspection result is determined to be data anomaly.

[0036] In a second aspect, the present disclosure provides a data quality inspection device supporting natural language processing, comprising:

[0037] An input module is used to receive data quality inspection requirement statements described by users in natural language;

[0038] A conversion module, used to parse the data quality inspection requirement statement and convert the data quality inspection requirement statement into a quality inspection rule statement described in a computer language;

[0039] The task generation module is used to combine quality inspection rule statements into data quality inspection tasks;

[0040] The execution module is used to execute data quality inspection tasks and obtain data quality inspection results.

[0041] Optionally, when parsing the data quality inspection requirement statement and converting the data quality inspection requirement statement into a quality inspection rule statement described in a computer language, the conversion module is specifically used to:

[0042] Convert data quality inspection requirement statements into SQL statements;

[0043] Match the corresponding data quality inspection rules according to the SQL statement, and fill the input parameters in the SQL statement into the data quality inspection rules to obtain the quality inspection rule statement.

[0044] Optionally, when parsing the data quality inspection requirement statement and converting the data quality inspection requirement statement into an SQL statement, the conversion module is specifically used to:

[0045] Perform semantic analysis on data quality inspection requirement statements to obtain semantic analysis results;

[0046] According to the semantic analysis results, the sentence structure of the data quality inspection requirement statement is mapped to the structure of the SQL statement, and the words in the data quality inspection requirement statement are replaced with SQL symbols to obtain the SQL statement converted from the data quality inspection requirement statement.

[0047] Optionally, the data quality inspection device further includes a configuration module, which is used to bind the pre-configured data quality inspection rules with the instruction words expressing the intent in the SQL language before matching the corresponding data quality inspection rules according to the SQL statement, so as to obtain the corresponding relationship between the data quality inspection rules and the SQL instruction words; wherein each data quality inspection rule corresponds to a data quality inspection indicator;

[0048] When the conversion module matches the corresponding data quality inspection rules according to the SQL statement, it is specifically used to:

[0049] Parse the keywords that express the intent in the SQL statement and determine the target keywords as SQL instruction words;

[0050] Determine the target data quality inspection rules corresponding to the target keywords based on the corresponding relationship;

[0051] Use the target data quality inspection rules as the data quality inspection rules corresponding to the SQL statement.

[0052] Optionally, the data quality inspection rule includes parameter input items, and the parameter input items at least include data source input items, statistical value parameters, comparison value parameters, and parameters related to result judgment;

[0053] The data source input item is used to define the data range that needs to be quality checked;

[0054] The statistical value parameter is used to define the calculation method of the statistical value. The statistical value is the value obtained after performing the corresponding quality inspection operation on the data that needs quality inspection.

[0055] The comparison value parameter is used to define the calculation method or source of the comparison value. The comparison value is the value used as the comparison target of the statistical value;

[0056] The parameters related to result judgment are used to define the method of judging whether the quality inspection data is abnormal.

[0057] Optionally, when the execution module performs the data quality inspection task and obtains the data quality inspection result, it is specifically used to:

[0058] Send the data quality inspection task to the execution component, and use the execution component to convert the data quality inspection task into the program parameters required by the data quality inspection component;

[0059] The program parameters are transmitted to the data quality check component, and the data quality check component verifies the corresponding data according to the program parameters to obtain a data verification result, which includes a statistical value and a comparison value of the corresponding data;

[0060] The data quality inspection results are determined based on the relevant parameters and data verification results.

[0061] Optionally, the parameters related to the result judgment include an operation formula, a comparison operator, and a user-defined threshold value;

[0062] When the execution module determines the data quality inspection result based on the result judgment related parameters and data verification results, it is specifically used to:

[0063] Fill the statistical value and comparison value in the data verification result into the calculation formula to obtain the calculation result;

[0064] If the comparison result between the operation result and the threshold value meets the comparison method defined by the comparison symbol, the data quality inspection result is determined to be data anomaly.

[0065] In a third aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method as described in any one of the first aspects is implemented.

[0066] In a fourth aspect, the present disclosure provides a computer-readable storage medium having program instructions stored thereon, which implement any method of the first aspect when the program instructions are executed.

[0067] In a fifth aspect, the present disclosure provides a computer program product, which is stored in a storage medium. When the program product is executed, the method of the first aspect can be implemented.

[0068] Compared with the prior art, the technical solution provided by the present invention has the following advantages:

[0069] The data quality inspection method, device, equipment and medium supporting natural language processing provided by the present disclosure parse the data quality inspection requirement statement after receiving the data quality inspection requirement statement described by the user in natural language, convert the data quality inspection requirement statement into a quality inspection rule statement described in computer language, and then combine the quality inspection rule statements into a data quality inspection task, execute the data quality inspection task, and obtain the data quality inspection result. The present disclosure converts the data quality inspection requirement statement described in natural language into a quality inspection rule statement described in computer language, so that the user can issue the quality inspection instructions of the data quality inspection rules and task operation requirements through simple and easy-to-understand natural language input, thereby reducing the technical threshold of data quality inspection, being simple and easy to use, and improving the versatility of the data quality inspection solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] Figure 1 A flow chart of a data quality inspection method supporting natural language processing provided by an embodiment of the present disclosure;

[0073] Figure 2 is a schematic diagram of a program architecture provided by an embodiment of the present disclosure;

[0074] Figure 3 A schematic diagram of the structure of a data quality inspection device supporting natural language processing provided by an embodiment of the present disclosure;

[0075] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0076] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0077] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0078] Figure 1 A flow chart of a data quality inspection method supporting natural language processing is provided in an embodiment of the present disclosure. The method can be executed by a data quality inspection device supporting natural language processing. The data quality inspection device supporting natural language processing can be implemented in software and / or hardware. The data quality inspection device supporting natural language processing can be configured in an electronic device, such as a server or a terminal.

[0079] like Figure 1 As shown, the data quality inspection method supporting natural language processing includes the following steps:

[0080] S101. Receive a data quality inspection requirement statement described by a user in natural language.

[0081] Natural language refers to a language that evolves naturally with culture. For example, Chinese and English are both natural languages. Users can directly enter data quality inspection requirements expressed in natural language in the program's input interface, such as "screening data with less than 5% missing values", and the program receives the data quality inspection requirement statement entered by the user.

[0082] S102, parsing the data quality inspection requirement statement, and converting the data quality inspection requirement statement into a quality inspection rule statement described in a computer language.

[0083] Use natural language processing technology to parse the quality inspection requirement statements entered by users and understand the specific requirements of data quality inspection, such as determining which fields need to be checked and what comparison criteria to use. Identify key parameters, such as statistical values, comparison values, data sources, etc. Based on the analysis results, determine the structure of the quality inspection rules, logical conditions and their corresponding specific parameters, usually including input items, comparison logic and output items. For example, "Field A cannot be empty" or "The value of Field B should be greater than 100", and then convert it into a quality inspection rule statement described in computer language.

[0084] Computer language is a language used to write programs and instructions that can be understood and executed by computers, including: languages ​​for managing and operating databases, programming languages, etc.

[0085] In some embodiments, the data quality inspection requirement statement is parsed and converted into a quality inspection rule statement described in a computer language, including: converting the data quality inspection requirement statement into an SQL statement; matching the corresponding data quality inspection rule according to the SQL statement, and filling the input parameters in the SQL statement into the data quality inspection rule to obtain the quality inspection rule statement.

[0086] Exemplarily, the corresponding SQL (Structured Query Language) statements can be automatically identified and generated by the NL2SQL algorithm tool. By writing queries in the SQL language, statistical values, comparison values, and error data can be calculated. For example, an SQL query is written to count the number of non-blank rows in a field and compare it with the comparison value. Then, by parsing the keywords in the statements in SQL, such as SELECT, etc., the corresponding quality inspection rules are matched, the required parameters are identified and filled in, and the rule statements are quality inspected.

[0087] In some embodiments, parsing a data quality inspection requirement statement and converting the data quality inspection requirement statement into an SQL statement includes: performing semantic analysis on the data quality inspection requirement statement to obtain a semantic analysis result; mapping the sentence structure of the data quality inspection requirement statement to the structure of the SQL statement based on the semantic analysis result, and replacing words in the data quality inspection requirement statement with SQL symbols to obtain an SQL statement converted from the data quality inspection requirement statement.

[0088] For example, the natural language processing library is used to parse the input data quality inspection requirement statement, including word segmentation and part-of-speech tagging, and the sentence structure is understood through syntactic analysis to extract key information. For example, intent: the operation that the user wants to perform (such as query, insert, update, etc.); entity: the table name, field name, condition value, etc. involved; condition: the condition used to filter data (such as WHERE clause).

[0089] Then, based on the parsing results, the sentence structure of the data quality inspection requirement statement can be mapped to the structure of the SQL statement, and then the intent and entities in the data quality inspection requirement statement can be mapped to the structure of the SQL statement. For example: Query intent: map the intent of "find all customers" to SELECT*FROM customers, condition mapping: map "age greater than 30 years old" to WHERE age>30. When constructing SQL statements, use the corresponding SQL symbols to replace keywords in natural language. For example: "find" replaces SELECT, "all" replaces *, "customer" replaces customers, "greater than" replaces >, and "and" replaces AND. Finally, combine the parsed and mapped parts into a complete SQL statement.

[0090] For example, the data quality inspection requirement statement is: "Find all customers older than 30 years old". The corresponding parsing result is: intent (query), table (customers), field (*), condition (age>30). The generated SQL statement is: SELECT * FROM customers WHERE age>30;

[0091] In some embodiments, before matching the corresponding data quality inspection rules according to the SQL statement, it also includes: binding the pre-configured data quality inspection rules with the instruction words expressing the intention in the SQL language to obtain the correspondence between the data quality inspection rules and the SQL instruction words; wherein each data quality inspection rule corresponds to a data quality inspection indicator.

[0092] Accordingly, the corresponding data quality inspection rules are matched according to the SQL statements, including: parsing the keywords expressing the intention in the SQL statements, and determining the target keywords as SQL instruction words; determining the target data quality inspection rules corresponding to the target keywords according to the corresponding relationship; and using the target data quality inspection rules as the data quality inspection rules corresponding to the SQL statements.

[0093] In the disclosed embodiment, by binding and configuring the pre-set data quality inspection rules and the specific instruction words expressing the intent in the SQL language, setting the correspondence between the data quality inspection rules and the SQL instruction words, and then parsing the keywords in the SQL statement, the corresponding quality inspection rules can be matched. Each data quality inspection rule corresponds to a data quality inspection indicator.

[0094] In some embodiments, the data quality inspection rules include parameter input items, which include at least data source input items, statistical value parameters, comparison value parameters and parameters related to result judgment; the data source input items are used to define the data range that needs to be quality inspected; the statistical value parameters are used to define the calculation method of the statistical value, and the statistical value is the value obtained after performing the corresponding quality inspection operation on the data that needs to be quality inspected; the comparison value parameters are used to define the calculation method or source of the comparison value, and the comparison value is the value used as a comparison target for the statistical value; the parameters related to result judgment are used to define the judgment method of whether the quality inspected data is abnormal.

[0095] Data quality inspection rules are mainly composed of two parts. The first is parameter input items: the core input items in data quality inspection rules include at least data source input items, statistical value parameters, comparison value parameters, and result judgment related parameters. The second is SQL definition: SQL needs to be defined to calculate statistical values, comparison values, and obtain error data.

[0096] The statistical value parameters are used to define the calculation method of the statistical value, and the comparison value parameters are used to define the calculation method or source of the comparison value. The statistical value refers to the value obtained after we perform a series of operations on the data to be tested, such as the total number of rows in a table or the number of rows with a certain field being empty; the comparison value refers to the value used as the comparison target. The comparison value can be a fixed value or a value calculated by a defined calculation logic. The parameters related to result judgment include the inspection method (that is, the comparison operation method of the statistical value and the comparison value), the comparison operator and the threshold set by the user, and the definition of the failure result. The parameters in this part are mainly used to define how to judge whether the data is abnormal and how to handle the abnormality.

[0097] Data quality inspection is to compare the statistical value obtained after calculating and counting the target data with the comparison value in a certain way to obtain a verification result. To facilitate user operation, the system has built-in a relatively rich set of comparison values, including but not limited to fixed values, fluctuations in the last 7 days, fluctuations in the last 30 days, etc. At the same time, users are also supported to customize comparison values. They only need to define the SQL statement used to calculate or obtain the comparison value, and the calculation method or source of the comparison value.

[0098] The disclosed embodiment uses natural language processing technology to convert data quality inspection requirements described in natural language into SQL statements, and then parses the keywords of the SQL statements. Through the correspondence between pre-configured SQL instruction words and data quality inspection rules, data quality inspection rules that match the keywords of the SQL statements are determined according to the correspondence, thereby converting the data quality inspection requirement statements described in natural language into data quality inspection rule statements described in computer language, which can lower the technical threshold of data quality inspection and make the data quality inspection solution simple and easy to use.

[0099] S103: Combine quality inspection rule statements into data quality inspection tasks.

[0100] The quality inspection rule statement corresponds to a data inspection standard, and one or more of the above steps are generated to form a quality inspection rule statement.

[0101] S104: Execute data quality inspection tasks and obtain data quality inspection results.

[0102] Figure 2 is a schematic diagram of a program architecture provided by an embodiment of the present disclosure, such as Figure 2 As shown, after the user enters the data quality inspection requirement statement described in natural language in the input interface, the main process generates a data quality inspection task based on the data quality inspection requirement statement entered by the user in natural language, and then the main process sends the data quality inspection task to the execution component for execution. After the execution component completes the data quality inspection task, it returns the task result to the main process, and the main process processes the task result to obtain the data quality inspection result.

[0103] The embodiment of the present disclosure parses the data quality inspection requirement statement after receiving the data quality inspection requirement statement described by the user in natural language, converts the data quality inspection requirement statement into a quality inspection rule statement described in computer language, and then combines the quality inspection rule statements into a data quality inspection task, executes the data quality inspection task, and obtains the data quality inspection result. The present disclosure converts the data quality inspection requirement statement described in natural language into a quality inspection rule statement described in computer language, so that the user can issue the quality inspection instructions of the data quality inspection rules and the task running requirements through simple and easy-to-understand natural language input, thereby reducing the technical threshold of data quality inspection, being simple and easy to use, and improving the versatility of the data quality inspection solution.

[0104] In some embodiments, executing a data quality inspection task and obtaining a data quality inspection result includes: sending the data quality inspection task to an execution component, using the execution component to convert the data quality inspection task into program parameters required by the data quality inspection component; transmitting the program parameters to the data quality inspection component, and having the data quality inspection component verify the corresponding data according to the program parameters to obtain a data verification result, wherein the data verification result includes statistical values ​​and comparison values ​​of the corresponding data; and determining the data quality inspection result based on the relevant parameters and the data verification result.

[0105] When the data quality inspection task starts to execute, the main process sends the task to the execution component. After receiving the task, the execution component will convert the parameter input items and SQL definitions of the data quality inspection rules into the parameters required by the data quality inspection component and pass them to the data quality inspection component for execution. The data quality inspection component parses the parameters to select the corresponding engine and constructs the corresponding operating environment and executor of the engine. At the same time, a series of reading components, conversion components and output components will be created according to the parameters. The executor will execute the logic in these components in a certain order to complete the entire data quality verification task. At the same time, one or more reading components are defined to meet the needs of different scenarios. The data output by the output component includes verification results, statistical data and error data.

[0106] For example, if the quality requirement of a piece of data is that it cannot be null and is within the threshold, it is necessary to first determine whether it is null, and then determine whether it is within the threshold, so as to complete the entire data quality verification task. After the data quality check component executes the calculation logic, calculates the statistical value and comparison value and writes it to the storage engine, it will send the task message information to the main process, which will perform the final step of result processing to determine the final data quality inspection result.

[0107] Among them, the main process is the component of the natural language processing algorithm, scheduling workflow and tasks in the system, the execution component is the component in the system responsible for actually executing tasks, and the data quality check component is the component responsible for executing computing tasks to complete actual data processing. The data quality check component is mainly composed of the execution engine (generally directly calling the existing data engine in the big data ecosystem) and the execution link component. The execution link component includes three types of components: reading component, conversion component and output component. The reading component is used to connect to the data source, the conversion component is used to execute SQL to process data, and the output component is used to output data to the specified storage.

[0108] In the process of executing the data quality inspection method of the embodiment of the present disclosure, a decentralized multi-master process and multi-execution component service architecture can be set to avoid excessive pressure on a single master process, and a task buffer queue can also be used to avoid task overload. In addition, the embodiment of the present disclosure can also divide the overall data quality inspection of massive data into different data quality inspection tasks through system processes, so that the quality inspection of massive data can be divided into multiple subtasks, and the data quality inspection tasks are distributed on each node of the cluster, thereby supporting the quality inspection of massive data.

[0109] In some embodiments, the parameters related to the result judgment include an operation formula, a comparison operator and a user-defined threshold; the data quality inspection result is determined based on the parameters related to the result judgment and the data verification result, including: filling the statistical value and comparison value in the data verification result into the operation formula to obtain the operation result; if the comparison result between the operation result and the threshold meets the comparison method defined by the comparison operator, the data quality inspection result is determined to be a data abnormality.

[0110] The calculation formula in the parameters related to result judgment indicates the inspection method, specifically the comparison method of statistical value and comparison value. For example, the calculation formula can be comparison value-statistical value, statistical value-comparison value, statistical value / comparison value, etc. It can define the type of specific data quality inspection, such as whether the data ratio of null values ​​is within the threshold. The comparison operator can be <, <=, >, >=, =, etc. The user-defined threshold can represent the standard of data quality requirements, etc.

[0111] This operation formula, comparison operator and user-defined threshold can form an expression to determine the data quality inspection result. The statistical value and comparison value can be filled into the expression to get the data quality inspection result. Assuming that the selected operation formula is comparison value-statistical value, the comparison operator is the operation formula, the statistical value is 9800, the comparison value is 10000, and the user-defined threshold is 100, then this expression means that when the difference between the comparison value and the statistical value is greater than or equal to 100, the data quality inspection result is data abnormality.

[0112] If the data quality inspection result is found to be abnormal, the corresponding failure strategy can also be executed. The embodiment of the present disclosure exemplarily provides two levels of failure strategies: 1. Alarm. The alarm level is that when the quality inspection result is abnormal, an alarm will be issued, but the task result will not be set to an error (false), and the entire workflow will not be blocked. For example, in the data quality inspection performed during data transmission, if the data quality inspection result is abnormal, the result of the entire data transmission task will not be set to an error, and the entire data transmission task will not be blocked. 2. Blocking. When the inspection result is abnormal, the first thing is to issue an alarm, and at the same time, the structure of the task will be set to an error, and the entire workflow will be blocked. When there is data abnormality, the user can view the relevant result data in the result list. The list can clearly understand the statistical values, comparison values, comparison methods and other information to help users understand the data abnormality. At the same time, the user can view the error data, clarify the data problem, and help the user to better repair the data.

[0113] Figure 3 The structure diagram of the data quality inspection device supporting natural language processing provided by the embodiment of the present disclosure is as follows. The data quality inspection device supporting natural language processing provided by the embodiment of the present disclosure can execute the processing flow provided by the data quality inspection method supporting natural language processing, such as Figure 3 As shown, the data quality inspection device 300 supporting natural language processing includes:

[0114] Input module 301, used to receive data quality inspection requirement statements described by users in natural language;

[0115] A conversion module 302, used to parse the data quality inspection requirement statement and convert the data quality inspection requirement statement into a quality inspection rule statement described in a computer language;

[0116] The task generation module 303 is used to combine the quality inspection rule statements into data quality inspection tasks;

[0117] The execution module 304 is used to execute the data quality inspection task and obtain the data quality inspection result.

[0118] In some embodiments, when the conversion module 302 parses the data quality inspection requirement statement and converts the data quality inspection requirement statement into a quality inspection rule statement described in a computer language, it is specifically used to: convert the data quality inspection requirement statement into an SQL statement; match the corresponding data quality inspection rule according to the SQL statement, and fill the input parameters in the SQL statement into the data quality inspection rule to obtain the quality inspection rule statement.

[0119] In some embodiments, when parsing a data quality inspection requirement statement and converting the data quality inspection requirement statement into an SQL statement, the conversion module 302 is specifically used to: perform semantic analysis on the data quality inspection requirement statement to obtain a semantic analysis result; based on the semantic analysis result, map the sentence structure of the data quality inspection requirement statement to the structure of the SQL statement, and replace the words in the data quality inspection requirement statement with SQL symbols to obtain an SQL statement converted from the data quality inspection requirement statement.

[0120] In some embodiments, the data quality inspection device 300 further includes a configuration module 305, which is used to bind the pre-configured data quality inspection rules with the instruction words expressing the intent in the SQL language before matching the corresponding data quality inspection rules according to the SQL statement, so as to obtain the corresponding relationship between the data quality inspection rules and the SQL instruction words; wherein each data quality inspection rule corresponds to a data quality inspection indicator;

[0121] When matching the corresponding data quality inspection rules according to the SQL statement, the conversion module 302 is specifically used to: parse the keywords expressing the intention in the SQL statement, and determine the target keywords as SQL instruction words; determine the target data quality inspection rules corresponding to the target keywords according to the corresponding relationship; and use the target data quality inspection rules as the data quality inspection rules corresponding to the SQL statement.

[0122] In some embodiments, the data quality inspection rules include parameter input items, which include at least data source input items, statistical value parameters, comparison value parameters and parameters related to result judgment; the data source input items are used to define the data range that needs to be quality inspected; the statistical value parameters are used to define the calculation method of the statistical value, and the statistical value is the value obtained after performing the corresponding quality inspection operation on the data that needs to be quality inspected; the comparison value parameters are used to define the calculation method or source of the comparison value, and the comparison value is the value used as a comparison target for the statistical value; the parameters related to result judgment are used to define the judgment method of whether the quality inspected data is abnormal.

[0123] In some embodiments, when the execution module 304 executes a data quality inspection task and obtains a data quality inspection result, it is specifically used to: send the data quality inspection task to the execution component, and use the execution component to convert the data quality inspection task into program parameters required by the data quality inspection component; transmit the program parameters to the data quality inspection component, and the data quality inspection component verifies the corresponding data according to the program parameters to obtain a data verification result, which includes the statistical value and comparison value of the corresponding data; determine the data quality inspection result based on the relevant parameters and the data verification result.

[0124] In some embodiments, the parameters related to the result determination include an operation formula, a comparison operator, and a user-defined threshold value;

[0125] When the execution module 304 determines the data quality inspection result based on the result judgment related parameters and data verification results, it is specifically used to: fill the statistical value and comparison value in the data verification result into the calculation formula to obtain the calculation result; if the comparison result of the calculation result and the threshold value meets the comparison method defined by the comparison symbol, then the data quality inspection result is determined to be a data abnormality.

[0126] Figure 3 The data quality inspection device supporting natural language processing in the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0127] Figure 4 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 4 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing the embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0128] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403 to implement the data quality inspection method supporting natural language processing according to the embodiment described in the present disclosure. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0129] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0130] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the data quality inspection method supporting natural language processing as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0131] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0132] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0133] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0134] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0135] Receive data quality inspection requirement statements described by users in natural language;

[0136] Parse data quality inspection requirement statements and convert them into quality inspection rule statements described in computer language;

[0137] Combine quality inspection rule statements into data quality inspection tasks;

[0138] Perform data quality inspection tasks and obtain data quality inspection results.

[0139] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0140] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0143] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] The embodiments of the present disclosure also provide a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any of the above embodiments can be implemented. The execution method and beneficial effects are similar and will not be repeated here.

[0146] The embodiments of the present disclosure also provide a computer program product, which is stored in a storage medium. When the program product is run, the method of any of the above embodiments can be implemented. The execution method and beneficial effects are similar and will not be repeated here.

[0147] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0148] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0149] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0150] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0151] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data quality inspection method supporting natural language processing, characterized in that: include: Receive data quality inspection requirement statements described by users in natural language; Parsing the data quality inspection requirement statement, and converting the data quality inspection requirement statement into a quality inspection rule statement described in a computer language; Combining the quality inspection rule statements into a data quality inspection task; Execute the data quality inspection task and obtain the data quality inspection result.

2. The data quality inspection method according to claim 1, characterized in that: The step of parsing the data quality inspection requirement statement and converting the data quality inspection requirement statement into a quality inspection rule statement described in a computer language includes: Convert the data quality inspection requirement statement into an SQL statement; The corresponding data quality inspection rule is matched according to the SQL statement, and the input parameters in the SQL statement are filled into the data quality inspection rule to obtain the quality inspection rule statement.

3. The data quality inspection method according to claim 2, characterized in that: The parsing of the data quality inspection requirement statement and converting the data quality inspection requirement statement into an SQL statement include: Performing semantic analysis on the data quality inspection requirement statement to obtain a semantic analysis result; According to the semantic analysis result, the sentence structure of the data quality inspection requirement statement is mapped to the structure of the SQL statement, and the words in the data quality inspection requirement statement are replaced with SQL symbols to obtain the SQL statement converted from the data quality inspection requirement statement.

4. The data quality inspection method according to claim 2, characterized in that: Before matching the corresponding data quality inspection rules according to the SQL statement, the method further includes: Bind the pre-configured data quality inspection rules with the instruction words that express the intent in the SQL language to obtain the corresponding relationship between the data quality inspection rules and the SQL instruction words; Among them, each data quality inspection rule corresponds to a data quality inspection indicator; The matching of the corresponding data quality inspection rules according to the SQL statement includes: Parsing the keywords representing the intent in the SQL statement, and determining the target keywords as SQL instruction words; Determine the target data quality inspection rule corresponding to the target keyword according to the corresponding relationship; The target data quality inspection rule is used as the data quality inspection rule corresponding to the SQL statement.

5. The data quality inspection method according to claim 2, characterized in that: The data quality inspection rules include parameter input items, which at least include data source input items, statistical value parameters, comparison value parameters, and parameters related to result judgment; The data source input item is used to define the data range that needs quality inspection; The statistical value parameter is used to define the calculation method of the statistical value. The statistical value is the value obtained after performing the corresponding quality inspection operation on the data that needs quality inspection. The comparison value parameter is used to define the calculation method or source of the comparison value, and the comparison value is a value used as a comparison target for the statistical value; The parameters related to result judgment are used to define the method of judging whether the quality inspection data is abnormal.

6. The data quality inspection method according to claim 5, characterized in that: The performing of the data quality inspection task to obtain the data quality inspection result includes: Sending the data quality inspection task to the execution component, and using the execution component to convert the data quality inspection task into program parameters required by the data quality inspection component; The program parameters are transmitted to the data quality check component, and the data quality check component verifies the corresponding data according to the program parameters to obtain a data verification result, wherein the data verification result includes a statistical value and a comparison value of the corresponding data; The data quality inspection result is determined based on the relevant parameters of the result judgment and the data verification result.

7. The data quality inspection method according to claim 5, characterized in that: Parameters related to result judgment include operation formula, comparison operator and user-defined threshold value; Determining the data quality inspection result according to the result-related parameters and the data verification result includes: Fill the statistical value and comparison value in the data verification result into the calculation formula to obtain the calculation result; If the comparison result between the operation result and the threshold value conforms to the comparison method defined by the comparison symbol, the data quality inspection result is determined to be data abnormality.

8. A data quality inspection device supporting natural language processing, characterized in that: include: An input module is used to receive data quality inspection requirement statements described by users in natural language; A conversion module, used to parse the data quality inspection requirement statement and convert the data quality inspection requirement statement into a quality inspection rule statement described in a computer language; A task generation module, used for combining the quality inspection rule statements into a data quality inspection task; The execution module is used to execute the data quality inspection task and obtain the data quality inspection result.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: Program instructions are stored thereon, and when the program instructions are executed, the method according to any one of claims 1 to 7 is implemented.

Images