Data quality detection method and device, electronic equipment and readable medium

Abstract

The present disclosure provides a data quality detection method, device, electronic equipment and readable medium, wherein the data quality detection method comprises: performing semantic splitting on a structured query language instruction used for quality detection to obtain a tree structure corresponding to the structured query language instruction, and leaves of the tree structure correspond to elements after semantic splitting; performing merging and assembling on the elements after semantic splitting according to a specified merging condition; and running the structured query language instruction after merging and assembling based on a YARN mode of Spark to obtain a data quality detection result. Through the embodiments of the present disclosure, the number of quality detection tasks can be effectively reduced, the overall quality detection efficiency is improved, larger data magnitudes and more complex rule detection scenarios are supported, and the implementation manner is flexible and variable, thereby reducing the maintenance cost of data.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and more specifically, to a data quality testing method, apparatus, electronic device, and readable medium. Background Technology

[0002] Currently, with the arrival of the big data era, facing the explosive growth in data volume, the realization of data value requires the support of its quality, making the automated detection of data quality an urgent matter.

[0003] In related technologies, a data table with dozens of fields is managed by different business departments according to different business meanings. Each department is responsible for the quality rules of the data content within its management scope. Usually, it is necessary to judge based on the description of data standards or metadata, add a quality inspection rule to a certain field in a certain database table, design the corresponding SQL statement for the rule, and then the backend system runs them one by one and generates problematic data and quality reports.

[0004] However, when the data volume is too large and the number of rules increases, the performance bottleneck of data quality detection will inevitably be encountered, resulting in low efficiency of the detection solution.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a data quality inspection method, apparatus, electronic device, and readable medium to overcome, at least to some extent, the problem of low efficiency in quality inspection caused by limitations and defects in related technologies.

[0007] According to a first aspect of the present disclosure, a data quality detection method is provided, comprising: semantically decomposing a structured query language instruction used for quality detection to obtain a tree structure corresponding to the structured query language instruction, wherein the leaves of the tree structure correspond to the elements after semantic decomposition; merging and assembling the elements after semantic decomposition according to specified merging conditions; and running the merged and assembled structured query language instruction based on Spark's YARN mode to obtain a data quality detection result.

[0008] In one exemplary embodiment of this disclosure, running merged and assembled Structured Query Language instructions based on Spark's YARN schema to obtain data quality inspection results includes:

[0009] Run the merged and assembled Structured Query Language instructions based on Spark's YARN mode to obtain multiple runtime feedback results;

[0010] The operational feedback results are split, categorized, and statistically analyzed based on the merged and assembled records to determine the data quality detection results.

[0011] In one exemplary embodiment of this disclosure, merging and assembling the semantically split elements according to specified merging conditions includes:

[0012] The specified merging condition is parsed as at least one of the following: merging condition for the same query object, merging condition for similar tree structure, and merging condition for the same intermediate data table;

[0013] Elements that have the same query object are merged and assembled according to the same query object merging conditions;

[0014] And / or, according to the similar tree structure merging condition, elements of structured query language instructions with similar tree structures are merged and assembled;

[0015] And / or, elements that have the same intermediate data table as the structured query language instructions are merged and assembled according to the same intermediate data table merging conditions.

[0016] In one exemplary embodiment of this disclosure, merging and assembling elements with the same query object according to the same query object merging condition includes:

[0017] Identify multiple elements that have the same Structured Query Language (SCL) instructions for querying the same data table;

[0018] Determine whether the tree structures corresponding to multiple structured query language commands are the same;

[0019] If it is determined that the tree structures corresponding to multiple structured query language instructions are the same, then the elements of the structured query language instructions are preferentially merged and assembled.

[0020] In one exemplary embodiment of this disclosure, merging and assembling elements of structured query language instructions with similar tree structures according to the similar tree structure merging condition includes:

[0021] Determine the semantic similarity among multiple tree structures;

[0022] The structured query language instructions corresponding to the tree structure with a semantic similarity greater than or equal to the preset semantic similarity are determined to have a similar tree structure;

[0023] Elements of multiple structured query language instructions with similar tree structures are merged and assembled.

[0024] In one exemplary embodiment of this disclosure, merging and assembling elements of structured query language instructions with similar tree structures according to the similar tree structure merging condition further includes:

[0025] Determine the complexity of a plurality of the structured query language instructions, the complexity including at least one of syntax structure complexity, lookup table complexity, and nesting level complexity;

[0026] The number of Structured Query Language instructions to be merged and assembled is determined based on the complexity.

[0027] Based on the stated quantity, elements of structured query language instructions with similar tree-like structures are merged and assembled.

[0028] In one exemplary embodiment of this disclosure, merging and assembling elements having the same intermediate data table according to the same intermediate data table merging condition includes:

[0029] Identify the intermediate data tables involved in the query processing of the structured query language instructions;

[0030] Calculate the intermediate table similarity between the intermediate data tables corresponding to multiple Structured Query Language instructions;

[0031] The structured query language instructions corresponding to the intermediate table similarity being greater than or equal to the preset intermediate table similarity are determined to have the same intermediate data table;

[0032] Elements of multiple Structured Query Language instructions that share the same intermediate data table are merged and assembled.

[0033] According to a second aspect of the present disclosure, a data quality detection apparatus is provided, comprising:

[0034] The splitting module is configured to perform semantic splitting on the structured query language instructions used for quality detection to obtain a tree structure corresponding to the structured query language instructions, wherein the leaves of the tree structure correspond to the elements after semantic splitting.

[0035] The merging module is configured to merge and assemble the semantically split elements according to specified merging conditions;

[0036] The query module is configured to run merged and assembled Structured Query Language instructions based on Spark's YARN mode to obtain data quality inspection results.

[0037] According to a third aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method as described in any of the preceding methods based on instructions stored in the memory.

[0038] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the data quality detection method as described in any of the preceding claims.

[0039] This embodiment of the disclosure semantically decomposes the structured query language instructions used for quality detection to obtain a tree structure corresponding to the structured query language instructions. The leaves of the tree structure correspond to the elements after semantic decomposition. The elements after semantic decomposition are merged and assembled according to specified merging conditions. Then, the merged and assembled structured query language instructions are run based on Spark's YARN mode to obtain data quality detection results. This can effectively reduce the number of quality detection tasks, improve the overall quality detection efficiency, support detection scenarios with larger data volumes and more complex rules, and the implementation method is flexible and versatile, reducing data maintenance costs.

[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0042] Figure 1 A schematic diagram of an exemplary system architecture for applying the data quality inspection scheme of embodiments of the present invention is shown;

[0043] Figure 2 This is a flowchart of a data quality detection method according to an exemplary embodiment of this disclosure;

[0044] Figure 3 This is a flowchart of another data quality detection method in an exemplary embodiment of this disclosure;

[0045] Figure 4 This is a flowchart of another data quality detection method in an exemplary embodiment of this disclosure;

[0046] Figure 5 This is a flowchart of another data quality detection method in an exemplary embodiment of this disclosure;

[0047] Figure 6 This is a flowchart of another data quality detection method in an exemplary embodiment of this disclosure;

[0048] Figure 7 This is a flowchart of another data quality detection method in an exemplary embodiment of this disclosure;

[0049] Figure 8 This is a flowchart of another data quality detection method in an exemplary embodiment of this disclosure;

[0050] Figure 9 This is a flowchart of another data quality detection scheme in an exemplary embodiment of this disclosure;

[0051] Figure 10 This is a schematic diagram of a tree structure of a data quality detection scheme in an exemplary embodiment of this disclosure;

[0052] Figure 11 This is a block diagram of a data quality detection apparatus according to an exemplary embodiment of this disclosure;

[0053] Figure 12 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0055] Furthermore, the accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0056] Figure 1 A schematic diagram of an exemplary system architecture for applying the data quality inspection scheme of embodiments of the present invention is shown.

[0057] like Figure 1 As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0058] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0059] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers, etc.

[0060] In some embodiments, the data quality detection method provided in this invention is generally executed by server 105, and correspondingly, the data quality detection device is generally located in terminal device 103 (or terminal device 101 or 102). In other embodiments, some terminals may have similar functions to the server device to execute this method.

[0061] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0062] Figure 2 This is a flowchart of a data quality detection method in an exemplary embodiment of this disclosure.

[0063] refer to Figure 2 Data quality testing methods may include:

[0064] Step S202: Semantically decompose the structured query language instructions used for quality detection to obtain a tree structure corresponding to the structured query language instructions, wherein the leaves of the tree structure correspond to the elements after semantic decomposition.

[0065] Step S204: Merge and assemble the semantically split elements according to the specified merging conditions.

[0066] Step S206: Run the merged and assembled Structured Query Language instructions based on Spark's YARN mode to obtain data quality detection results.

[0067] This embodiment of the disclosure semantically decomposes the structured query language instructions used for quality detection to obtain a tree structure corresponding to the structured query language instructions. The leaves of the tree structure correspond to the elements after semantic decomposition. The elements after semantic decomposition are merged and assembled according to specified merging conditions. Then, the merged and assembled structured query language instructions are run based on Spark's YARN mode to obtain data quality detection results. This can effectively reduce the number of quality detection tasks, improve the overall quality detection efficiency, support detection scenarios with larger data volumes and more complex rules, and the implementation method is flexible and versatile, reducing data maintenance costs.

[0068] The following section provides a detailed explanation of each step in the data quality inspection method.

[0069] In one exemplary embodiment of this disclosure, such as Figure 3 As shown, the data quality assessment results are obtained by running the merged and assembled Structured Query Language instructions based on Spark's YARN mode, including:

[0070] Step S302: Run the merged and assembled Structured Query Language instructions based on Spark's YARN mode to obtain multiple runtime feedback results.

[0071] In one exemplary embodiment of this disclosure, the merged quality inspection task is executed concurrently based on Spark's YARN mode, which better utilizes cluster resources for high-performance data processing and supports horizontal scaling to adapt to the ever-increasing data volume and inspection rules. In addition, it also supports multiple data sources such as Hive, HBase, and JDBC. Among them, the syntax of Hive is basically the same as that of MySQL and can support massive data storage.

[0072] Step S304: Based on the merged and assembled records, the operation feedback results are split and classified statistically analyzed to determine the data quality detection results.

[0073] In one exemplary embodiment of this disclosure, such as Figure 4 As shown, merging and assembling the semantically split elements according to the specified merging conditions includes:

[0074] Step S402: Parse the specified merging condition as at least one of the following: merging condition for the same query object, merging condition for similar tree structure, and merging condition for the same intermediate data table.

[0075] Step S404: Merge and assemble elements that have the same query object according to the same query object merging condition.

[0076] In one exemplary embodiment of this disclosure, the query object in the merging condition for the same query object can be a data table or a database, but is not limited thereto.

[0077] Step S406: Merge and assemble elements of structured query language instructions that have similar tree structures according to the similar tree structure merging conditions.

[0078] In one exemplary embodiment of this disclosure, similar tree structure merging conditions are used to define the content of leaves, the hierarchy of leaves, and branching relationships, but are not limited thereto.

[0079] Step S408: Merge and assemble elements that have the same intermediate data table instructions according to the same intermediate data table merging conditions.

[0080] In one exemplary embodiment of this disclosure, the same intermediate data table merging condition is used to limit the intermediate data tables involved in the execution of SQL.

[0081] In one exemplary embodiment of this disclosure, such as Figure 5 As shown, merging and assembling elements with the same query object according to the same query object merging condition includes:

[0082] Step S502: Identify multiple elements that have the same Structured Query Language (SCL) instructions for querying the same data table.

[0083] Step S504: Determine whether the tree structures corresponding to the multiple structured query language instructions are the same.

[0084] Step S506: If it is determined that the tree structures corresponding to multiple structured query language instructions are the same, the elements of the structured query language instructions are preferentially merged and assembled.

[0085] In one exemplary embodiment of this disclosure, by determining that the tree structure is the same, it can be determined that the SQL semantics are similar or even the same, and that they have the same query data table. Then, the number of quality inspection tasks can be reduced by merging the elements of the device architecture query language instructions.

[0086] In one exemplary embodiment of this disclosure, such as Figure 6 As shown, merging and assembling elements of structured query language instructions with similar tree structures according to the aforementioned similar tree structure merging conditions includes:

[0087] Step S602: Determine the semantic similarity between the multiple tree structures.

[0088] Step S604: Determine the structured query language instructions corresponding to the tree structure with semantic similarity greater than or equal to the preset semantic similarity as having a similar tree structure.

[0089] Step S606: Merge and assemble the elements of multiple structured query language instructions that have similar tree structures.

[0090] In one exemplary embodiment of this disclosure, by individually determining the semantic similarity between multiple tree structures and merging and assembling elements of multiple structured query language instructions with similar tree structures, the number of quality inspection tasks is effectively reduced.

[0091] In one exemplary embodiment of this disclosure, such as Figure 7 As shown, merging and assembling elements of structured query language instructions with similar tree structures according to the aforementioned similar tree structure merging conditions further includes:

[0092] Step S702: Determine the complexity of the plurality of structured query language instructions, the complexity including at least one of syntax structure complexity, lookup table complexity, and nesting layer complexity.

[0093] Step S704: Determine the number of structured query language instructions to be merged and assembled based on the complexity.

[0094] Step S706: Merge and assemble the elements of the structured query language instructions that have similar tree structures according to the quantity.

[0095] In one exemplary embodiment of this disclosure, the complexity of a plurality of structured query language instructions is determined, including at least one of syntax structure complexity, lookup table complexity, and nesting layer complexity, and the number of structured query language instructions to be merged and assembled is determined based on the complexity. The complexity and the number can be negatively correlated, that is, the higher the complexity, the fewer the number of structured query language instructions to be merged, so as to improve the accuracy of the quality detection results of high-complexity structured query language instructions while reducing the quality detection task.

[0096] In one exemplary embodiment of this disclosure, such as Figure 8 As shown, merging and assembling elements with the same intermediate data table instruction according to the same intermediate data table merging condition includes:

[0097] Step S802: Determine the intermediate data tables involved in the query processing of the structured query language instructions.

[0098] Step S804: Calculate the intermediate table similarity between the intermediate data tables corresponding to the multiple structured query language instructions.

[0099] Step S806: The structured query language instruction corresponding to the intermediate table similarity being greater than or equal to the preset intermediate table similarity is determined to have the same intermediate data table.

[0100] Step S808: Merge and assemble the elements of multiple Structured Query Language instructions that have the same intermediate data table.

[0101] In one exemplary embodiment of this disclosure, by determining the intermediate data table involved in the query processing of the structured query language instruction, and merging and assembling the elements of multiple structured query language instructions that have the same intermediate data table, the number of quality inspection tasks for the structured query language instructions is reduced, and the quality inspection efficiency is improved.

[0102] In one exemplary embodiment of this disclosure, such as Figure 9 As shown, the data quality inspection scheme includes:

[0103] Step S902: Extract the SQL rules contained in the quality inspection task.

[0104] Step S904, SQL structure parsing, that is, parsing the syntax of the SQL instance to obtain the tree structure of logical operations.

[0105] Step S906, rule matching, including merging tables with the same target database, merging tables with the same structure, merging tables with the same (similar) fields, and reusing temporary tables (intermediate tables).

[0106] Step S908, SQL merging, merges and assembles SQL elements that meet the strategy. Examples include merging tables from the same target database, merging tables with the same structure, merging tables with the same (similar) fields, and reusing temporary tables (intermediate tables). Each rule is also labeled to help split the detection results.

[0107] Step S910: Generate MR Task 1 (Rule 1 and Rule 2).

[0108] Step S912, generate MR task 2 (rule 3).

[0109] Step S914: Generate MR task 3 (rules 4, 5, and 6).

[0110] Step S916: Output or store the result.

[0111] Step S918: Analyze the execution results of the above tasks.

[0112] In one exemplary embodiment of this disclosure, combined with Figure 9 and Figure 10 As shown, the tree structure 1000 of the data quality inspection scheme includes:

[0113] The SQL query statement is broken down into multiple leaves including "select", "field", "from", "tables", "where", and "conditions", and the quality control rules are shown in Table 1 below.

[0114] Table 1

[0115]

[0116] Using SQL parsing tools, such as JSqlParse or Altlr4, it can be broken down into... Figure 10 The syntax tree shown (representing the structure after decomposing Rule 1 and Rule 2) reveals that, through step-by-step decomposition, analysis, and strategy matching, Rule 1 and Rule 2 have the same structure and both query the user_table table, requiring the retrieval of all field contents. Rule 2 and Rule 3 also have the same structure and both query the user_table table, but the former queries more fields than the latter. Rule 3 and Rule 4 query different tables, but the queried fields are the same or very similar. The UNION ALL algorithm can be used to merge the above quality inspection rules to reduce the number of quality inspection jobs. Related strategies can be configured and continuously expanded based on the above priorities, and step-by-step matching can be performed.

[0117] After strategy matching, SQL elements are merged and assembled, and labeled. During merging, the number of SQLs involved in the merging can be determined based on the complexity of the SQL structure.

[0118] In one exemplary embodiment of this disclosure, Table 2 is used as the original data information table A, and the merged result of Rule 1 and Rule 2 is used to annotate the problem rule type using the CASE WHEN structure.

[0119] Table 2 Original Data Information Table A

[0120] ID name gender phoneNo 1 Mike male 13100001234 2 Tony malee 13212340000 3 Maria female 13101020304 4 Peter 5 James 13110202233 6 Mary Female

[0121] Table 3 Test Results Table B

[0122] ID name gender phoneNo gender_null_ret gender_value_ret 2 Tony malee 13212340000 0 -1 4 Peter -1 0 5 James 13110202233 -1 0 6 Mary Female 0 -1

[0123] Table 4. Record of Questions Where Gender is Empty (C)

[0124] ID name gender phoneNo 4 Peter 5 James 13110202233

[0125] Table 5. Record of Gender Enumeration Values ​​(D)

[0126] ID name gender phoneNo 2 Tony malee 13212340000 6 Mary Female

[0127] The script program parses and splits the detected merged raw anomaly data, obtaining a problem data list for each rule as shown in Table 3, and compiling a quality summary table as shown in Tables 4 and 5. Depending on actual usage needs, it is possible to choose not to save the detection result table and problem record table, only generating the summary table. Furthermore, in actual production, the proportion of problem data in the original full data is relatively low, and the computational workload of processing the detection results using the program is far less than performing complex calculations on the original data.

[0128] Corresponding to the above method embodiments, this disclosure also provides a data quality detection device, which can be used to execute the above method embodiments.

[0129] Figure 11 This is a block diagram of a data quality inspection apparatus according to an exemplary embodiment of the present disclosure.

[0130] refer to Figure 11 The data quality inspection device 1100 may include:

[0131] The splitting module 1102 is configured to perform semantic splitting on the structured query language instructions used for quality detection to obtain a tree structure corresponding to the structured query language instructions, wherein the leaves of the tree structure correspond to the elements after semantic splitting.

[0132] The merging module 1104 is configured to merge and assemble the semantically split elements according to specified merging conditions.

[0133] The query module 1106 is configured to run merged and assembled Structured Query Language instructions in Spark-based YARN mode to obtain data quality inspection results.

[0134] In one exemplary embodiment of this disclosure, the query module 1106 is further configured as follows:

[0135] Run the merged and assembled Structured Query Language instructions based on Spark's YARN mode to obtain multiple runtime feedback results;

[0136] The operational feedback results are split, categorized, and statistically analyzed based on the merged and assembled records to determine the data quality detection results.

[0137] In one exemplary embodiment of this disclosure, the merging module 1104 is further configured to:

[0138] The specified merging condition is parsed as at least one of the following: merging condition for the same query object, merging condition for similar tree structure, and merging condition for the same intermediate data table;

[0139] Elements that have the same query object are merged and assembled according to the same query object merging conditions;

[0140] And / or, according to the similar tree structure merging condition, elements of structured query language instructions with similar tree structures are merged and assembled;

[0141] And / or, elements that have the same intermediate data table as the structured query language instructions are merged and assembled according to the same intermediate data table merging conditions.

[0142] In one exemplary embodiment of this disclosure, the merging module 1104 is further configured to:

[0143] Identify multiple elements that have the same Structured Query Language (SCL) instructions for querying the same data table;

[0144] Determine whether the tree structures corresponding to multiple structured query language commands are the same;

[0145] If it is determined that the tree structures corresponding to multiple structured query language instructions are the same, then the elements of the structured query language instructions are preferentially merged and assembled.

[0146] In one exemplary embodiment of this disclosure, the merging module 1104 is further configured to:

[0147] Determine the semantic similarity among multiple tree structures;

[0148] The structured query language instructions corresponding to the tree structure with a semantic similarity greater than or equal to the preset semantic similarity are determined to have a similar tree structure;

[0149] Elements of multiple structured query language instructions with similar tree structures are merged and assembled.

[0150] In one exemplary embodiment of this disclosure, the merging module 1104 is further configured to:

[0151] Determine the complexity of a plurality of the structured query language instructions, the complexity including at least one of syntax structure complexity, lookup table complexity, and nesting level complexity;

[0152] The number of Structured Query Language instructions to be merged and assembled is determined based on the complexity.

[0153] Based on the stated quantity, elements of structured query language instructions with similar tree-like structures are merged and assembled.

[0154] In one exemplary embodiment of this disclosure, the merging module 1104 is further configured to:

[0155] Identify the intermediate data tables involved in the query processing of the structured query language instructions;

[0156] Calculate the intermediate table similarity between the intermediate data tables corresponding to multiple Structured Query Language instructions;

[0157] The structured query language instructions corresponding to the intermediate table similarity being greater than or equal to the preset intermediate table similarity are determined to have the same intermediate data table;

[0158] Elements of multiple Structured Query Language instructions that share the same intermediate data table are merged and assembled.

[0159] Since the functions of the device 1100 have been described in detail in their respective method embodiments, they will not be repeated here.

[0160] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0161] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0162] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0163] The following reference Figure 12 To describe an electronic device 1200 according to this embodiment of the present invention. Figure 12 The electronic device 1200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0164] like Figure 12 As shown, the electronic device 1200 is manifested in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one storage unit 1220, and a bus 1230 connecting different system components (including storage unit 1220 and processing unit 1210).

[0165] The storage unit stores program code that can be executed by the processing unit 1210, causing the processing unit 1210 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1210 can perform the method shown in the embodiments of this disclosure.

[0166] Storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 12201 and / or cache memory 12202, and may further include a read-only memory (ROM) 12203.

[0167] Storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0168] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0169] Electronic device 1200 can also communicate with one or more external devices 1240 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1200, and / or any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0170] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0171] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0172] The program product for implementing the above-described method according to embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0173] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0174] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0175] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0176] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0177] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0178] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and concept of this disclosure are indicated by the claims.

Claims

1. A data quality inspection method, characterized in that, include: The structured query language instructions used for quality inspection are semantically decomposed to obtain a tree structure corresponding to the structured query language instructions, and the leaves of the tree structure correspond to the elements after semantic decomposition. Merging and assembling the semantically split elements according to specified merging conditions includes: The specified merging condition is parsed as at least one of the following: merging condition for the same query object, merging condition for similar tree structure, and merging condition for the same intermediate data table; Elements that have the same query object are merged and assembled according to the same query object merging conditions; And / or, according to the similar tree structure merging condition, elements of structured query language instructions with similar tree structures are merged and assembled; And / or, according to the same intermediate data table merging conditions, elements that have the same intermediate data table structured query language instructions are merged and assembled; The data quality assessment results are obtained by running the merged and assembled Structured Query Language instructions based on Spark's YARN mode.

2. The data quality detection method as described in claim 1, characterized in that, The data quality assessment results are obtained by running the merged and assembled Structured Query Language instructions based on Spark's YARN schema, including: Run the merged and assembled Structured Query Language instructions based on Spark's YARN mode to obtain multiple runtime feedback results; The operational feedback results are split, categorized, and statistically analyzed based on the merged and assembled records to determine the data quality detection results.

3. The data quality detection method as described in claim 1, characterized in that, Merging and assembling elements with the same query object according to the same query object merging conditions includes: Identify multiple elements that have the same Structured Query Language (SCL) instructions for querying the same data table; Determine whether the tree structures corresponding to multiple structured query language commands are the same; If it is determined that the tree structures corresponding to multiple structured query language instructions are the same, then the elements of the structured query language instructions are preferentially merged and assembled.

4. The data quality detection method as described in claim 1, characterized in that, Merging and assembling elements with similar tree structures according to the aforementioned similar tree structure merging conditions includes: Determine the semantic similarity among multiple tree structures; The structured query language instructions corresponding to the tree structure with a semantic similarity greater than or equal to the preset semantic similarity are determined to have a similar tree structure; Elements of multiple structured query language instructions with similar tree structures are merged and assembled.

5. The data quality detection method as described in claim 1, characterized in that, Merging and assembling elements with similar tree structures according to the aforementioned similar tree structure merging conditions also includes: Determine the complexity of a plurality of the structured query language instructions, the complexity including at least one of syntax structure complexity, lookup table complexity, and nesting level complexity; The number of Structured Query Language instructions to be merged and assembled is determined based on the complexity. Based on the stated quantity, elements of structured query language instructions with similar tree-like structures are merged and assembled.

6. The data quality detection method as described in claim 1, characterized in that, Merging and assembling elements that share the same intermediate data table according to the aforementioned merging conditions includes: Identify the intermediate data tables involved in the query processing of the structured query language instructions; Calculate the intermediate table similarity between the intermediate data tables corresponding to multiple Structured Query Language instructions; The structured query language instructions corresponding to the intermediate table similarity being greater than or equal to the preset intermediate table similarity are determined to have the same intermediate data table; Elements of multiple Structured Query Language instructions that share the same intermediate data table are merged and assembled.

7. A data quality detection device, characterized in that, include: The splitting module is configured to perform semantic splitting on the structured query language instructions used for quality detection to obtain a tree structure corresponding to the structured query language instructions, wherein the leaves of the tree structure correspond to the elements after semantic splitting. The merging module is configured to merge and assemble the semantically split elements according to specified merging conditions, including: The specified merging condition is parsed as at least one of the following: merging condition for the same query object, merging condition for similar tree structure, and merging condition for the same intermediate data table; Elements that have the same query object are merged and assembled according to the same query object merging conditions; And / or, according to the similar tree structure merging condition, elements of structured query language instructions with similar tree structures are merged and assembled; And / or, according to the same intermediate data table merging conditions, elements that have the same intermediate data table structured query language instructions are merged and assembled; The query module is configured to run merged and assembled Structured Query Language instructions based on Spark's YARN mode to obtain data quality inspection results.

8. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the data quality inspection method as described in any one of claims 1-6 based on instructions stored in the memory.

9. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the data quality detection method as described in any one of claims 1-6.

Images