Data quality efficient monitoring methods, devices, electronic equipment and storage media

Abstract

This application relates to the field of data monitoring technology, and provides a method, apparatus, electronic device, and storage medium for efficient data quality monitoring. The method includes: obtaining parameter information from a received monitoring request; obtaining an instance configuration table to be monitored based on the parameter information; obtaining a query statement from the instance configuration table; parsing the query statement to generate a monitoring task; and executing the monitoring task to monitor the data in the instance configuration table, thereby obtaining an execution result. This application improves the efficiency of data quality monitoring by obtaining the instance configuration table to be monitored and monitoring the instance configuration table using a monitoring task generated from the query statement in the instance configuration table.

Description

Technical Field

[0001] This application relates to the field of data management technology, specifically to a method, apparatus, electronic device, and storage medium for efficient monitoring of data quality. Background Technology

[0002] As business grows and data volume increases, data reports may involve multi-dimensional, multi-channel, and multi-period data from multiple merchants, and the number of reports is constantly increasing. Currently, there are also open-source frameworks for quality monitoring, which users can use to monitor data quality.

[0003] However, the increase in the number of reports has made the open-source quality monitoring framework too large and required to be compatible with existing service clusters, resulting in low efficiency and accuracy of data quality monitoring. Summary of the Invention

[0004] In view of the above, it is necessary to propose a method, device, electronic device and storage medium for efficient data quality monitoring. By obtaining the instance configuration table to be monitored and generating monitoring tasks based on the query statements in the instance configuration table, the efficiency of data quality monitoring is improved.

[0005] The first aspect of this application provides a method for efficient monitoring of data quality, the method comprising: obtaining parameter information from a received monitoring request; obtaining an instance configuration table to be monitored based on the parameter information; obtaining a query statement from the instance configuration table; parsing the query statement to generate a monitoring task; executing the monitoring task to monitor the data in the instance configuration table, and obtaining an execution result.

[0006] Optionally, obtaining the instance configuration table to be monitored based on the parameter information includes: obtaining the identification code of the instance to be monitored from the parameter information; and obtaining the corresponding instance configuration table according to the identification code of the instance to be monitored.

[0007] Optionally, the step of parsing the query statement to generate a monitoring task includes: calling the compiler of the data warehouse tool to parse the query statement and obtain the parsing result; generating an abstract syntax tree based on the parsing result; traversing the abstract syntax tree to obtain query blocks; converting the query blocks into logical structures, wherein the logical structures contain multiple logical operators; and converting the multiple logical operators into monitoring tasks.

[0008] Optionally, the step of calling the compiler of the data warehouse tool to parse the query statement and obtain the parsing result includes: calling the compiler of the data warehouse tool to read the code of each letter in the query statement; merging the read code according to preset rules to obtain a one-dimensional array; converting the one-dimensional array into a tree structure, and determining the tree structure as the parsing result.

[0009] Optionally, converting the query block into a logical structure includes: determining the main table, sub-table, and table attribute information between the main table and sub-table in the abstract syntax tree based on the query block; and generating the logical structure based on the main table, the sub-table, and the table attribute information between the main table and the sub-table.

[0010] Optionally, before obtaining the instance configuration table to be monitored based on the parameter information, the method further includes: obtaining the metadata to be monitored and the monitoring information corresponding to the metadata from the monitoring request; and creating the monitoring instance configuration table and the monitoring instance alarm configuration table based on the metadata and the monitoring information corresponding to the metadata.

[0011] Optionally, after the monitoring task is executed to monitor the data in the instance configuration table and obtain the execution result, the method further includes: matching the execution result with the corresponding alarm configuration result in the monitoring instance alarm configuration table; when the execution result does not match the corresponding alarm configuration result in the monitoring instance alarm configuration table, executing an alarm and outputting the obtained abnormal data.

[0012] A second aspect of this application provides a data quality high-efficiency monitoring device, the device comprising: a parameter acquisition module for acquiring parameter information from a received monitoring request; a configuration table acquisition module for acquiring an instance configuration table to be monitored based on the parameter information; a statement acquisition module for acquiring a query statement from the instance configuration table; a parsing module for parsing the query statement to generate a monitoring task; and a monitoring module for executing the monitoring task to monitor the data in the instance configuration table and obtain an execution result.

[0013] A third aspect of this application provides an electronic device including a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the aforementioned efficient data quality monitoring method.

[0014] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned efficient data quality monitoring method.

[0015] In summary, the data quality efficient monitoring method, apparatus, electronic device, and storage medium described in this application obtain the instance configuration table to be monitored through parameter information. Since the instance configuration table contains multiple configuration items, these items are considered during subsequent data quality monitoring, thereby improving the efficiency of subsequent data quality monitoring. A query statement is obtained from the instance configuration table, parsed to generate a monitoring task, and executed to monitor the data in the instance configuration table, yielding the execution result. During data quality monitoring, the monitoring task generated from the query statement in the instance configuration table monitors the instance configuration table, eliminating the need to create an open-source quality monitoring framework, thus improving the efficiency of data quality monitoring. Attached Figure Description

[0016] Figure 1 This is a flowchart of the efficient data quality monitoring method provided in Embodiment 1 of this application.

[0017] Figure 2 This is a structural diagram of the data quality high-efficiency monitoring device provided in Embodiment 2 of this application.

[0018] Figure 3 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of this application. Detailed Implementation

[0019] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0021] Example 1

[0022] Figure 1 This is a flowchart of the efficient data quality monitoring method provided in Embodiment 1 of this application.

[0023] In this embodiment, the efficient data quality monitoring method can be applied to electronic devices. For electronic devices that require efficient data quality monitoring, the functions provided by the efficient data quality monitoring method of this application can be directly integrated into the electronic device, or it can be run in the electronic device in the form of a software development kit (SDK).

[0024] like Figure 1As shown, the efficient data quality monitoring method specifically includes the following steps. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0025] 101. Obtain parameter information from the received monitoring request.

[0026] In one embodiment of this application, if the efficient data quality monitoring method is applied in the testing field, based on test-related data, when an electronic device receives a monitoring request, it needs to parse the monitoring request and obtain the message, and obtain parameter information from the message. The parameter information includes metadata to be monitored, identification code of the instance to be monitored, table name of the quality table to be monitored, user-end monitoring requirements, etc.

[0027] 102. Obtain the instance configuration table to be monitored based on the parameter information.

[0028] In one embodiment of this application, the instance configuration table refers to a MySQL data table obtained by importing business data using MySQL commands in advance.

[0029] In an optional embodiment, before obtaining the instance configuration table to be monitored based on the parameter information, the method further includes: obtaining the metadata to be monitored and the monitoring information corresponding to the metadata from the monitoring request; and creating the monitoring instance configuration table and the monitoring instance alarm configuration table based on the metadata and the monitoring information corresponding to the metadata.

[0030] In this embodiment, the monitoring instance configuration table includes multiple first monitoring configuration items; the monitoring instance alarm configuration table includes multiple second monitoring configuration items. After obtaining the monitoring instance configuration table and the monitoring instance alarm configuration table, the created monitoring instance configuration table and monitoring instance alarm configuration table are stored in the configuration center.

[0031] In this embodiment, the first monitoring configuration item includes one or more of the following combinations: monitoring instance ID; monitoring rule ID; monitoring table name; monitoring field; monitoring statement; person in charge of monitoring results; creator, creation time, updater, and whether the data table is effective, etc. The monitoring rule ID includes a foreign key and a monitoring type table rule ID; the monitoring table name can be in the format [tableA, tableB…tableN]; monitoring time range, which includes incremental monitoring time range, full monitoring time range, and time range corresponding to specified range monitoring; query statement; creator; monitoring rule contact person; person in charge of monitoring results; creation time; updater; update time; and whether the configuration data table is effective.

[0032] In this embodiment, the second monitoring configuration item includes one or more of the following combinations: alarm ID; monitoring instance ID, wherein the monitoring instance ID includes a foreign key ID and a monitoring instance table ID; alarm threshold; alarm threshold calculation type, wherein the calculation type includes greater than sign, equal sign, or equal sign, etc.; creator; creation time; updater; update time; whether the monitoring instance table is effective.

[0033] In this embodiment, the monitoring instance ID is used to uniquely identify the corresponding monitoring instance; the foreign key ID is used to uniquely identify the corresponding foreign key; the monitoring instance table ID is used to uniquely identify the corresponding monitoring instance; and the alarm ID is used to uniquely identify the corresponding alarm parameter.

[0034] In an optional embodiment, obtaining the instance configuration table to be monitored based on the parameter information includes: obtaining the identification code of the instance to be monitored from the parameter information; and obtaining the corresponding instance configuration table according to the identification code of the instance to be monitored.

[0035] In this embodiment, the identification code of the instance to be monitored is used to uniquely identify the interface of the instance to be monitored, and the corresponding instance configuration table is obtained according to the interface of the instance to be monitored.

[0036] In this embodiment, since the instance configuration table contains multiple configuration items, such as monitoring fields and the creator of the data table, the configuration items in the configuration table are taken into account when performing subsequent data quality monitoring, thereby improving the efficiency of subsequent data quality monitoring.

[0037] 103. Obtain the query statement from the instance configuration table.

[0038] In this embodiment, since the instance configuration table contains a query statement, the query statement corresponding to the instance configuration table is obtained directly.

[0039] 104. Parse the query statement to generate a monitoring task.

[0040] In this embodiment, the monitoring task refers to the table, table field, or table parameter that needs to be monitored. For example, the monitoring task is to monitor whether parameter B in table A is consistent with parameter C in table B, where parameter B and parameter C are the same indicator parameter.

[0041] In an optional embodiment, parsing the query statement to generate a monitoring task includes: calling the compiler of a data warehouse tool to parse the query statement and obtain a parsing result; generating an abstract syntax tree based on the parsing result; traversing the abstract syntax tree to obtain query blocks; converting the query blocks into logical structures, wherein the logical structures contain multiple logical operators; and converting the multiple logical operators into monitoring tasks.

[0042] In this embodiment, the compiler of the data warehouse tool can be a parser. The parser performs syntactic and semantic analysis on the query statement, expands the view in the query, and divides it into small query blocks. Each query block is the most basic unit of a SQL statement, containing the input source, calculation process, and output. An abstract syntax tree is generated based on the divided query blocks, the syntactic analysis results, and the lexical analysis results.

[0043] Specifically, the process of generating query blocks from the abstract syntax tree is a recursive process. By creating query block objects, the child nodes are recursively traversed, and the syntax of the table name is saved to the aliasToTabs and other properties of the query block object. The syntax of the output target is saved to the nameToDest property of the QBParseInfo object, and the syntax of the query expression is saved to the destToSelExpr, destToAggregationExprs, and destToDistinctFuncExprs properties respectively. The syntax of the Where part is saved to the destToWhereExpr property of the QBParseInfo object.

[0044] In this embodiment, after obtaining the query block and the corresponding logical result, a query optimizer is used to generate a set of monitoring tasks that may be used for the query statement.

[0045] Furthermore, the process of calling the compiler of the data warehouse tool to parse the query statement and obtain the parsing result includes: calling the compiler of the data warehouse tool to read the code of each letter in the query statement; merging the read code according to preset rules to obtain a one-dimensional array; converting the one-dimensional array into a tree structure, and determining the tree structure as the parsing result.

[0046] In this embodiment, a syntax parser is used to read the code of each letter in the query statement one by one. The code is then compared with predefined JavaScript keywords to generate corresponding code. This code is an indivisible minimum unit and can include each keyword, identifier, operator, or punctuation mark. For example, the four characters "keep" as a whole cannot be semantically broken down further; therefore, it is a single piece of code.

[0047] In this embodiment, the query statement can be divided into a one-dimensional array.

[0048] Further, converting the query block into a logical structure includes: determining the main table, sub-table, and table attribute information between the main table and sub-table in the abstract syntax tree based on the query block; and generating the logical structure based on the main table, the sub-table, and the table attribute information between the main table and the sub-table.

[0049] In this embodiment, the table attribute information between the main table and the sub-table includes the table names, fields and values ​​of the main table and the sub-table, as well as the correspondence between the main table and the sub-table; a logical structure can be generated based on the main table, the sub-table, the table names, fields and values ​​of the main table and the sub-table, as well as the correspondence between the main table and the sub-table.

[0050] 105. Execute the monitoring task to monitor the data in the instance configuration table and obtain the execution result.

[0051] In this embodiment, the execution result is obtained by monitoring each monitoring task.

[0052] In an optional embodiment, after the monitoring task is executed to monitor the data in the instance configuration table and obtain the execution result, the method further includes: matching the execution result with the corresponding alarm configuration result in the monitoring instance alarm configuration table; when the execution result does not match the corresponding alarm configuration result in the monitoring instance alarm configuration table, executing an alarm and outputting the obtained abnormal data.

[0053] In this embodiment, the alarm configuration table of the monitoring instance stores the alarm threshold for each alarm parameter. The alarm threshold is preset according to the actual monitoring scenario, and this embodiment does not impose any restrictions on it.

[0054] In other optional embodiments, the method further includes: responding to a received new monitoring data instruction; parsing the monitoring data instruction to obtain the monitoring data and the identification code of the configuration table of the monitoring data; and configuring the monitoring data into the target configuration table corresponding to the identification code according to a preset rule.

[0055] In this embodiment, the identification code of the configuration table is used to uniquely identify the identity of the configuration table. For example, if the identification code of the configuration table is M, the corresponding configuration table is identified as a monitoring instance configuration table; if the identification code of the configuration table is N, the corresponding configuration table is identified as a monitoring instance alarm configuration table; if the identification code of the configuration table is P, the corresponding configuration table is identified as both a monitoring instance configuration table and a monitoring instance alarm configuration table.

[0056] In this embodiment, when adding new monitoring data, the new monitoring data is written into the corresponding target configuration table, and data quality monitoring is performed based on the target configuration table. Configuration data can be added or removed at any time, and monitoring tasks generated directly from the query statements in the instance configuration table can monitor the instance configuration table. This can achieve data quality monitoring, solve the problems of high cost and low efficiency caused by traditional open-source quality monitoring frameworks, and improve the efficiency of data quality monitoring.

[0057] In addition, the flexible configuration of the configuration table and the clear identification of monitoring instances and responsible persons improve the maintenance efficiency of data quality monitoring.

[0058] In summary, the efficient data quality monitoring method described in this embodiment obtains the instance configuration table to be monitored through parameter information. Since the instance configuration table contains multiple configuration items, these items are considered during subsequent data quality monitoring, thereby improving the efficiency of subsequent data quality monitoring. A query statement is obtained from the instance configuration table, parsed to generate a monitoring task, and executed to monitor the data in the instance configuration table, yielding the execution result. During data quality monitoring, the monitoring task generated from the query statement in the instance configuration table monitors the instance configuration table, eliminating the need to create an open-source quality monitoring framework, thus improving the efficiency of data quality monitoring.

[0059] Example 2

[0060] Figure 2 This is a structural diagram of the data quality high-efficiency monitoring device provided in Embodiment 2 of this application.

[0061] In some embodiments, the data quality high-efficiency monitoring device 20 may include multiple functional modules composed of program code segments. The program code of each program segment in the data quality high-efficiency monitoring device 20 may be stored in the memory of an electronic device and executed by the at least one processor to perform (see details). Figure 1 (Description) Functions of efficient data quality monitoring methods.

[0062] In this embodiment, the data quality high-efficiency monitoring device 20 can be divided into multiple functional modules according to its functions. These functional modules may include: a parameter acquisition module 201, a configuration table acquisition module 202, a statement acquisition module 203, a parsing module 204, and a monitoring module 205. The term "module" in this application refers to a series of computer-readable instruction segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0063] The parameter acquisition module 201 is used to obtain parameter information from the received monitoring request.

[0064] The configuration table acquisition module 202 is used to acquire the configuration table of the instance to be monitored based on the parameter information.

[0065] The statement acquisition module 203 is used to acquire query statements from the instance configuration table.

[0066] The parsing module 204 is used to parse the query statement to generate a monitoring task.

[0067] The monitoring module 205 is used to perform the monitoring task to monitor the data in the instance configuration table and obtain the execution result.

[0068] In an optional embodiment, before the configuration table acquisition module 202 acquires the configuration table of the instance to be monitored based on the parameter information, the metadata to be monitored and the monitoring information corresponding to the metadata are acquired from the monitoring request; the monitoring instance configuration table and the monitoring instance alarm configuration table are created based on the metadata and the monitoring information corresponding to the metadata.

[0069] In an optional embodiment, the configuration table acquisition module 202 is used to: obtain the identification code of the instance to be monitored from the parameter information; and obtain the corresponding instance configuration table according to the identification code of the instance to be monitored.

[0070] In an optional embodiment, the parsing module 204 is configured to: call the compiler of the data warehouse tool to parse the query statement and obtain the parsing result; generate an abstract syntax tree based on the parsing result; traverse the abstract syntax tree to obtain query blocks; convert the query blocks into logical structures, wherein the logical structures contain multiple logical operators; and convert the multiple logical operators into monitoring tasks.

[0071] Furthermore, the process of calling the compiler of the data warehouse tool to parse the query statement and obtain the parsing result includes: calling the compiler of the data warehouse tool to read the code of each letter in the query statement; merging the read code according to preset rules to obtain a one-dimensional array; converting the one-dimensional array into a tree structure, and determining the tree structure as the parsing result.

[0072] Further, converting the query block into a logical structure includes: determining the main table, sub-table, and table attribute information between the main table and sub-table in the abstract syntax tree based on the query block; and generating the logical structure based on the main table, the sub-table, and the table attribute information between the main table and the sub-table.

[0073] In an optional embodiment, after the monitoring module 205 executes the monitoring task to monitor the data in the instance configuration table and obtains the execution result, the method further includes: matching the execution result with the corresponding alarm configuration result in the monitoring instance alarm configuration table; when the execution result does not match the corresponding alarm configuration result in the monitoring instance alarm configuration table, executing an alarm and outputting the obtained abnormal data.

[0074] In other optional embodiments, in response to the received new monitoring data instruction; the monitoring data instruction is parsed to obtain the monitoring data and the identification code of the configuration table of the monitoring data; and the monitoring data is configured into the target configuration table corresponding to the identification code according to preset rules.

[0075] In this embodiment, the identification code of the configuration table is used to uniquely identify the identity of the configuration table. For example, if the identification code of the configuration table is M, the corresponding configuration table is identified as a monitoring instance configuration table; if the identification code of the configuration table is N, the corresponding configuration table is identified as a monitoring instance alarm configuration table; if the identification code of the configuration table is P, the corresponding configuration table is identified as both a monitoring instance configuration table and a monitoring instance alarm configuration table.

[0076] In this embodiment, when adding new monitoring data, the new monitoring data is written into the corresponding target configuration table, and data quality monitoring is performed based on the target configuration table. Configuration data can be added or removed at any time, and monitoring tasks generated directly from the query statements in the instance configuration table can monitor the instance configuration table. This can achieve data quality monitoring, solve the problems of high cost and low efficiency caused by traditional open-source quality monitoring frameworks, and improve the efficiency of data quality monitoring.

[0077] In addition, the flexible configuration of the configuration table and the clear identification of monitoring instances and responsible persons improve the maintenance efficiency of data quality monitoring.

[0078] In summary, the data quality monitoring device described in this embodiment obtains the instance configuration table to be monitored through parameter information. Since the instance configuration table contains multiple configuration items, these items are considered during subsequent data quality monitoring, thereby improving the efficiency of subsequent data quality monitoring. A query statement is obtained from the instance configuration table, parsed to generate a monitoring task, and executed to monitor the data in the instance configuration table, yielding the execution result. During data quality monitoring, the monitoring task generated from the query statement in the instance configuration table monitors the instance configuration table, eliminating the need to create an open-source quality monitoring framework, thus improving the efficiency of data quality monitoring.

[0079] Example 3

[0080] See Figure 3The diagram shown is a structural schematic of an electronic device provided in Embodiment 3 of this application. In a preferred embodiment of this application, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0081] Those skilled in the art should understand that Figure 3 The structure of the electronic device shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The electronic device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0082] In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 3 may also include client devices, including, but not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablet computers, smartphones, and digital cameras.

[0083] It should be noted that the electronic device 3 is merely an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0084] In some embodiments, the memory 31 is used to store program code and various data, such as the data quality high-efficiency monitoring device 20 installed in the electronic device 3, and to achieve high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0085] In some embodiments, the at least one processor 32 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The at least one processor 32 is the control unit of the electronic device 3, connecting various components of the entire electronic device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data of the electronic device 3.

[0086] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc.

[0087] Although not shown, the electronic device 3 may also include a power supply (such as a battery) to power the various components. Optionally, the power supply may be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0088] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0089] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of this application.

[0090] In a further embodiment, combined with Figure 2 The at least one processor 32 can execute the operating device of the electronic device 3 and various installed applications (such as the data quality high-efficiency monitoring device 20), program code, etc., for example, the various modules mentioned above.

[0091] The memory 31 stores program code, and the at least one processor 32 can call the program code stored in the memory 31 to execute related functions. For example, Figure 2 The modules described herein are program codes stored in the memory 31 and executed by the at least one processor 32, thereby realizing the functions of the modules to achieve the purpose of a data quality efficient monitoring method.

[0092] For example, the program code can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete this application. The one or more modules / units can be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the program code in the electronic device 3. For example, the program code can be divided into a parameter acquisition module 201, a configuration table acquisition module 202, a statement acquisition module 203, a parsing module 204, and a monitoring module 205.

[0093] In one embodiment of this application, the memory 31 stores a plurality of computer-readable instructions, which are executed by the at least one processor 32 to implement the function of a data quality efficient monitoring method.

[0094] Specifically, the specific implementation method of the above instructions by the at least one processor 32 can be found in [reference]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0096] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0098] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or, and the singular does not exclude the plural. Multiple elements or devices recited in this application may also be implemented by a single element or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for efficient monitoring of data quality, characterized in that, The method includes: Obtain parameter information from the received monitoring request; Based on the parameter information, obtain the instance configuration table to be monitored; Retrieve the query statement from the instance configuration table; Parsing the query statement to generate a monitoring task includes: calling the compiler of a data warehouse tool to parse the query statement and obtain the parsing result; generating an abstract syntax tree based on the parsing result; traversing the abstract syntax tree to obtain query blocks; converting the query blocks into logical structures, wherein the logical structures contain multiple logical operators; and converting the multiple logical operators into the monitoring task. The monitoring task is executed to monitor the data in the instance configuration table and the execution result is obtained.

2. The data quality efficient monitoring method as described in claim 1, characterized in that, The process of obtaining the instance configuration table to be monitored based on the parameter information includes: Obtain the identification code of the instance to be monitored from the parameter information; Obtain the corresponding instance configuration table based on the identification code of the instance to be monitored.

3. The data quality efficient monitoring method as described in claim 1, characterized in that, The compiler of the data warehouse tool parses the query statement, and the parsing results include: The compiler of the data warehouse tool is invoked to read the code of each letter in the query statement; The read code is merged according to preset rules to obtain a one-dimensional array; The one-dimensional array is converted into a tree structure, and the tree structure is determined as the parsing result.

4. The data quality efficient monitoring method as described in claim 1, characterized in that, The step of converting the query block into a logical structure includes: Based on the query block, determine the main table, sub-table, and table attribute information between the main table and sub-table in the abstract syntax tree; The logical structure is generated based on the main table, the sub-table, and the table attribute information between the main table and the sub-table.

5. The data quality efficient monitoring method as described in claim 1, characterized in that, Before obtaining the instance configuration table to be monitored based on the parameter information, the method further includes: Obtain the metadata to be monitored and the monitoring information corresponding to the metadata from the monitoring request; The monitoring instance configuration table and the monitoring instance alarm configuration table are created based on the metadata and the corresponding monitoring information.

6. The data quality efficient monitoring method as described in claim 5, characterized in that, After the monitoring task is executed to monitor the data in the instance configuration table and the execution result is obtained, the method further includes: The execution result is matched with the corresponding alarm configuration result in the monitoring instance alarm configuration table; When the execution result does not match the alarm configuration result in the monitoring instance alarm configuration table, an alarm is triggered and the acquired abnormal data is output.

7. A data quality high-efficiency monitoring device, characterized in that, The device includes: The parameter acquisition module is used to obtain parameter information from the received monitoring requests; The configuration table acquisition module is used to acquire the configuration table of the instance to be monitored based on the parameter information. The statement retrieval module is used to retrieve query statements from the instance configuration table; The parsing module is used to parse the query statement to generate a monitoring task, including: calling the compiler of the data warehouse tool to parse the query statement and obtain the parsing result; generating an abstract syntax tree based on the parsing result; traversing the abstract syntax tree to obtain query blocks; converting the query blocks into a logical structure, wherein the logical structure contains multiple logical operators; and converting the multiple logical operators into the monitoring task. The monitoring module is used to perform the monitoring task to monitor the data in the instance configuration table and obtain the execution result.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the efficient data quality monitoring method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the efficient data quality monitoring method as described in any one of claims 1 to 6.

Images