A data quality inspection method and apparatus

Abstract

This invention provides a data quality inspection method and apparatus. The method includes: receiving an inspection request sent by a client and transmitted via a first server; the inspection request is generated based on a functional instruction input by a user on the client; constructing entity class inspection rules corresponding to the functional instruction using the builder pattern based on an inspection rule template corresponding to the functional instruction, so as to perform data quality inspection on the data to be inspected according to the entity class inspection rules. This method can improve the flexibility of inspection rule settings and reduce the development cycle and cost of inspection rule extensions.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data quality detection method and apparatus. Background Technology

[0002] In recent years, with the development of computer technology, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually transforming into Fintech. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology. For example, based on computer technology, most financial business operations are processed online, accelerating business processing efficiency, and consequently, the amount of data generated is increasing exponentially. Furthermore, the addition of more business processing functions further amplifies the data volume. Consequently, the generation of massive amounts of data makes data quality inspection technology more challenging.

[0003] In existing technologies, data quality inspection services typically employ various methods, such as setting inspection rules (e.g., cross-table inspection rules, single-table inspection rules, file inspection rules, and multi-indicator inspection rules). While this approach allows for the separate inspection of data based on different rules to obtain data quality inspection results, the fixed nature of these rules means that adding new rules requires significant development work, increasing development time and costs.

[0004] Therefore, there is an urgent need for a data quality inspection method that can improve the flexibility of inspection rule settings and reduce the development cycle and cost of inspection rule expansion. Summary of the Invention

[0005] This invention provides a data quality detection method that can improve the flexibility of detection rule settings and reduce the development cycle and cost of expanding detection rules.

[0006] In a first aspect, embodiments of the present invention provide a data quality detection method, the method comprising:

[0007] Receive a detection request sent by the client and transmitted via the first server; the detection request is generated based on a functional command input by the user on the client.

[0008] Based on the detection rule template corresponding to the functional instruction, the entity class detection rule corresponding to the functional instruction is constructed using the builder pattern, so as to perform data quality detection on the data to be detected according to the entity class detection rule.

[0009] In the above method, the data quality inspection service receives the inspection request sent by the client through a first server, determines the inspection rule template corresponding to the functional instruction in the inspection request, and constructs the entity class inspection rule corresponding to the functional instruction using the builder pattern. Thus, compared to the prior art where the inspection rules are fixedly written in the code of the data quality inspection service, this application decouples the inspection rules from the data quality inspection service by setting the inspection rules as inspection rule templates and corresponding functional instructions. Entity class inspection rules can be updated by changing or adding functional instructions / function parameter values, improving the flexibility of the inspection rules and reducing the development cycle and cost of inspection rule extensions.

[0010] Optionally, the correspondence between each functional instruction and each detection rule template is predefined through an enumeration class; based on the detection rule template corresponding to the functional instruction, the entity class detection rule corresponding to the functional instruction is constructed using the builder pattern, including:

[0011] The detection rule template corresponding to the functional instruction is determined by the enumeration class.

[0012] The entity class detection rules corresponding to the functional instruction are constructed by combining the function parameter values ​​in the functional instruction with the abstract classes in the detection rule template corresponding to the functional instruction using the builder pattern; each abstract class includes basic library table attributes, data indicator attributes, abnormal data threshold attributes, and return attributes.

[0013] In the above method, the correspondence between each functional instruction and each detection rule template is pre-defined through an enumeration class. This allows for the rapid determination of the corresponding detection rule template based on the functional instruction in the detection request, thus generating entity class detection rules. The detection rule template contains rule attributes for multiple abstract classes. Therefore, the data quality detection service can set multiple attribute items from the functional instruction into the rule attributes of the corresponding abstract classes to generate entity class detection rules, making the entity class detection rule generation method simple and fast. Specifically, the basic database table attribute is used by the entity class detection rule to obtain the data to be detected (e.g., database table name); the data indicator attribute is used by the entity class detection rule to perform corresponding indicator detection on the data to be detected (e.g., number of primary key null values); the abnormal data threshold attribute is used by the entity class detection rule to determine whether the amount of abnormal data exceeds the threshold (e.g., if the number of primary key null values ​​is greater than 0, a notification is generated to the client); if the threshold is exceeded, an alarm or notification is generated; and the return attribute is used to return the detection result.

[0014] Optionally, after receiving the detection request sent by the client and transmitted by the first server, and before constructing the entity class detection rules corresponding to the functional instructions using the builder pattern, the method further includes:

[0015] The signature in the detection request is decompressed to obtain a second concatenation field, wherein the second concatenation field includes an access token and a compression field;

[0016] The compressed field is decompressed to obtain a first concatenated field, wherein the first concatenated field includes an application identifier, a random number, and a timestamp;

[0017] Verify the application identifier, random number, timestamp, and access token;

[0018] After successful verification, the step of constructing the entity class detection rules corresponding to the functional instructions using the builder pattern is executed.

[0019] In the above method, the data quality inspection service and the intermediate service—that is, in this solution—data transmission between services can be performed using signature verification to ensure data transmission security. The signature contains information such as application identifier, random number, timestamp, and access token. During transmission, the signature can be compressed. The application identifier, random number, and timestamp are concatenated to obtain a first concatenation field. This first concatenation field is compressed to obtain a compressed field. This compressed field is then concatenated with the access token to obtain a second concatenation field. The second concatenation field is then compressed to obtain the compressed signature. In other words, transmitting the signature in compressed form increases data transmission security while reducing the amount of data transmitted, achieving simple and fast verification of secure data transmission.

[0020] Optionally, the detection request is generated based on functional instructions entered by the user on the client, including:

[0021] Based on the functional instructions input by the user on the client, the client obtains the function parameter values ​​of each functional instruction from the configuration file, thereby obtaining the detection rule function and sending it to the first server;

[0022] The first server encapsulates the detection rule function into the detection request.

[0023] In the above method, the data quality inspection service receives functional instructions from the client, retrieves corresponding function parameter values ​​from the configuration file based on the functional instructions, and encapsulates and obtains an inspection request according to the functional instructions and function parameter values. Thus, compared to existing technologies where inspection rules are fixedly written in the code of the data quality inspection service, and the server directly determines the inspection rules based on the inspection request transmitted by the client, this application decouples the function parameter values ​​and functional instructions by setting the inspection request as a functional instruction and corresponding function parameter values ​​in the configuration file. The specific function of the inspection rule can be updated by changing or adding function parameter values, thereby improving the flexibility of the inspection rules and reducing the development cycle and cost of extending the inspection rules.

[0024] Optionally, it also includes updating the values ​​of each function parameter in the configuration file based on the configuration change command entered by the user on the client.

[0025] In the above method, the configuration change command is received by the client to update the parameters in the configuration file. This allows for a simple and quick modification of the function parameter values ​​of the functional commands for detection rules.

[0026] Optionally, after constructing the entity class detection rules corresponding to the functional instructions using the builder pattern, the method further includes:

[0027] The second server whose idle resources meet the entity class detection rules is determined from among the servers.

[0028] The entity class detection rules are sent to the second server; the second server is used to detect the data to be detected according to the entity class detection rules.

[0029] The detection results from the second server are obtained and displayed by the client.

[0030] In the above method, the entity class detection rules are run on the server with the most idle resources among all servers to obtain the detection results. This speeds up the detection process and improves resource utilization.

[0031] Optional, also includes:

[0032] The interface is configured to receive template extension instructions; the interface contains the abstract classes required to generate the detection rule template.

[0033] A new detection rule template is generated based on the template extension instruction, and a new functional instruction corresponding to the new detection rule template is set in the enumeration class. The function parameter value of the new functional instruction is added to the configuration file.

[0034] In the above method, template extension instructions are received through the setting interface in the first server, new detection rule templates are generated based on the template extension instructions, and new functional instructions corresponding to the new detection rule templates are added to the enumeration class by the client, and the function parameter values ​​of the new functional instructions are added to the configuration file. In this way, detection rule extension can be easily and quickly implemented by simply generating new detection rule templates on the server and generating new functional instructions and corresponding function parameter values ​​on the client using template extension instructions.

[0035] Secondly, embodiments of the present invention provide a data quality detection device, the device comprising:

[0036] The transceiver module is used to receive detection requests sent by the client and transmitted by the first server; the detection requests are generated based on functional instructions input by the user on the client.

[0037] The processing module is used to construct entity class detection rules corresponding to the functional instructions using the builder pattern based on the detection rule template corresponding to the functional instructions, so as to perform data quality detection on the data to be detected according to the entity class detection rules.

[0038] Thirdly, embodiments of this application also provide a computing device, including: a memory for storing a program; and a processor for calling the program stored in the memory and executing the method described in various possible designs of the first aspect according to the obtained program.

[0039] Fourthly, embodiments of this application also provide a computer-readable non-volatile storage medium including a computer-readable program that, when read and executed by a computer, causes the computer to perform the method described in various possible designs of the first aspect.

[0040] These or other implementations of this application will become clearer and easier to understand in the following description of the embodiments. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the architecture of a data quality detection system provided in an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating a data quality detection method provided in an embodiment of the present invention.

[0044] Figure 3 This is a flowchart illustrating a data quality detection method provided in an embodiment of the present invention.

[0045] Figure 4 This is a schematic diagram of a data quality detection device provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0047] Figure 1This invention provides a system architecture for a data quality inspection system. The client 101 includes a function list of functional instructions. This function list is a visual function list (which can be considered a function list of detection rule semantic functions, including functions such as null value detection and primary key detection). Users can copy the functional instructions (which can also be considered detection rule semantic functions) from this function list and input them into the command entry point. Thus, developers can directly input functional instructions without opening the visual function list. If the user is a business person or other person unfamiliar with functional instructions (each functional instruction in the function list has corresponding explanatory text to help business personnel identify the copied instruction; for example, the description for the null value detection functional instruction is: null value detection, whether to delete failed indicator values, whether to upload indicator values, whether to upload abnormal indicators, whether to block, whether to specify execution parameters), they can input the functional instructions by copying them from the visual function list. Therefore, compared to existing technical solutions that use a set of elements in the human-computer interaction interface or command line (functional instructions) to handle tasks such as configuration, execution, and problem discovery for monitoring, this approach lacks the coexistence of human-computer interaction and command line interaction (i.e., in existing technologies, the input method for functional instructions and corresponding function parameter values ​​is either by selecting the functional instruction and corresponding function parameter value in the human-computer interaction interface or by manually inputting the functional instruction and corresponding function parameter value). This single entry point hinders communication, tracking, and resolution of data quality issues among different roles such as business personnel, developers, and operations personnel, and makes it difficult to improve the familiarity and understanding of functional instructions and functions among various personnel in the short term. In contrast, this application displays a function list through a visual interface, making the functional instructions in the function list copyable and providing corresponding explanations. This allows for convenient and rapid execution of entity detection rules and acquisition of detection results for various users.

[0048] After receiving the functional instruction, client 101 generates a function instruction based on the corresponding function parameter values ​​in the configuration file. For ease of understanding, it can be considered that client 101 receives the detection rule semantic function, determines the corresponding function parameter values ​​in the configuration file, and obtains the detection rule function. Client 101 sends the function instruction to the first server in server cluster 102 (the first server can be any server in server cluster 102, without specific restrictions here). After receiving the function instruction, the first server generates a detection request containing a signature based on the function instruction and information such as application identifier, timestamp, random value, and access token. Then, the data quality detection service in the first server verifies the signature in the detection request. If the verification passes, it determines the detection rule template corresponding to the functional instruction based on the pre-defined correspondence between the functional instructions in the enumeration class and the detection rule template. Based on the function parameter values ​​of the functional instruction and the abstract classes in the detection rule template, it uses the builder pattern to set the corresponding function parameter values ​​into the abstract classes, constructing the entity class detection rule corresponding to the functional instruction. The first server generates a detection allocation request based on entity class detection rules and a signature generated from information such as application identifier, timestamp, random value, and access token. This request is then sent to the intermediate service. The intermediate service verifies the signature in the detection allocation request. If the verification is successful, it determines the available resources of each server in server cluster 102 (including the first and second servers), identifies the second server with the largest available resource, and sends the entity class detection rules to it. Upon receiving the entity class detection rules, the second server runs them. Based on the data source parameters specified in the entity class detection rules, it retrieves the data to be detected from database 103 (database 103 contains multiple databases, which can be of various types, such as Spark databases, Hive databases, etc., without specific restrictions). It then performs detection on the data according to the entity class detection rules and obtains the detection results. The detection results are stored offline in the database containing the data to be detected and sent to the intermediate service of the first server. After receiving the detection results, the intermediate service determines that the detection task of the entity class detection rule is completed and sends the detection results to the data quality detection service of the first server. The data quality detection service of the first server verifies the detection results to determine whether the data quality in the detection results is unqualified. If it is unqualified, an alarm or notification is generated and sent to the client to be displayed to the user.

[0049] Based on this, embodiments of this application provide a flow chart for a data quality detection method, such as... Figure 2 As shown, it includes:

[0050] Step 201: Receive a detection request sent by the client and transmitted via the first server; the detection request is generated based on the functional instructions input by the user on the client.

[0051] Step 202: Based on the detection rule template corresponding to the functional instruction, construct the entity class detection rule corresponding to the functional instruction using the builder pattern, so as to perform data quality detection on the data to be detected according to the entity class detection rule.

[0052] In the above method, the data quality inspection service receives the inspection request sent by the client through a first server, determines the inspection rule template corresponding to the functional instruction in the inspection request, and constructs the entity class inspection rule corresponding to the functional instruction using the builder pattern. Thus, compared to the prior art where the inspection rules are fixedly written in the code of the data quality inspection service, this application decouples the inspection rules from the data quality inspection service by setting the inspection rules as inspection rule templates and corresponding functional instructions. Entity class inspection rules can be updated by changing or adding functional instructions / function parameter values, improving the flexibility of the inspection rules and reducing the development cycle and cost of inspection rule extensions.

[0053] This application provides a method for constructing entity class detection rules. The correspondence between each functional instruction and each detection rule template is pre-defined through an enumeration class. Based on the detection rule template corresponding to the functional instruction, the entity class detection rules corresponding to the functional instruction are constructed using the builder pattern. This includes: determining the detection rule template corresponding to the functional instruction through the enumeration class; constructing the entity class detection rules corresponding to the functional instruction by associating each function parameter value in the functional instruction with each abstract class in the detection rule template corresponding to the functional instruction using the builder pattern; each abstract class includes basic library table attributes, data indicator attributes, abnormal data threshold attributes, and return attributes.

[0054] In other words, the correspondence between each functional instruction and each detection rule template is maintained through an enumeration class, such as EXPECT_SQL_PASS(code: 0L, name: "expectSqlPass"), EXPECT_FILE_SIZE_PASS(code: 0L, name: "expectFileSizePass"), and EXPECT_FILE_AMOUNT_COUNT(code: 0L, name: "expectFileAmountCount"), etc. Among them, EXPECT_SQL_PASS, EXPECT_FILE_SIZE_PASS, and EXPECT_FILE_AMOUNT_COUNT are detection rule templates, while expectSqlPass, expectFileSizePass, and expectFileAmountCount are functional instructions. In one example, the functional instruction: expectColumnNotNull corresponds to the detection rule template: EXPECT_CLOUMN_NOT_NULL. EXPECT_CLOUMN_NOT_NULL corresponds to the constructor AddRuleRequestBuilder. EXPECT_CLOUMN_NOT_NULL contains four abstract classes: basic database table attributes, data indicator attributes, and abnormal data threshold attributes.

[0055] In this embodiment of the invention, the specific process of constructing the entity class detection rule corresponding to the functional instruction using the builder pattern is as follows:

[0056] The constructor sets the function parameter values ​​from the functional instructions to the four abstract classes of EXPECT_CLOUMN_NOT_NULL, thus obtaining the entity class detection rule functions. In one example, there can be at least four types of functional instructions: functional instructions for single-table validation, functional instructions for cross-table comparison, functional instructions for file validation, and functional instructions for multi-index validation. These four types of functional instructions can correspond to their respective constructors. For example, the AddRuleRequestBuilder constructor constructs the detection rule template for functional instructions for single-table validation, the AddCustomRequestBuilder constructor constructs the detection rule template for functional instructions for cross-table comparison, the AddFileRequestBuilder constructor constructs the detection rule template for functional instructions for file validation, and the AddMultiRequestBuilder constructor constructs the detection rule template for functional instructions for multi-index validation. It should be noted that the types of functional instructions and their corresponding detection rule templates, the functional instructions and constructors corresponding to the detection rule templates, the number and types of abstract classes they include, the types of constructors and their corresponding detection rule templates, and the types of abstract classes in this example are not specifically limited and can be set as needed. In addition, the core work of data quality inspection service is the parsing and execution of functional instructions. The data quality inspection service can use the relevant Builder functional call SDK (Software Development Kit) to construct the real inspection rule code by adding the attribute values ​​corresponding to various abstract classes to the abstract classes through the builder pattern and abstract classes, and complete the creation of rules.

[0057] In another example, in the interface provided to the data quality inspection service, a function instruction containing the parameters: expectColumnNotNull(function instruction name), addRuleMetric(metric name).fixValueEqual(50(abnormal data threshold)), returnRequest(return type), and basicinfoWithDataSource(data source identifier) ​​is passed in. JEXL (a reflection mechanism) will be used to execute the following code: AddDirecotr.expectColumnNotNull(...).addRuleMetric(metric name).fixValueEqual(50)... This can encapsulate the received function instruction into a rule request body, obtain the inspection request containing the rule request body, and complete the addition and execution of the inspection rule in the same transaction.

[0058] This application embodiment provides a verification method. After receiving a detection request sent by a client and transmitted by a first server, before constructing the entity class detection rule corresponding to the functional instruction through the builder pattern, the method further includes: decompressing the signature in the detection request to obtain a second concatenation field, wherein the second concatenation field includes an access token and a compression field.

[0059] The compressed field is decompressed to obtain a first concatenated field, wherein the first concatenated field includes an application identifier, a random number, and a timestamp;

[0060] Verify the application identifier, random number, timestamp, and access token;

[0061] After successful verification, the step of constructing the entity class detection rules corresponding to the functional instruction using the builder pattern is executed. That is, after receiving the functional instruction (containing the functional instruction and its corresponding function parameter values), the first server generates a compressed signature based on the application identifier, random number, timestamp, and access token. This functional instruction is then encapsulated into a detection rule request body. Based on this request body and the signature, a detection request is obtained (e.g., the request format can be HTTP for inter-service transmission; this example is just one example and can be customized as needed). The functional instruction is sent to the data quality detection service as is via the detection request, along with parameters such as the user and service address. Simultaneously, the first server negotiates the signature with the data quality detection service. This means the data quality detection service decompresses the compressed signature in the received detection request to obtain a second concatenation field, namely the access token and the compressed field. This compressed field is further decompressed to obtain the application identifier, random number, and timestamp. Finally, verification is performed based on the application identifier, random number, timestamp, and access token. If the verification passes, the entity class detection rules are constructed. In one example, the signature method can uniformly employ the SHA-256 algorithm, calculating the signature using the random value (nonce), timestamp, system-configured appId, and appToken. The specific generation process includes: obtaining the system-configured application identifier, random number, and timestamp; concatenating these three elements to obtain the first concatenated field; then compressing this first concatenated field using a hash function to obtain the compressed field; finally, concatenating this compressed field with the access token to obtain the second concatenated field; and finally compressing this second concatenated field again using a hash function to generate the signature. For better understanding, the following method can be used:

[0062] For example, signature=Sha256Encoder.encode(Sha256Encoder.encode(appId+nonce+timestamp)+appToken).

[0063] In this embodiment of the invention, the unique application identifier is combined with random numbers and timestamp information for concatenation, ensuring the uniqueness and timeliness of subsequent signature information generation. Furthermore, after the initial concatenation, the access token is combined again, effectively employing a double encryption method to reduce the probability of signature forgery and improve the security of the signature information. It should be understood that in other embodiments, other hash functions may be used, and this is not limited here.

[0064] This application provides a method for generating a detection request, wherein the detection request is generated based on a functional command input by a user on a client, including:

[0065] Based on the functional instructions input by the user on the client, the client retrieves the function parameter values ​​of each functional instruction from the configuration file, thereby obtaining the detection rule function and sending it to the first server; the first server encapsulates the detection rule function into the detection request. In other words, the configuration file contains the function parameter values ​​corresponding to each functional instruction, and a function instruction-detection rule function is generated based on the functional instruction and its function parameter values ​​and sent to the first server. Thus, compared to the cumbersome clicking and selection of functional instructions and configurations on the human-computer interaction interface in existing technologies, this application provides functional instructions through a function list, directly obtaining the detection rule function from the configuration file by retrieving the function parameter values ​​of the functional instructions. This eliminates the need for importing dependencies, declaring variables, or writing logic like in any programming language; it only requires calling the function. By adding the specification of function parameter values ​​in the configuration file, the input efficiency of the instructions is significantly improved. After the functional instruction is input, the client obtains the file path of the configuration file, reads the configuration file using a file stream, calls the load method to parse the file stream of the configuration file to obtain key-value pairs, where the key in the key-value pair corresponds to the functional instruction and the value corresponds to the function parameter value. Then, the function parameter value of the corresponding functional instruction in the configuration file is extracted, and the function parameter value is provided to set and replace to obtain the detection rule function - function instruction.

[0066] This application provides a method for updating function parameter values ​​in a configuration file, and further includes:

[0067] Based on the configuration change commands input by the user on the client, the values ​​of each function parameter in the configuration file are updated. In other words, the corresponding function parameter values ​​in the configuration file can be changed through configuration change commands. Thus, the function parameter values ​​corresponding to the same functional command can be different, resulting in multiple different detection rule functions. In one example, simply changing the data source parameter value in the configuration file corresponding to a functional command can yield detection rule functions for different data sources—the data to be detected.

[0068] This application provides a method for running entity class detection rules. After constructing the entity class detection rules corresponding to the functional instructions using the builder pattern, the method further includes:

[0069] The second server whose idle resources meet the entity class detection rules is determined from among the servers.

[0070] The entity class detection rules are sent to the second server; the second server is used to detect the data to be detected according to the entity class detection rules.

[0071] The detection results from the second server are obtained and displayed by the client. In other words, the intermediate service of the first server (e.g., Linkis service) can, after receiving the detection rules, determine the idle resource status of each server and send the detection rules to the second server with the most idle resources. The second server with the most idle resources then runs the detection rules and obtains the detection results. This improves detection efficiency and resource utilization.

[0072] This application provides a method for extending detection rule templates, and further includes:

[0073] The interface is configured to receive template extension instructions; the interface contains the abstract classes required to generate the detection rule template.

[0074] A new detection rule template is generated based on the template extension instructions, and a new functional instruction corresponding to the new detection rule template is set in the enumeration class. The function parameter values ​​of the new functional instruction are added to the configuration file. In other words, compared to the prior art where adding a new detection rule requires redeveloping the detection rule within the data quality detection service, this application can receive template extension instructions through a setting interface, generate a new detection rule template based on the template extension instructions and the abstract classes required for generating the detection rule template in the setting interface, and correspondingly set a new functional instruction corresponding to the new detection rule template in the enumeration class, adding the function parameter values ​​of the new functional instruction to the configuration file. This achieves simple and quick addition of detection rules, improves the flexibility of setting detection rules, and reduces the cycle and cost of adding detection rules.

[0075] Based on the above method flow, embodiments of this application provide a data quality detection method flow, such as... Figure 3 As shown, it includes:

[0076] Step 301: Obtain the function instructions input by the user.

[0077] Step 302: Obtain the function parameter value corresponding to the function instruction in the configuration file based on the function instruction.

[0078] Step 303: Generate the detection rule function based on the function parameter values ​​and the function instructions.

[0079] Step 304: Send the detection rule function to the first server.

[0080] Step 305: The first server generates a detection request based on the detection rule function and the signature generated by the application identifier, random number, timestamp and access token.

[0081] Step 306: The first server sends the detection request to the data quality detection service.

[0082] Step 307: The data quality inspection service receives the inspection request and, after verifying the signature in the inspection request, determines the inspection rule template corresponding to the functional instruction of the inspection rule function in the inspection request.

[0083] Step 308: Construct detection rules based on the detection rule function and the detection rule template.

[0084] Step 309: The data quality inspection service sends the inspection rule to the intermediate service.

[0085] Step 310: The intermediate service determines the server with the most idle resources among all servers - the second server.

[0086] Step 311: The intermediate service sends the detection rule to the second server.

[0087] Step 312: The second server runs the detection rule, performs detection on the data to be detected, obtains the detection results, and associates the detection results with the data to be detected and stores them in the database where the data to be detected is located.

[0088] Step 313: The second server sends the detection result to the intermediate service of the first server.

[0089] Step 314: After the intermediate service determines the detection task of the detection rule, it sends the detection result to the data quality detection service.

[0090] Step 315: The data quality inspection service receives the inspection results, verifies the results, and determines whether an alarm or notification needs to be generated and sent to the client for display to the user.

[0091] For example, if the detection result shows that the number of null values ​​in the data to be detected reaches 10, exceeding the null value abnormal data threshold of 5, an alarm or notification will be generated and sent to the client to be displayed to the user.

[0092] Step 316: The client receives the configuration change instruction and updates the corresponding function parameter values ​​in the configuration file according to the configuration change instruction.

[0093] Step 317: Receive template extension instructions through the interface, generate new detection rule templates according to the template extension instructions, set new functional instructions corresponding to the new detection rule templates in the enumeration class, and add the function parameter values ​​of the new functional instructions in the configuration file.

[0094] It should be noted that the above method flow is not unique. For example, steps 316 / 317 may or may not be executed. Steps 316 / 317 may also be executed before step 301. The above method flow is only for clearly illustrating one example of the solution and does not limit the specific implementation of the solution.

[0095] Based on the same concept, embodiments of the present invention provide a data quality detection device. Figure 4 This is a schematic diagram of a data quality detection device provided in an embodiment of this application, as shown below. Figure 4 The following are examples:

[0096] The transceiver module 401 is used to receive a detection request sent by the client and transmitted by the first server; the detection request is generated based on a functional instruction input by the user on the client.

[0097] The processing module 402 is used to construct entity class detection rules corresponding to the functional instructions based on the detection rule template corresponding to the functional instructions using the builder pattern, so as to perform data quality detection on the data to be detected according to the entity class detection rules.

[0098] Optionally, the processing module 402 is specifically used to: determine the detection rule template corresponding to the functional instruction through the enumeration class; construct the entity class detection rule corresponding to the functional instruction by combining the function parameter values ​​in the functional instruction with the abstract classes in the detection rule template corresponding to the functional instruction through the builder pattern; the abstract classes include basic library table attributes, data indicator attributes, abnormal data threshold attributes, and return attributes.

[0099] Optionally, the processing module 402 is further configured to decompress the signature in the detection request to obtain a second concatenation field, wherein the second concatenation field includes an access token and a compression field;

[0100] The compressed field is decompressed to obtain a first concatenated field, wherein the first concatenated field includes an application identifier, a random number, and a timestamp;

[0101] The application identifier, random number, timestamp, and access token are verified; and after the verification is successful, the step of constructing the entity class detection rule corresponding to the functional instruction through the builder pattern is executed.

[0102] Optionally, the processing module 402 is specifically used to: based on the functional instructions input by the user on the client, the client obtains the function parameter values ​​of each function instruction from the configuration file, thereby obtaining a detection rule function and sending it to the first server; the first server encapsulates the detection rule function into the detection request.

[0103] Optionally, the processing module 402 is further configured to update the values ​​of each function parameter in the configuration file based on the configuration change command input by the user on the client.

[0104] Optionally, the processing module 402 is further configured to: determine from each server that the idle resources of the second server conform to the entity class detection rules; send the entity class detection rules to the second server; the second server is configured to detect the data to be detected according to the entity class detection rules; obtain the detection results of the second server; and display the detection results through the client.

[0105] Optionally, the processing module 402 is further configured to receive template extension instructions through a setting interface; the setting interface has various abstract classes required to generate detection rule templates; generate new detection rule templates according to the template extension instructions, set new functional instructions corresponding to the new detection rule templates in the enumeration class, and add the function parameter values ​​of the new functional instructions in the configuration file.

[0106] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data quality detection method, characterized by, The method includes: Receive a detection request sent by the client and transmitted via the first server; the detection request is generated based on a functional command input by the user on the client. The detection rule template corresponding to the functional instruction is determined by an enumeration class; the correspondence between each functional instruction and each detection rule template is preset by the enumeration class. The function parameter values ​​in the functional instruction and the abstract classes in the detection rule template corresponding to the functional instruction are used to construct the entity class detection rules corresponding to the functional instruction through the builder pattern, so as to perform data quality detection on the data to be detected according to the entity class detection rules; each abstract class includes basic library table attributes, data indicator attributes, abnormal data threshold attributes, and return attributes; The method further includes: The interface is configured to receive template extension instructions; the interface contains the abstract classes required to generate the detection rule template. A new detection rule template is generated based on the template extension instruction, and a new functional instruction corresponding to the new detection rule template is set in the enumeration class. The function parameter value of the new functional instruction is added to the configuration file.

2. The method as claimed in claim 1, characterized in that, After receiving the detection request sent by the client and transmitted by the first server, before constructing the entity class detection rules corresponding to the functional instructions using the builder pattern, the process further includes: The signature in the detection request is decompressed to obtain a second concatenation field, wherein the second concatenation field includes an access token and a compression field; The compressed field is decompressed to obtain a first concatenated field, wherein the first concatenated field includes an application identifier, a random number, and a timestamp; Verify the application identifier, random number, timestamp, and access token; After successful verification, the step of constructing the entity class detection rules corresponding to the functional instructions using the builder pattern is executed.

3. The method as claimed in claim 1, wherein, The detection request is generated based on functional instructions entered by the user on the client, including: Based on the functional instructions input by the user on the client, the client obtains the function parameter values ​​of each functional instruction from the configuration file, thereby obtaining the detection rule function and sending it to the first server; The first server encapsulates the detection rule function into the detection request.

4. The method as claimed in claim 3, characterized in that, Also includes: Based on the configuration change command entered by the user on the client, the values ​​of each function parameter in the configuration file are updated.

5. The method of any one of claims 1-4, wherein, After constructing the entity class detection rules corresponding to the functional instructions using the builder pattern, the following is also included: The second server whose idle resources meet the entity class detection rules is determined from among the servers. The entity class detection rules are sent to the second server; the second server is used to detect the data to be detected according to the entity class detection rules. The detection results from the second server are obtained and displayed by the client.

6. A data quality detection apparatus characterized by comprising: The device includes: The transceiver module is used to receive detection requests sent by the client and transmitted by the first server; the detection requests are generated based on functional instructions input by the user on the client. The processing module is used to determine the detection rule template corresponding to the functional instruction through an enumeration class; the correspondence between each functional instruction and each detection rule template is preset through the enumeration class. The function parameter values ​​in the functional instruction and the abstract classes in the detection rule template corresponding to the functional instruction are used to construct the entity class detection rules corresponding to the functional instruction through the builder pattern, so as to perform data quality detection on the data to be detected according to the entity class detection rules; each abstract class includes basic library table attributes, data indicator attributes, abnormal data threshold attributes, and return attributes; The processing module is also used for: The interface is configured to receive template extension instructions; the interface contains the abstract classes required to generate the detection rule template. A new detection rule template is generated based on the template extension instruction, and a new functional instruction corresponding to the new detection rule template is set in the enumeration class. The function parameter value of the new functional instruction is added to the configuration file.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when run on a computer, causes the computer to perform the method of any one of claims 1 to 5.

8. A computer device, comprising: include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method as described in any one of claims 1 to 5 according to the obtained program.

Images