Data quality inspection method, device, electronic device and computer readable medium

Abstract

The present application discloses a data quality inspection method, device, electronic device and computer-readable medium, and relates to the field of computer technology. The method includes: obtaining incremental data, and then determining the type of each data item in the incremental data, and determining the corresponding verification method according to the type; determining the target quality inspection node pool according to the type, determining the priority of the idle quality inspection node in the target quality inspection node pool according to the verification method, determining the total amount of data that the idle quality inspection node can accommodate, and in response to the data amount corresponding to the incremental data being greater than the total data amount, defining new idle quality inspection nodes based on the data amount and the total data amount, and then determining the number of all idle quality inspection nodes, splitting the incremental data based on the number to obtain split data; distributing the split data to the idle quality inspection nodes according to the priority to perform quality inspection and output the quality inspection results. Flexible data verification can be achieved while ensuring low cost, and the versatility of data detection is high.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data quality inspection method, device, electronic device and computer-readable medium. Background Art

[0002] At present, the license databases in various provinces and cities across the country have a great demand for quality inspection, and have high requirements for the concurrency and response time of the interface. In the process of processing big data, it is necessary to detect the quality of big data. The test is carried out by using manual sampling test methods. Due to the high cost of manual inspection, uneven sample distribution, small sample size, long verification cycle, and the test results are affected by the personal quality of the inspectors: customized inspection programs can usually only solve the data quality of a specific scenario, and the versatility is not high.

[0003] In the process of implementing this application, the inventors found that there are at least the following problems in the prior art:

[0004] Existing data quality detection methods have low versatility and high data quality detection costs. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a data quality inspection method, device, electronic device and computer-readable medium, which can solve the problems of low versatility and high cost of data quality inspection in existing data quality inspection methods.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present application, a data quality inspection method is provided, comprising:

[0007] Obtain incremental data, determine the type of each data item in the incremental data, and determine the corresponding verification method according to the type;

[0008] Determine a target quality inspection node pool according to the type, determine the priority of idle quality inspection nodes in the target quality inspection node pool according to the verification method, determine the total amount of data that can be accommodated by the idle quality inspection nodes, and in response to the data amount corresponding to the incremental data being greater than the total data amount, define new idle quality inspection nodes based on the data amount and the total data amount, and then determine the number of all idle quality inspection nodes, and split the incremental data based on the number to obtain split data;

[0009] Based on the priority, the split data is distributed to the idle quality inspection nodes, so that the idle quality inspection nodes perform quality inspection on the split data based on the verification method corresponding to the split data and output the quality inspection results.

[0010] Optionally, obtain incremental data, including:

[0011] Receive a data quality inspection request, obtain a corresponding incremental identifier, and obtain corresponding incremental data based on the incremental identifier.

[0012] Optionally, obtaining corresponding incremental data based on the incremental identifier includes:

[0013] Obtain a data source identifier in the data quality inspection request, and obtain corresponding source data from a corresponding source database based on the data source identifier;

[0014] According to the incremental identifier, the corresponding incremental data is obtained from the source data.

[0015] Optionally, obtaining incremental data further includes:

[0016] The data acquisition frequency identifier carried in the data quality inspection request is obtained, and then the corresponding incremental data is obtained according to the incremental identifier at the frequency corresponding to the data acquisition frequency identifier.

[0017] Optionally, obtain incremental data, including:

[0018] In response to the current time reaching the execution time of the data quality inspection task, obtaining the data quality inspection configuration information corresponding to the data quality inspection task, and then obtaining the incremental identifier in the data quality inspection configuration information;

[0019] Get the corresponding incremental data according to the incremental identifier.

[0020] Optionally, the incremental data is split according to the quantity to obtain split data, including:

[0021] Determine the amount of data that each idle quality inspection node can currently accommodate;

[0022] Based on the quantity and the amount of each data, the incremental data is split to obtain split data.

[0023] Optionally, the split data is distributed to idle quality inspection nodes based on priority, including:

[0024] Determine the amount of each split data corresponding to each split data, and then pair the split data with the idle quality inspection nodes according to the amount of each split data and each data amount;

[0025] The split data is distributed to the paired idle QC nodes based on the order of the idle QC nodes corresponding to the priorities.

[0026] In addition, the present application also provides a data quality inspection device, including:

[0027] an acquisition unit, configured to acquire incremental data, and then determine the type of each data item in the incremental data, and determine a corresponding verification method according to the type;

[0028] The data splitting unit is configured to determine a target quality inspection node pool according to the type, determine the priority of an idle quality inspection node in the target quality inspection node pool according to the verification method, determine the total amount of data that can be accommodated by the idle quality inspection node, and in response to the data amount corresponding to the incremental data being greater than the total data amount, define a new idle quality inspection node based on the data amount and the total data amount, further determine the number of all the idle quality inspection nodes, and split the incremental data based on the number to obtain split data;

[0029] The data quality inspection unit is configured to distribute the split data to the idle quality inspection nodes based on the priority, so that the idle quality inspection nodes perform quality inspection on the split data based on the verification method corresponding to the split data and output the quality inspection result.

[0030] Optionally, the acquisition unit is further configured to:

[0031] Receive a data quality inspection request, obtain a corresponding incremental identifier, and obtain corresponding incremental data based on the incremental identifier.

[0032] Optionally, the acquisition unit is further configured to:

[0033] Obtain a data source identifier in the data quality inspection request, and obtain corresponding source data from a corresponding source database based on the data source identifier;

[0034] According to the incremental identifier, the corresponding incremental data is obtained from the source data.

[0035] Optionally, the acquisition unit is further configured to:

[0036] The data acquisition frequency identifier carried in the data quality inspection request is obtained, and then the corresponding incremental data is obtained according to the incremental identifier at the frequency corresponding to the data acquisition frequency identifier.

[0037] Optionally, the acquisition unit is further configured to:

[0038] In response to the current time reaching the execution time of the data quality inspection task, obtaining the data quality inspection configuration information corresponding to the data quality inspection task, and then obtaining the incremental identifier in the data quality inspection configuration information;

[0039] Get the corresponding incremental data according to the incremental identifier.

[0040] Optionally, the data splitting unit is further configured to:

[0041] Determine the amount of data that each idle quality inspection node can currently accommodate;

[0042] Based on the quantity and the amount of each data, the incremental data is split to obtain split data.

[0043] Optionally, the data quality inspection unit is further configured to:

[0044] Determine the amount of each split data corresponding to each split data, and then pair the split data with the idle quality inspection nodes according to the amount of each split data and each data amount;

[0045] The split data is distributed to the paired idle QC nodes based on the order of the idle QC nodes corresponding to the priorities.

[0046] In addition, the present application also provides a data quality inspection electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the data quality inspection method as described above.

[0047] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data quality inspection method as described above is implemented.

[0048] To achieve the above objective, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0049] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, the data quality inspection method provided by the embodiment of the present application is implemented.

[0050] One embodiment of the above invention has the following advantages or beneficial effects: the present application obtains incremental data, and then determines the type of each data item in the incremental data, and determines the corresponding verification method according to the type; determines the target quality inspection node pool according to the type, determines the priority of the idle quality inspection node in the target quality inspection node pool according to the verification method, determines the total amount of data that the idle quality inspection node can accommodate, and in response to the data amount corresponding to the incremental data being greater than the total data amount, defines a new idle quality inspection node based on the data amount and the total data amount, and then determines the number of all idle quality inspection nodes, and splits the incremental data based on the number to obtain split data; based on the priority, distributes the split data to the idle quality inspection node, so that the idle quality inspection node performs quality inspection on the split data based on the verification method corresponding to the split data and outputs the quality inspection result. The verification method can be configured according to the data item type, and flexible data verification can be achieved while ensuring low cost, and the horizontal expansion of the idle quality inspection node can be achieved. The data detection is highly versatile, and the orderly quality inspection of the corresponding data items can be achieved according to the urgency of the data quality inspection, thereby improving the speed and quality of data quality detection.

[0051] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.

[0053] Figure 1 is a schematic diagram of the main process of the data quality inspection method according to the first embodiment of the present application;

[0054] Figure 2 is a schematic diagram of the main process of the data quality inspection method according to the second embodiment of the present application;

[0055] Figure 3 is a schematic diagram of an application scenario of a data quality inspection method according to the third embodiment of the present application;

[0056] Figure 4 is a schematic diagram of the main units of the data quality inspection device according to an embodiment of the present application;

[0057] Figure 5 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0058] Figure 6 It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. The acquisition, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations.

[0060] Figure 1 is a schematic diagram of the main process of the data quality inspection method according to the first embodiment of the present application, such as Figure 1 As shown in the figure, the data quality inspection methods include:

[0061] Step S101, obtaining incremental data, and then determining the type of each data item in the incremental data, and determining the corresponding verification method according to the type.

[0062] In this embodiment, the execution subject of the data quality inspection method (for example, a server) can obtain incremental data through a wired connection or a wireless connection. Incremental data can be, for example, newly uploaded data when handling business A, and can be newly uploaded pictures, audio, video, etc. The embodiment of this application does not specifically limit the type and content of incremental data.

[0063] Specifically, obtaining incremental data includes: in response to the current time reaching the execution time of the data quality inspection task, obtaining data quality inspection configuration information corresponding to the data quality inspection task, and then obtaining the incremental identifier in the data quality inspection configuration information; obtaining the corresponding incremental data according to the incremental identifier.

[0064] Incremental data can be acquired based on scheduled data quality inspection tasks. When the current time reaches the task execution time corresponding to the data quality inspection task in the task pool, the execution subject can execute the data quality inspection task that reaches the task execution time, and then obtain the corresponding incremental data according to the incremental identifier in the data quality inspection configuration information corresponding to the data quality inspection task. Specifically, the data quality inspection configuration information may include an incremental identifier and data quality inspection rules, where the incremental identifier is used to characterize which data to be processed. The data quality inspection rules, such as the size of the picture, the clarity of the picture, the size of the file and other information, the embodiment of the present application does not limit the specific content contained in the data quality inspection configuration information.

[0065] The types of data items include: data items for the certificate holder code, data items for the issuing authority information, data items for the certificate type information, and data items for the issuance date and validity period information. Verification methods include: configuring custom methods for quality verification, configuring corresponding third-party verification interfaces for interface-level real-time verification, configuring local or third-party database directory tables for library-table-level quality verification, and configuring regular expressions for quality verification.

[0066] Specifically, the corresponding verification method is determined according to the type of each data item, and the specific implementation is as follows:

[0067] The execution subject first configures the quality inspection rules of the data items through the quality inspection module. Custom rule configuration can be performed according to the characteristics of the data items. The rule configuration supports multiple types such as jar packages, interfaces, library tables, regular expressions, lengths, etc. For the certificate holder's code data items, the verification jar package and verification method path can be uploaded to configure a custom method for quality verification; for the issuing agency information data items, the corresponding third-party verification interface can be configured for interface-level real-time verification; for the license type information data items, the local or third-party database directory table can be configured for library-level quality verification; for the issuance date and validity period information items, regular expressions can be configured for quality verification

[0068] Step S102, determine the target quality inspection node pool according to the type, determine the priority of the idle quality inspection nodes in the target quality inspection node pool according to the verification method, determine the total amount of data that the idle quality inspection nodes can accommodate, and in response to the data amount corresponding to the incremental data being greater than the total data amount, define new idle quality inspection nodes based on the data amount and the total data amount, and then determine the number of all idle quality inspection nodes, and split the incremental data based on the number to obtain split data.

[0069] In the embodiment of the present application, when the execution subject determines that the amount of data corresponding to the incremental data is less than or equal to the total amount of data that can be accommodated by the idle quality inspection nodes already in the target quality inspection node pool, the number of idle quality inspection nodes at this time is the number of idle quality inspection nodes already in the current target quality inspection node pool. When the execution subject determines that the amount of data corresponding to the incremental data is greater than the total amount of data that can be accommodated by the idle quality inspection nodes already in the target quality inspection node pool, the execution subject can horizontally expand the quality inspection module, specifically, it can be determined according to the difference S between the amount of data corresponding to the incremental data and the total amount of data that can be accommodated by the idle quality inspection nodes already in the target quality inspection node pool to determine how many idle quality inspection nodes to expand, and update the number of idle quality inspection nodes. Specifically, the execution subject can divide the difference S by the amount of data F that can be accommodated by each idle quality inspection node to obtain the number D of idle quality inspection nodes that need to be expanded, and then add it to the number N of idle quality inspection nodes already in the current target quality inspection node pool to obtain the number D+N of idle quality inspection nodes that need to be used.

[0070] The type of each data item in the incremental data can correspond to a quality inspection node pool. The execution subject can determine the quality inspection node pool corresponding to each data item according to the type of each data item in the incremental data. For example, the incremental data includes data item a, data item b, and data item c, and the corresponding data item types are a, b, and c, respectively, and the corresponding quality inspection node pools are A, B, and C, respectively. Then the quality inspection node pools are A, B, and C, which are the target quality inspection node pools corresponding to the types a, b, and c of each data item in the incremental data. In the embodiment of the present application, each verification method corresponds to the priority of a quality inspection node. For example, when the verification method is t, the priority of the corresponding quality inspection node is 9, when the verification method is y, the priority of the corresponding quality inspection node is 8, and when the verification method is z, the priority of the corresponding quality inspection node is 7. The larger the number corresponding to the priority in the embodiment of the present application, the higher the priority, and the embodiment of the present application does not specifically limit this. The higher the priority of the quality inspection node (for example, an idle quality inspection node M), the earlier the corresponding split data m is distributed to the idle quality inspection node M, that is, the split data m is preferentially inspected by the quality inspection node M. The execution entity can determine the number of idle quality inspection nodes required for the incremental data, such as D+N. Then, the execution entity can split the incremental data based on the number of idle quality inspection nodes D+N. The split can be an average split, or it can be split according to a certain splitting rule based on the number of idle quality inspection nodes to obtain each split data. The number N of idle quality inspection nodes can be the sum of the number of idle quality inspection nodes in the target quality inspection node pool (for example, A, B, C). When the amount of data corresponding to the incremental data is less than or equal to the total amount of data that can be accommodated by the existing idle quality inspection nodes in the target quality inspection node pool, D is equal to 0.

[0071] The idle quality inspection node corresponds to the quality inspection module. Each quality inspection module can be a quality inspection node, which is responsible for the quality inspection rule configuration of the source data and the multi-threaded data quality inspection work. It can be horizontally expanded according to the amount of existing data, and multiple quality inspection modules constitute a quality inspection node pool. The execution subject can determine the quality inspection node that currently has no data quality inspection task according to the working status of each quality inspection node in the target quality inspection node pool, and then determine the quality inspection node that currently has no data quality inspection task as an idle quality inspection node. Then the execution subject can determine the number of quality inspection nodes that are currently idle. The data quality inspection capabilities of each idle quality inspection node can be different. The execution subject can split the incremental data to be inspected according to the number of quality inspection nodes in the target quality inspection node pool that are idle to obtain split data. The execution subject can also split the incremental data according to the total number of idle quality inspection nodes D+N after horizontal expansion to obtain split data.

[0072] Specifically, the incremental data is split according to the quantity to obtain split data, including: determining the amount of data that each idle quality inspection node can currently accommodate; and splitting the incremental data based on the quantity and each data amount to obtain split data.

[0073] The execution subject has different amounts of data to be quality inspected according to the idle quality inspection nodes (the idle quality inspection nodes here may include newly defined idle quality inspection nodes obtained after horizontal expansion). The execution subject can split the incremental data according to the amount of data to be quality inspected that each idle quality inspection node can bear and the number of idle quality inspection nodes. For example, the amount of data to be quality inspected that the idle quality inspection node 1 can bear is 5, the amount of data to be quality inspected that the idle quality inspection node 2 can bear is 8, and the amount of data to be quality inspected that the idle quality inspection node 3 can bear is 15. If there are 25 incremental data, the process of splitting the incremental data can be as follows: the execution subject can divide 25 (the amount of incremental data) by 3 (the number of idle quality inspection nodes) to obtain a quotient of 8 with a remainder of 1. The execution subject can first allocate 8 incremental data to the idle quality inspection node 2 for data quality inspection (because the amount of data to be quality inspected that the idle quality inspection node 2 can bear is exactly 8), and then the execution subject can allocate the other 8 incremental data to the idle quality inspection node 3 for data quality inspection (because The number of data to be inspected that the idle quality inspection node 3 can bear exceeds 8), that is, firstly, the idle quality inspection nodes that can bear the number of data to be inspected that exceeds the quotient value (that is, 8 in this embodiment) are allocated incremental data equal to the quotient value, and then the idle quality inspection nodes that can bear the number of data to be inspected that is less than or equal to the quotient value are allocated incremental data equal to the number of data to be inspected that they can bear (that is, 5 incremental data are allocated to the idle quality inspection node 1 for data quality inspection), and the remaining incremental data (3 plus the remainder 1, a total of 4) are allocated again to the idle quality inspection nodes that can bear the number of data to be inspected that exceeds the quotient value (that is, the idle quality inspection node 3), so as to realize the use of higher performance idle quality inspection nodes. Based on the above splitting process, the incremental data is finally split into 5, 8, and 12.

[0074] It is understandable that when each idle quality inspection node can bear the same amount of data to be inspected, the execution entity can divide the amount of incremental data by the number of idle quality inspection nodes, that is, split the incremental data into n equal parts m, where m is the ratio of the amount of incremental data to the number of idle nodes n, to achieve average distribution. The embodiments of the present application do not limit the specific distribution method.

[0075] Step S103: Distribute the split data to idle quality inspection nodes based on the priority, so that the idle quality inspection nodes perform quality inspection on the split data based on the verification method corresponding to the split data and output the quality inspection result.

[0076] The priority of the idle quality inspection node determines the order in which the idle quality inspection node receives the split data.

[0077] For example, the priority of idle quality inspection node 1 is 9, the priority of idle quality inspection node 2 is 8, and the priority of idle quality inspection node 3 is 7. In the embodiment of the present application, the larger the number corresponding to the priority, the higher the priority. The amount of data to be inspected that can be undertaken by idle quality inspection node 1 is 5, the amount of data to be inspected that can be undertaken by idle quality inspection node 2 is 8, and the amount of data to be inspected that can be undertaken by idle quality inspection node 3 is 15. If there are 25 incremental data, then under this premise:

[0078] Specifically, the split data is distributed to idle quality inspection nodes, including:

[0079] Determine the amount of each split data corresponding to each split data (for example, 5, 8, 12), and then pair the split data (5, 8, 12) with idle quality inspection nodes (for example, idle quality inspection node 1, idle quality inspection node 2, idle quality inspection node 3) according to the amount of each split data and the amount of each data (for example, 5, 8, 10), and the examples may be 5-idle quality inspection node 1, 8-idle quality inspection node 2, and 12-idle quality inspection node 3; distribute the split data to the paired idle quality inspection nodes, that is, in order of priority, distribute 5 incremental data to idle quality inspection node 1, distribute 8 incremental data to idle quality inspection node 2, and distribute 12 incremental data to idle quality inspection node 3. In this way, orderly quality inspection of corresponding data items can be achieved according to the urgency of data quality inspection, and the speed and quality of data quality inspection can be improved.

[0080] This embodiment obtains incremental data, and then determines the type of each data item in the incremental data, and determines the corresponding verification method according to the type; determines the total amount of data that can be accommodated by the idle quality inspection nodes in the target quality inspection node pool, and in response to the data amount corresponding to the incremental data being greater than the total data amount, defines new idle quality inspection nodes based on the data amount and the total data amount, and then determines the number of all idle quality inspection nodes, and splits the incremental data based on the number to obtain split data; distributes the split data to the idle quality inspection nodes, so that the idle quality inspection nodes perform quality inspection on the split data based on the verification method corresponding to the split data and output the quality inspection results. It can be realized to configure the verification method according to the data item type, realize flexible data verification while ensuring low cost, realize horizontal expansion of idle quality inspection nodes, and have high versatility for data detection. It can realize orderly quality inspection of corresponding data items according to the urgency of data quality inspection, and improve the speed and quality of data quality detection.

[0081] Figure 2 is a schematic diagram of the main flow of the data quality inspection method according to the second embodiment of the present application. Figure 2 As shown in the figure, the data quality inspection methods include:

[0082] Step S201, receiving a data quality inspection request, obtaining a corresponding incremental identifier, and obtaining corresponding incremental data based on the incremental identifier.

[0083] The execution subject may receive a data quality inspection request sent by a user and respond to the data quality inspection request to execute the data quality inspection process, specifically obtaining an incremental identifier carried in the data quality inspection request and obtaining corresponding incremental data based on the incremental identifier.

[0084] Specifically, obtaining corresponding incremental data based on the incremental identifier includes: obtaining a data source identifier in a data quality inspection request, and obtaining corresponding source data from a corresponding source database based on the data source identifier; and obtaining corresponding incremental data from the source data according to the incremental identifier.

[0085] The data source identifier is used to indicate which source database the incremental data is obtained from. The execution subject can obtain the corresponding source data from the corresponding source database according to the data source identifier carried in the data quality inspection request, and then obtain the corresponding incremental data from the obtained source data according to the incremental identifier.

[0086] Specifically, acquiring the incremental data further includes: acquiring a data acquisition frequency identifier carried in the data quality inspection request, and then acquiring corresponding incremental data according to the incremental identifier at a frequency corresponding to the data acquisition frequency identifier.

[0087] The data acquisition frequency identifier is used to characterize the data acquisition frequency. The execution subject can obtain incremental data according to the data acquisition frequency corresponding to the data acquisition frequency identifier, thereby ensuring the orderly processing of incremental data and improving the quality of quality inspection of incremental data.

[0088] Step S202, determine the type of each data item in the incremental data, and determine the corresponding verification method according to the type.

[0089] Step S203, determine the target quality inspection node pool according to the type, determine the priority of the idle quality inspection nodes in the target quality inspection node pool according to the verification method, determine the total amount of data that the idle quality inspection nodes can accommodate, and in response to the data amount corresponding to the incremental data being greater than the total data amount, define new idle quality inspection nodes based on the data amount and the total data amount, and then determine the number of all idle quality inspection nodes, and split the incremental data based on the number to obtain split data.

[0090] The total amount of data that can be accommodated by the idle quality inspection nodes is the total amount of data that can be accommodated by the existing idle quality inspection nodes in the target quality inspection node pool.

[0091] The principle A for allocating idle quality inspection nodes to split data is: the amount of split data is less than or equal to the amount of to-be-inspected data that the allocated idle quality inspection nodes can bear.

[0092] Step S204: Distribute the split data to idle quality inspection nodes based on the priority, so that the idle quality inspection nodes perform quality inspection on the split data based on the verification method corresponding to the split data and output the quality inspection result.

[0093] Based on principle A (i.e., the amount of split data is less than or equal to the amount of data to be inspected that the allocated idle quality inspection node can bear) and the priority of the idle quality inspection node, a corresponding idle quality inspection node is assigned to each split data to optimize the performance of the idle quality inspection node.

[0094] The principles of steps S202 to S204 are similar to those of steps S101 to S103 and will not be described in detail here.

[0095] The embodiments of the present application can be horizontally expanded according to the amount of source data, and the quality inspection of data items can be customized according to business rules.

[0096] Figure 3 Schematic diagram of an application scenario of the data quality inspection method according to the third embodiment of the present application. The data quality inspection method of the embodiment of the present application is applied to the scenario of quality inspection of incremental data. Figure 3 As shown, to implement quality inspection of incremental data, the functions provided by the extraction module, distribution module, quality inspection module, and ES module are required. Among them, the extraction module: connects to the business party data source library and extracts source data for task distribution. The distribution module: distributes source data, and can distribute data according to the configuration of the quality inspection cluster to ensure the continuity of quality inspection data. Quality inspection module: configures quality inspection rules for source data and multi-threaded data quality inspection work, which can be horizontally expanded according to the amount of stock data. ES module: stores and queries data after quality inspection.

[0097] The extraction module configures the connection address of the source database, database type, user name, password, data extraction frequency, single extraction data volume, and data extraction rules. The extraction module will extract data from the source database (for example Figure 3 Extract the current incremental certificate type information, issuing authority information, holder information, certificate number, unique identifier, issuance date, validity period information, and face-to-face information from the source database 1, source database 2, and source database 3 in the source database. Push the extracted source data to the distribution module. Configure the node information of the quality inspection module through the distribution module. After receiving the data, the distribution module first performs node detection to obtain the number of idle nodes, and then splits the data according to the number of idle nodes, distributes the split data, and pushes the data to the currently idle quality inspection module (for example, Figure 3The quality inspection module 1, quality inspection module 2, quality inspection module 3, quality inspection module 4...) nodes in the data item are used for batch quality inspection. The quality inspection rules of the data items can be configured through the quality inspection module. Custom rule configuration can be performed according to the characteristics of the data items. The rule configuration supports multiple types such as jar packages, interfaces, library tables, regular expressions, and lengths. For the certificate holder code data item, the jar package for verification can be uploaded and the verification method path can be configured for quality verification; for the issuing agency information data item, the corresponding third-party verification interface can be configured for interface-level real-time verification; for the license type information data item, the local or third-party database directory table can be configured for library-table-level quality verification; for the issuance date and validity period information items, regular expressions can be configured for quality verification; finally, the node information of the ES module must be configured, and the data that has passed the quality inspection will be pushed to the ES module, and the abnormal data will fall into the local persistent database and be provided to the data source for source data rectification. After receiving the data that has passed the quality inspection, the ES module provides unified business query and statistical related function support to the outside world. It can achieve horizontal expansion according to the amount of source data, and the data item quality inspection can be customized according to business rules.

[0098] Figure 4 Schematic diagram of the main units of the data quality inspection device according to an embodiment of the present application. Figure 4 As shown, the data quality inspection device 400 includes an acquisition unit 401, a data splitting unit 402 and a data quality inspection unit 403.

[0099] The acquisition unit 401 is configured to acquire the incremental data, and then determine the type of each data item in the incremental data, and determine the corresponding verification method according to the type.

[0100] The data splitting unit 402 is configured to determine the target quality inspection node pool according to the type, determine the priority of the idle quality inspection nodes in the target quality inspection node pool according to the verification method, determine the total amount of data that the idle quality inspection nodes can accommodate, and in response to the data amount corresponding to the incremental data being greater than the total data amount, define new idle quality inspection nodes based on the data amount and the total data amount, and then determine the number of all idle quality inspection nodes, and split the incremental data based on the number to obtain split data.

[0101] The data quality inspection unit 403 is configured to distribute the split data to the idle quality inspection nodes based on the priority, so that the idle quality inspection nodes perform quality inspection on the split data based on the verification method corresponding to the split data and output the quality inspection result.

[0102] In some embodiments, the acquisition unit 401 is further configured to: receive a data quality inspection request, acquire a corresponding incremental identifier, and acquire corresponding incremental data based on the incremental identifier.

[0103] In some embodiments, the acquisition unit 401 is further configured to: acquire a data source identifier in the data quality inspection request, and acquire corresponding source data from the corresponding source database based on the data source identifier; and acquire corresponding incremental data from the source data according to the incremental identifier.

[0104] In some embodiments, the acquisition unit 401 is further configured to: acquire a data acquisition frequency identifier carried in the data quality inspection request, and then acquire corresponding incremental data according to the incremental identifier at a frequency corresponding to the data acquisition frequency identifier.

[0105] In some embodiments, the acquisition unit 401 is further configured to: in response to the current time reaching the execution time of the data quality inspection task, obtain the data quality inspection configuration information corresponding to the data quality inspection task, and then obtain the incremental identifier in the data quality inspection configuration information; obtain the corresponding incremental data according to the incremental identifier.

[0106] In some embodiments, the data splitting unit 402 is further configured to: determine the amount of data that each idle quality inspection node can currently accommodate; and split the incremental data based on the quantity and the amount of each data to obtain split data.

[0107] In some embodiments, the data quality inspection unit 403 is further configured to: determine the amounts of each split data corresponding to each split data, and then pair the split data with idle quality inspection nodes according to the amounts of each split data and each data amount; and distribute the split data to the paired idle quality inspection nodes based on the order of the idle quality inspection nodes corresponding to the priority.

[0108] It should be noted that the data quality inspection method and the data quality inspection device in the present application have a corresponding relationship in terms of specific implementation content, so the repeated content will not be explained again.

[0109] Figure 5 An exemplary system architecture 500 is shown to which the data quality inspection method or data quality inspection device according to the embodiment of the present application can be applied.

[0110] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0111] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0112] The terminal devices 501, 502, 503 may be various electronic devices having a data quality inspection processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0113] Server 505 can be a server that provides various services, such as a background management server that provides support for the incremental data obtained by the user using terminal devices 501, 502, and 503 (only as an example). The background management server can obtain incremental data, and then determine the type of each data item in the incremental data, and determine the corresponding verification method according to the type; determine the target quality inspection node pool according to the type, determine the priority of the idle quality inspection node in the target quality inspection node pool according to the verification method, determine the total amount of data that the idle quality inspection node can accommodate, and in response to the amount of data corresponding to the incremental data being greater than the total amount of data, define new idle quality inspection nodes based on the amount of data and the total amount of data, and then determine the number of all idle quality inspection nodes, and split the incremental data based on the number to obtain split data; based on the priority, distribute the split data to the idle quality inspection node, so that the idle quality inspection node performs quality inspection on the split data based on the verification method corresponding to the split data and outputs the quality inspection result. It can be realized that the verification method is configured according to the data item type, flexible data verification is achieved while ensuring low cost, and horizontal expansion of idle quality inspection nodes is achieved, and the versatility of data detection is high. It is possible to carry out orderly quality inspection of corresponding data items according to the urgency of data quality inspection, thereby improving the speed and quality of data quality inspection.

[0114] It should be noted that the data quality inspection method provided in the embodiment of the present application is generally executed by the server 505 , and accordingly, the data quality inspection device is generally set in the server 505 .

[0115] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0116] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0118] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0119] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.

[0120] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, a data splitting unit, and a data quality inspection unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0123] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device obtains incremental data, and then determines the type of each data item in the incremental data, and determines the corresponding verification method according to the type; determines the target quality inspection node pool according to the type, determines the priority of the idle quality inspection node in the target quality inspection node pool according to the verification method, determines the total amount of data that the idle quality inspection node can accommodate, and in response to the amount of data corresponding to the incremental data being greater than the total amount of data, defines a new idle quality inspection node based on the data amount and the total amount of data, and then determines the number of all idle quality inspection nodes, and splits the incremental data based on the number to obtain split data; based on the priority, distributes the split data to the idle quality inspection node, so that the idle quality inspection node performs quality inspection on the split data based on the verification method corresponding to the split data and outputs the quality inspection result.

[0124] The computer program product of the present application includes a computer program, and when the computer program is executed by a processor, the data quality inspection method in the embodiment of the present application is implemented.

[0125] According to the technical solution of the embodiment of the present application, it is possible to configure the verification method according to the data item type, realize flexible data verification while ensuring low cost, realize horizontal expansion of idle quality inspection nodes, and have high versatility for data detection. It is possible to carry out orderly quality inspection of corresponding data items according to the urgency of data quality inspection, and improve the speed and quality of data quality inspection.

[0126] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data quality inspection method, characterized in that: include: Acquiring incremental data includes: in response to the current time reaching the execution time of the data quality inspection task, acquiring data quality inspection configuration information corresponding to the data quality inspection task, and then acquiring an incremental identifier in the data quality inspection configuration information; acquiring corresponding incremental data according to the incremental identifier; Determine the type of each data item in the incremental data, and determine a corresponding verification method according to the type; Determine a target quality inspection node pool according to the type, determine the priority of an idle quality inspection node in the target quality inspection node pool according to the verification method, determine the total amount of data that can be accommodated by the idle quality inspection node, and in response to the data amount corresponding to the incremental data being greater than the total amount of data, define a new idle quality inspection node based on the data amount and the total amount of data, further determine the number of all idle quality inspection nodes, and split the incremental data based on the number to obtain split data; Based on the priority, distributing the split data to the idle quality inspection nodes includes: determining the split data amounts corresponding to the split data, and then pairing the split data with the idle quality inspection nodes according to the split data amounts and the data amounts; distributing the split data to the paired idle quality inspection nodes based on the order of the idle quality inspection nodes corresponding to the priority; So that the idle quality inspection node performs quality inspection on the split data based on the verification method corresponding to the split data and outputs the quality inspection result.

2. The method according to claim 1, characterized in that The obtaining of incremental data includes: A data quality inspection request is received, a corresponding incremental identifier is obtained, and corresponding incremental data is obtained based on the incremental identifier.

3. The method according to claim 2, characterized in that The acquiring corresponding incremental data based on the incremental identifier includes: Obtaining a data source identifier in the data quality inspection request, and obtaining corresponding source data from a corresponding source database based on the data source identifier; According to the incremental identifier, corresponding incremental data is obtained from the source data.

4. The method according to claim 2, characterized in that: The obtaining of incremental data further comprises: The data acquisition frequency identifier carried in the data quality inspection request is obtained, and then the corresponding incremental data is acquired according to the incremental identifier at the frequency corresponding to the data acquisition frequency identifier.

5. The method according to claim 1, characterized in that The step of splitting the incremental data according to the quantity to obtain split data includes: Determine the amount of data that each of the idle quality inspection nodes can currently accommodate; Based on the quantity and each data amount, the incremental data is split to obtain split data.

6. A data quality inspection device, characterized in that: include: The acquisition unit is configured to acquire incremental data, including: in response to the current time reaching the execution time of the data quality inspection task, acquiring data quality inspection configuration information corresponding to the data quality inspection task, and then acquiring the incremental identifier in the data quality inspection configuration information; acquiring the corresponding incremental data according to the incremental identifier; and then determining the type of each data item in the incremental data, and determining the corresponding verification method according to the type; a data splitting unit configured to determine a target quality inspection node pool according to the type, determine the priority of an idle quality inspection node in the target quality inspection node pool according to the verification method, determine a total amount of data that can be accommodated by the idle quality inspection node, and in response to the amount of data corresponding to the incremental data being greater than the total amount of data, define a new idle quality inspection node based on the amount of data and the total amount of data, further determine the number of all idle quality inspection nodes, and split the incremental data based on the number to obtain split data; The data quality inspection unit is configured to distribute the split data to the idle quality inspection nodes based on the priority, including: determining the split data amounts corresponding to each of the split data, and then pairing the split data with the idle quality inspection nodes according to the split data amounts and the data amounts; distributing the split data to the paired idle quality inspection nodes based on the order of the idle quality inspection nodes corresponding to the priority; so that the idle quality inspection nodes perform quality inspection on the split data based on the verification method corresponding to the split data and output the quality inspection results.

7. The device according to claim 6, characterized in that The acquisition unit is further configured to: A data quality inspection request is received, a corresponding incremental identifier is obtained, and corresponding incremental data is obtained based on the incremental identifier.

8. The device according to claim 7, characterized in that The acquisition unit is further configured to: Obtaining a data source identifier in the data quality inspection request, and obtaining corresponding source data from a corresponding source database based on the data source identifier; According to the incremental identifier, corresponding incremental data is obtained from the source data.

9. The device according to claim 7, characterized in that The acquisition unit is further configured to: The data acquisition frequency identifier carried in the data quality inspection request is obtained, and then the corresponding incremental data is acquired according to the incremental identifier at the frequency corresponding to the data acquisition frequency identifier.

10. The device according to claim 6, characterized in that The data splitting unit is further configured to: Determine the amount of data that each of the idle quality inspection nodes can currently accommodate; Based on the quantity and each data amount, the incremental data is split to obtain split data.

11. A data quality inspection electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images