A data quality checking method, device and equipment

Abstract

Embodiments of the present specification disclose a data quality checking method, device and equipment, the method can acquire a data table to be checked and business attribute information of fields in the data table, wherein one field corresponds to one business attribute, and business attributes of different fields are same or different; a first mining strategy is determined according to the business attribute information of a target field in the data table, wherein the first mining strategy is a data quality checking rule mining strategy corresponding to a target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field; a first data quality checking rule for the target field is mined from historical data corresponding to the target field in the data table according to the first mining strategy, and is added to a rule library; and the quality of newly added data in the data table is checked according to data quality checking rules in the rule library.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a data quality verification method, apparatus, and device. Background Technology

[0002] With the development of data storage and big data computing technologies, the importance of data is self-evident. Currently, many new technologies and products, such as shopping preference recommendation models and public opinion monitoring models, are developed based on big data. It is foreseeable that even more new technologies and products will flourish on the foundation of big data in the future. Of course, the prerequisite for the healthy development of these new technologies and products is that the data they rely on for analysis is healthy and of high quality, which necessitates data quality verification.

[0003] Currently, data quality verification rules are manually written to check data quality. However, manually writing these rules requires professionals who are very familiar with the data. Because training professionals is slow, and the manual writing process is also slow, the efficiency of writing data quality verification rules is low, resulting in low overall data quality verification efficiency. Summary of the Invention

[0004] This specification provides a data quality verification method, apparatus, and device to improve the efficiency of data quality verification.

[0005] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0006] Firstly, a data quality verification method is proposed, including:

[0007] Obtain the business attribute information of the data table to be verified and the fields in the data table, wherein one field corresponds to one business attribute, and different fields may have the same or different business attributes;

[0008] A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field;

[0009] According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base;

[0010] The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base.

[0011] Secondly, a data quality verification device is proposed, comprising:

[0012] The data acquisition module acquires the data table to be verified and the business attribute information of the fields in the data table. Each field corresponds to a business attribute, and different fields may have the same or different business attributes.

[0013] The first determining module determines a first mining strategy based on the business attribute information of the target field in the data table. The first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute. The target field is any field in the data table, and the target business attribute is the business attribute of the target field.

[0014] The first mining module extracts a first data quality verification rule for the target field from the historical data corresponding to the target field in the data table according to the first mining strategy and adds it to the rule base.

[0015] The quality verification module verifies the quality of newly added data in the data table according to the data quality verification rules in the rule base.

[0016] Thirdly, an electronic device is proposed, comprising:

[0017] Processor; and

[0018] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations:

[0019] Obtain the business attribute information of the data table to be verified and the fields in the data table, wherein one field corresponds to one business attribute, and different fields may have the same or different business attributes;

[0020] A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field;

[0021] According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base;

[0022] The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base.

[0023] Fourthly, a computer-readable storage medium is provided that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations:

[0024] Obtain the business attribute information of the data table to be verified and the fields in the data table, wherein one field corresponds to one business attribute, and different fields may have the same or different business attributes;

[0025] A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field;

[0026] According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base;

[0027] The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base.

[0028] The at least one technical solution provided in the embodiments of this specification can automatically mine data quality verification rules for field dimensions from the historical data of the data table to be verified and add them to the rule base. Then, it can automatically verify the quality of newly added data in the data table according to the data quality verification rules in the rule base. Therefore, it can improve the efficiency of data quality verification. Attached Figure Description

[0029] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and their descriptions, serving to explain this specification and do not constitute an undue limitation thereof.

[0030] In the picture:

[0031] Figure 1 This is a flowchart illustrating a data quality verification method provided in one embodiment of this specification.

[0032] Figure 2 This is a schematic diagram of a filtering model for filtering fields based on business attributes, provided in one embodiment of this specification.

[0033] Figure 3 This is a flowchart illustrating a data quality verification method provided in another embodiment of this specification.

[0034] Figure 4This is a schematic diagram of the implementation architecture of a data quality verification method provided in one embodiment of this specification.

[0035] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification.

[0036] Figure 6 This is a schematic diagram of the structure of a data quality verification device provided in one embodiment of this specification.

[0037] Figure 7 This is a schematic diagram of a data quality verification device provided in another embodiment of this specification. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this document.

[0039] To improve the efficiency of data quality verification, embodiments of this specification provide a data quality verification method and apparatus. This method and apparatus can be executed by one or more electronic devices, or by software or hardware devices installed in electronic devices. The electronic devices here include, but are not limited to, terminal devices and servers. The terminal devices include, but are not limited to, any one of the following smart terminal devices: smartphones, personal computers (PCs), laptops, tablets, e-readers, smart TVs, wearable devices, etc. The servers include, but are not limited to, any one of the following: a single server, multiple servers, a server cluster, and a cloud server.

[0040] The following describes a data quality verification method provided in the embodiments of this specification.

[0041] like Figure 1 As shown in the embodiments of this specification, a data quality verification method may include:

[0042] Step 102: Obtain the data table to be verified and the business attribute information of the fields in the data table.

[0043] The number of data tables to be verified can be one or several. That is, the data quality verification method provided in this specification supports batch operations on data tables (including batch mining of data quality verification rules and batch data quality verification), specifically supporting batch submission and calculation operations of thousands or more data tables.

[0044] The business attribute information of a field includes the business attributes of that field. Generally, one field corresponds to one business attribute, and different fields may have the same or different business attributes.

[0045] The types of business attributes of fields in a data table are related to the business scenario of the data recorded in the table. For different business scenarios, the types and number of fields contained in the data table may be different, and correspondingly, the business attributes of the fields in the data table may also be different. Taking the online shopping scenario as an example, the fields contained in its data table may include user account, order amount, product category, order number, payment currency, and payment time, etc.

[0046] For business scenarios like online shopping, the business attributes of fields in a data table can be categorized as follows:

[0047] 1) Fund type, used to identify the business attributes of fields related to fund amount. These fields can be fields such as order amount, coupon amount, etc.

[0048] 2) Status type, used to identify business attributes of fields whose values ​​are enumerable, or in other words, for set partitioning. These fields can be product category, order status, etc.

[0049] 3) Transaction type, used to identify the business attribute of a field whose value in each row is unique (such as a unique identification code). Such fields can be order number, express tracking number, account number, etc.

[0050] 4) Currency type: This is a business attribute used to identify fields whose values ​​are currency codes. The currency code can be CNY, USD, etc.

[0051] 5) Time type, used to identify the business attribute of a field whose value is time.

[0052] 6) Other types, namely business attributes other than the five types mentioned above. Generally, fields of other types are not within the scope of data quality control. Therefore, data quality verification rules need not be mined for such fields later.

[0053] Of course, for business scenarios such as online shopping or other business scenarios, the business attributes of the field can be of other types, and the embodiments in this specification do not limit this.

[0054] Optionally, before step 102, the business attribute information of the fields in the data table can be obtained by identifying the business attributes of the fields in the data table.

[0055] There are many ways to identify the business attributes of fields in the data table. Two examples are listed below.

[0056] First identification method

[0057] The data table is traversed, and the following steps are performed for the current field: the business attributes of the current field are identified and marked according to the business attribute identification rules. In specific implementation, corresponding identification rules can be set for different business attributes (as shown in Table 1 below). By traversing the business attribute identification rules corresponding to these business attributes and determining whether the current field matches some or all of the identification rules corresponding to the current business attribute, if it matches, the current business attribute is determined as the business attribute of the current field.

[0058] Table 1 Business Attribute Identification Rules

[0059]

[0060] The second identification method

[0061] Based on the hierarchical business attribute identification rules, the fields in the data table are filtered layer by layer, and each layer of filtered fields is identified with the business attribute corresponding to that layer. Each layer corresponds to one business attribute.

[0062] This business attribute identification method is essentially a funnel-shaped filtering model. The input to this model is the data table, and the model includes multiple levels of filters. Each level of the filter is used to select a field with a specific business attribute from the data table. Specifically, the filtering can be performed using the identification rules corresponding to that business attribute (as shown in Table 1 above). Figure 2 As shown, five levels of screens can be set from the opening to the bottom of the funnel, and the level number of the screens gradually increases from top to bottom. After the data table is input into the screening model, fields with corresponding business attributes can be screened layer by layer. As the screen level number increases, the number of fields to be screened gradually decreases, and the screening criteria also gradually become more lenient. Fields not screened in the first five levels are classified into other types of business attributes.

[0063] As an example, the correspondence between the above five levels and business attributes can include:

[0064] The business attribute corresponding to the first level is time type;

[0065] The business attribute corresponding to the second level is the currency type;

[0066] The business attribute corresponding to the third level is the status type;

[0067] The business attribute corresponding to the fourth level is the transaction type;

[0068] The business attribute corresponding to the fifth level is the type of funds.

[0069] The above-described correspondence between screening levels and business attributes represents a superior approach. First, the first level filters out the most distinctive or easily filtered time-related fields. Then, each level filters out and identifies fields based on currency type, status type, transaction history, and fund type. This improves the overall efficiency of business attribute identification in the data table. Therefore, when designing the business attributes for each screening level, more distinctive and easier-to-filter business attributes can be associated with levels having lower sequence numbers to further enhance the efficiency of business attribute identification in the data table.

[0070] Of course, the correspondence between the screening level and the business attribute can be other cases, and the embodiments in this specification do not limit this.

[0071] The comparison revealed that, compared to the first identification method, the second identification method can improve the efficiency of identifying business attributes of fields because it does not require iterating through all the identification rules corresponding to each business attribute for each field.

[0072] Step 104: Determine a first mining strategy based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field.

[0073] As an example, determining the first mining strategy based on the business attribute information of the target field in the data table may include: determining the first mining strategy based on the first correspondence and the business attribute information of the target field in the data table.

[0074] The first correspondence is a relationship between at least one business attribute, at least one data quality verification rule type, and at least one data quality verification rule mining strategy. The at least one business attribute includes the target business attribute. When the first correspondence is known, the target data quality verification rule type corresponding to the target business attribute can be found from the first correspondence based on the target business attribute, and then the first mining strategy can be found from the first correspondence based on the target data quality verification rule type.

[0075] Using the example above, the at least one business attribute may include at least one of the following: fund type, status type, transaction type, currency type, and time type. Based on this, the first correspondence can be represented by Table 2 below.

[0076] Table 2 First Correspondence Relationship

[0077]

[0078]

[0079] As can be seen from Table 2:

[0080] If the business attribute is a fund type, the data quality verification rule type corresponding to the business attribute includes at least one of the following: numeric field group stability rule, numeric field threshold rule, numeric field mean rule, and null value rate detection rule;

[0081] If the business attribute is a status type or currency type, the data quality verification rule type corresponding to the business attribute includes at least one of the following: group stability rule of enumeration field, enumeration range rule, and null value rate detection rule;

[0082] If the business attribute is a transaction type, the data quality verification rule type corresponding to the business attribute includes at least one of the following: field length detection rule, null value rate detection rule, and duplicate data detection rule;

[0083] If the business attribute is of type time, the data quality verification rule type corresponding to the business attribute includes at least one of the following: field length detection rule, field format detection rule, and null value rate detection rule.

[0084] Step 106: Based on the first mining strategy, mine the first data quality verification rule for the target field from the historical data corresponding to the target field in the data table and add it to the rule base.

[0085] For example, the data quality verification rule mining strategy in Table 1 can be used to mine the first data quality verification rule for the target field from the historical data corresponding to the target field in the data table and add it to the rule base.

[0086] Step 108: Verify the quality of newly added data in the data table according to the data quality verification rules in the rule base.

[0087] In a rule base, there may generally be multiple data quality verification rules of the same type. For example, there may be data quality verification rules for the same field mined at different time points. In this case, it may be necessary to select one of these data quality verification rules to perform the corresponding data quality verification. Therefore, step 108 above may specifically include: determining the data quality verification capability (control capability) of data quality verification rules of the same type in the rule base; selecting a data quality verification rule with a high data quality verification capability from the data quality verification rules of the same type in the rule base, and verifying the quality of newly added data in the data table.

[0088] More specifically, assuming that data quality verification is required for the target field, step 108 may include: determining the data quality verification capability of the first data quality verification rule of the same type in the rule base; selecting the first data quality verification rule with high data quality verification capability from the first data quality verification rules of the same type in the rule base, and verifying the quality of the newly added data under the target field in the data table.

[0089] In practical applications, a rule recommendation system can be set up based on the rule base. This system selects data quality verification rules with high data quality verification capabilities from the same type of data quality verification rules in the rule base, and uses these rules to verify the quality of newly added data in the data table. In other words, data quality verification rules with strong control capabilities will be given priority.

[0090] Specifically, different types of data quality verification rules in the rule base can be sorted according to their data quality verification capabilities. Then, data quality verification rules with high data quality verification capabilities can be selected based on the sorting to verify the quality of newly added data in the data table.

[0091] As an example, data quality verification capability (control capability) can be divided into three levels: high, medium, and low. Furthermore, the sorting strategies may differ for different types of data quality verification rules. Table 3 below lists some sorting strategies for different types of data quality verification rules for reference.

[0092] Table 3 Sorting Strategy for Data Quality Verification Rules

[0093]

[0094]

[0095] Figure 1The illustrated embodiment provides a data quality verification method that can automatically mine data quality verification rules for each field dimension from the historical data of the data table to be verified and add them to a rule base. Then, it automatically verifies the quality of newly added data in the data table based on the data quality verification rules in the rule base. Therefore, it can significantly improve the efficiency of data quality verification rule mining, thereby improving the overall efficiency of data quality verification. For example, assuming that 1,720,827 data quality verification rules need to be mined, and each rule takes 5 minutes to write manually, it would require approximately 3,585 people working for one day. In contrast, the development and deployment of this solution requires approximately 100 person-days. Compared to this, this solution not only significantly improves the efficiency of data quality verification rule mining but also reduces a large amount of manual labor.

[0096] also, Figure 1 The data quality verification method provided in the embodiment can extract data quality verification rules for fields with business attributes (such as methods for field-level stability rules), instead of treating all fields indiscriminately. Therefore, it can improve the accuracy of data quality verification on the one hand, and save computing and time resources on the other.

[0097] besides, Figure 1 The illustrated embodiment provides a data quality verification method that, by identifying the business attributes of field dimensions in a data table and mining data quality verification rules, can support numerical characteristic verification of fields whose metadata type is string. Here, "numerical characteristic verification of fields whose metadata type is string" means that this solution can identify that the business attribute of a field with a value of '123…' is a numeric type.

[0098] Furthermore, the data quality verification method provided in the embodiments of this specification supports batch operations on data tables (including batch mining of data quality verification rules and batch data quality verification), specifically supporting batch submission and calculation operations of thousands or more data tables, which further improves the efficiency of data quality verification.

[0099] Optional, such as Figure 3 As shown in the embodiments of this specification, a data quality verification method includes:

[0100] Step 102: Obtain the data table to be verified and the business attribute information of the fields in the data table.

[0101] The number of data tables to be verified can be one or several. That is, the data quality verification method provided in this specification supports batch operations on data tables (including batch mining of data quality verification rules and batch data quality verification), specifically supporting batch submission and calculation operations of thousands or more data tables.

[0102] The business attribute information of a field includes the business attributes of that field. Generally, one field corresponds to one business attribute, and different fields may have the same or different business attributes.

[0103] The types of business attributes of fields in a data table are related to the business scenarios of the data recorded in the data table. For different business scenarios, the types and number of fields contained in the data table may be different, and correspondingly, the business attributes of the fields in the data table may also be different.

[0104] Optionally, before step 102, the business attribute information of the fields in the data table can be obtained by identifying the business attributes of the fields in the data table. Please refer to the above for specific identification methods, which will not be repeated here.

[0105] Step 104: Determine a first mining strategy based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field.

[0106] As an example, determining the first mining strategy based on the business attribute information of the target field in the data table may include: determining the first mining strategy based on the first correspondence and the business attribute information of the target field in the data table.

[0107] The first correspondence is a relationship between at least one business attribute, at least one data quality verification rule type, and at least one data quality verification rule mining strategy. The at least one business attribute includes the target business attribute. When the first correspondence is known, the target data quality verification rule type corresponding to the target business attribute can be found from the first correspondence based on the target business attribute, and then the first mining strategy can be found from the first correspondence based on the target data quality verification rule type.

[0108] Using the example above, the at least one business attribute may include at least one of the following: fund type, status type, transaction type, currency type, and time type. Based on this, the first correspondence can be referenced in Table 2 above.

[0109] Step 106: Based on the first mining strategy, mine the first data quality verification rule for the target field from the historical data corresponding to the target field in the data table and add it to the rule base.

[0110] For example, the data quality verification rule mining strategy in Table 1 can be used to mine the first data quality verification rule for the target field from the historical data corresponding to the target field in the data table and add it to the rule base.

[0111] Step 110: Determine the second mining strategy, wherein the second mining strategy is the data quality verification rule mining strategy for the table dimension.

[0112] As an example, a second mining strategy can be determined based on a second correspondence, wherein the second correspondence is the correspondence between at least one table dimension's data quality verification rule type and at least one table dimension's data quality verification rule mining strategy. The at least one table dimension's data quality verification rule type may include, but is not limited to, at least one of row count detection rules and idempotency rules. Based on this, Table 4 below shows some examples of the second correspondence.

[0113] Table 4 Second Correspondence

[0114]

[0115] It should be noted that, in the embodiments of this specification, the data in the data table can be divided into data partitions based on the time of data generation or other criteria. For example, data generated on the same date can be divided into one data partition, and data generated on different dates can be divided into different data partitions. A data partition contains at least one row of data.

[0116] Step 112: Based on the second mining strategy, mine the second data quality verification rules of the table dimension from the historical data in the data table and add them to the rule base.

[0117] For example, the data quality verification rule mining strategy in Table 4 can be used to mine the first data quality verification rule of the table dimension from the historical data in the data table and add it to the rule base.

[0118] As can be seen, this embodiment can uncover not only data quality verification rules at the field level but also at the table level, thereby enabling quality verification of newly added data in the data table from these two aspects. Specifically, the table-level data quality verification rules primarily prevent sudden changes in data volume between different partitions in the data table (implemented using row count detection rules) and the occurrence of duplicate data (implemented using idempotency rules).

[0119] Step 108: Verify the quality of newly added data in the data table according to the data quality verification rules in the rule base.

[0120] Specifically, in this embodiment, step 108 may include: verifying the quality of newly added data under the target field in the data table according to the first data quality verification rule in the rule base; and verifying the quality of newly added data in the table dimension of the data table according to the second data quality verification rule in the rule base.

[0121] In a rule base, there may be multiple data quality verification rules for the same type. For example, there may be data quality verification rules for the same field mined at different time points. In this case, it may be necessary to select one of the data quality verification rules to perform the corresponding data quality verification.

[0122] Therefore, the step of verifying the quality of newly added data under the target field in the data table according to the first data quality verification rule in the rule base may include: determining the data quality verification capability of the first data quality verification rule of the same type in the rule base; selecting the first data quality verification rule with high data quality verification capability from the first data quality verification rules of the same type in the rule base, and verifying the quality of newly added data under the target field in the data table.

[0123] The step of verifying the quality of newly added data in the table dimension of the data table according to the second data quality verification rule in the rule base may include: determining the data quality verification capability of the second data quality verification rule of the same type in the rule base; selecting the second data quality verification rule with high data quality verification capability from the second data quality verification rules of the same type in the rule base, and verifying the quality of newly added data in the table dimension of the data table.

[0124] In practical applications, a rule recommendation system can be set up based on the rule base. This system selects data quality verification rules with high data quality verification capabilities from the same type of data quality verification rules in the rule base, and uses these rules to verify the quality of newly added data in the data table. In other words, data quality verification rules with strong control capabilities will be given priority.

[0125] Specifically, different types of data quality verification rules in the rule base can be sorted according to their data quality verification capabilities. Then, data quality verification rules with high data quality verification capabilities can be selected based on the sorting to verify the quality of newly added data in the data table.

[0126] As an example, data quality verification capability (control capability) can be divided into three levels: high, medium, and low. Furthermore, the sorting strategies may differ for different types of data quality verification rules. Table 3 above lists some sorting strategies for different types of data quality verification rules for reference. Please refer to Table 3 for details.

[0127] Figure 3 The illustrated embodiment provides a data quality verification method that, in addition to being able to obtain... Figure 1 The technical effect of the data quality verification method provided in the illustrated embodiment is that it can automatically extract data quality verification rules from the historical data of the data table to be verified in terms of both table dimensions and field dimensions. Therefore, it can automatically realize the quality verification of newly added data in the data table in these two aspects, making the data quality verification more comprehensive.

[0128] Optional, in Figure 1 or Figure 3 Based on any of the embodiments shown, the data quality verification method provided in this specification may further include: issuing an alarm and generating alarm information when the quality of newly added data in the data table is abnormal. The abnormality here may include data abnormalities at the table level or data abnormalities at the field level.

[0129] It is understandable that issuing an alarm when the quality of newly added data in the data table is abnormal can remind the data table maintenance personnel to investigate or perform other operations on the abnormalities in the data table, so as to avoid using problematic data for the development of new technologies and new products.

[0130] Furthermore, the data quality verification method provided in this specification may further include: determining a high-risk alarm judgment strategy based on the data quality verification rules for generating the alarm information; determining whether the alarm corresponding to the alarm information is a high-risk alarm based on the high-risk alarm judgment strategy; and outputting the high-risk alarm if it is.

[0131] The high-risk alarm judgment strategy may differ for different types of data quality verification rules. Table 5 below lists some data quality verification rules corresponding to different data quality verification rules. Please refer to Table 5 for details.

[0132] Table 5 High-risk alarm judgment strategy for data quality verification rules

[0133]

[0134] Furthermore, the data quality verification method provided in this specification may also include:

[0135] Determine whether the newly added data corresponding to the alarm information is abnormal;

[0136] If no anomalies are found, the data quality verification rules for generating the alarm information are re-mined from the data table.

[0137] The purpose of this embodiment is to verify whether the newly added data corresponding to the alarm information is truly abnormal. If an abnormality is indeed found, the data table maintenance personnel are reminded to investigate or perform other operations on the abnormality in the data table. If no abnormality is found, it indicates that the data quality verification rules corresponding to the alarm information may have expired or are no longer applicable, and need to be re-mined to find data quality verification rules that can detect the actual abnormality. This step can be regarded as the process of updating or keeping the data quality verification rules fresh, so as to ensure that the data quality verification rules are effective (not expired).

[0138] It should be noted that in the embodiments of this specification, the data table to be verified can be an offline data table or an online data table, but it is usually an offline data table because the data in an offline data table is more stable.

[0139] Figure 4 This diagram illustrates an implementation architecture of a data quality verification method provided in an embodiment of this specification. Figure 4 As shown, the implementation architecture may include a big data storage system 41, a business attribute identification system 42, an intelligent rule mining system 43, a rule recommendation system 44, a rule execution system 45, and an alarm analysis system 46.

[0140] Big data storage system 41 is used to store data tables to be verified. Specifically, big data storage system 41 can serve multiple business lines. The big data resources of different business lines are generally divided according to the organization method of project space. That is to say, the data tables of different business lines are stored in different project spaces. The big data storage system, also known as a big data engine, provides services such as data storage and analysis.

[0141] In practical applications, the smallest deployment unit of the data quality verification method provided in this specification can be a project space of a big data storage system. The field business attribute identification system 42 and the intelligent rule mining system 43 can be deployed to different project spaces in a distributed manner. A project space is a management unit for big data resources; a project space is a unit with complete big data service functions, and there are clear boundaries between project spaces. Access control is implemented when resources in different project spaces access each other. The rule recommendation system 44 and the alarm analysis system 46 can be implemented using web services. The big data storage system 41 itself and the rule execution system 45 can be third-party platforms.

[0142] Based on this, specifically:

[0143] The business attribute identification system 42 can be used to identify the business attributes of fields in data tables from the project space.

[0144] The intelligent rule mining system 43 can be used to perform step 106 above to mine data quality verification rules for field dimensions from the data table and add them to the rule base, and / or can be used to perform step 112 above to mine data quality verification rules for table dimensions from the data table and add them to the rule base.

[0145] The rule recommendation system 44 can be used to determine the data quality verification capability (control capability) of data quality verification rules of the same type in the rule base; from the data quality verification rules of the same type in the rule base, it selects data quality verification rules with high data quality verification capability and recommends them to the rule execution system 45 so as to verify the quality of newly added data in the data table.

[0146] The rule execution system 45 can be used to execute the above step 108 to verify the quality of newly added data in the data table according to the data quality verification rules in the rule base.

[0147] The alarm analysis system 46 can be used to generate alarm information when the quality of newly added data in the data table is abnormal. Optionally, it can also be used to determine a high-risk alarm judgment strategy based on the data quality verification rules for generating the alarm information; determine whether the alarm corresponding to the alarm information is a high-risk alarm based on the high-risk alarm judgment strategy; if so, output the high-risk alarm; further, it can also be used to determine whether the newly added data corresponding to the alarm information is abnormal; if there is no abnormality, re-mining the data quality verification rules for generating the alarm information from the data table to update the data quality verification rules corresponding to the alarm information.

[0148] As can be seen from the above implementation architecture, different steps in the data quality verification method provided in the embodiments of this specification may be implemented by different execution entities, and are not limited to one execution entity.

[0149] The methods and their implementation architecture provided in this manual have been described above. The electronic devices provided in this manual are described below.

[0150] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification. Please refer to it. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0151] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0152] Memory stores programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0153] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a data quality verification device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0154] Obtain the business attribute information of the data table to be verified and the fields in the data table, wherein one field corresponds to one business attribute, and different fields may have the same or different business attributes;

[0155] A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field;

[0156] According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base;

[0157] The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base.

[0158] The above is as described in this instruction manual. Figure 1 or Figure 3The methods disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in one or more embodiments of this specification can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in one or more embodiments of this specification can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0159] The electronic device can also perform Figure 1 or Figure 3 The methods provided in the illustrated embodiments will not be described again in this specification.

[0160] Of course, in addition to software implementation, the electronic device described in this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0161] This specification also provides an embodiment of a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1 or Figure 3 The method of the illustrated embodiment is specifically used to perform the following operations:

[0162] Obtain the business attribute information of the data table to be verified and the fields in the data table, wherein one field corresponds to one business attribute, and different fields may have the same or different business attributes;

[0163] A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field;

[0164] According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base;

[0165] The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base.

[0166] like Figure 6 As shown, one embodiment of this specification provides a data quality verification device 600. In one software implementation, the device 600 may include: a data acquisition module 601, a first determination module 602, a first mining module 603, and a quality verification module 604.

[0167] The data acquisition module 601 acquires the data table to be verified and the business attribute information of the fields in the data table. Each field corresponds to a business attribute, and the business attributes of different fields may be the same or different.

[0168] The number of data tables to be verified can be one or several. That is, the data quality verification method provided in this specification supports batch operations on data tables (including batch mining of data quality verification rules and batch data quality verification), specifically supporting batch submission and calculation operations of thousands or more data tables.

[0169] The business attribute information of a field includes the business attributes of that field. Generally, one field corresponds to one business attribute, and different fields may have the same or different business attributes.

[0170] The types of business attributes of fields in a data table are related to the business scenario of the data recorded in the table. For different business scenarios, the types and number of fields contained in the data table may be different, and correspondingly, the business attributes of the fields in the data table may also be different. Taking the online shopping scenario as an example, the fields contained in its data table may include user account, order amount, product category, order number, payment currency, and payment time, etc.

[0171] For business scenarios like online shopping, the business attributes of fields in the data table can be categorized as follows: fund type, status type, transaction type, currency type, time type, and other types. Generally, other types of fields are not within the scope of data quality control; therefore, data quality verification rules will not be developed for these fields in the following sections.

[0172] Of course, for business scenarios such as online shopping or other business scenarios, the business attributes of the field can be other possible types, and the embodiments in this specification do not limit this.

[0173] Optionally, the device 600 may further include: a business attribute identification module, which identifies the business attributes of the fields in the data table to obtain the business attribute information of the fields in the data table.

[0174] There are many ways to identify the business attributes of fields in the data table. Two examples are listed below.

[0175] First identification method

[0176] The data table is traversed, and the following steps are performed for the current field: the business attributes of the current field are identified and marked according to the business attribute identification rules. In specific implementation, corresponding identification rules can be set for different business attributes (as shown in Table 1 below). By traversing the business attribute identification rules corresponding to these business attributes and determining whether the current field matches some or all of the identification rules corresponding to the current business attribute, if it matches, the current business attribute is determined as the business attribute of the current field.

[0177] The second identification method

[0178] Based on the hierarchical business attribute identification rules, the fields in the data table are filtered layer by layer, and each layer of filtered fields is identified with the business attribute corresponding to that layer. Each layer corresponds to one business attribute.

[0179] This business attribute identification method is equivalent to using a funnel-shaped filtering model. The input of the filtering model is the data table. The filtering model includes multiple levels of screens. Each level of screen is used to filter out a field of a business attribute from the data table. Specifically, the filtering can be performed using the identification rules corresponding to the business attribute (such as the business attribute identification rules shown in Table 1 above).

[0180] The comparison revealed that, compared to the first identification method, the second identification method can improve the efficiency of identifying business attributes of fields because it does not require iterating through all the identification rules corresponding to each business attribute for each field.

[0181] The first determining module 602 determines a first mining strategy based on the business attribute information of the target field in the data table. The first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute. The target field is any field in the data table, and the target business attribute is the business attribute of the target field.

[0182] As an example, determining the first mining strategy based on the business attribute information of the target field in the data table may include: determining the first mining strategy based on the first correspondence and the business attribute information of the target field in the data table.

[0183] The first correspondence is a relationship between at least one business attribute, at least one data quality verification rule type, and at least one data quality verification rule mining strategy. The at least one business attribute includes the target business attribute. When the first correspondence is known, the target data quality verification rule type corresponding to the target business attribute can be found from the first correspondence based on the target business attribute, and then the first mining strategy can be found from the first correspondence based on the target data quality verification rule type.

[0184] The first mining module 603, according to the first mining strategy, mines the first data quality verification rule for the target field from the historical data corresponding to the target field in the data table and adds it to the rule base.

[0185] For example, the data quality verification rule mining strategy in Table 1 can be used to mine the first data quality verification rule for the target field from the historical data corresponding to the target field in the data table and add it to the rule base.

[0186] The quality verification module 604 verifies the quality of newly added data in the data table according to the data quality verification rules in the rule base.

[0187] In a rule base, there may generally be multiple data quality verification rules of the same type. For example, there may be data quality verification rules for the same field mined at different time points. In this case, it may be necessary to select one of these data quality verification rules to perform the corresponding data quality verification. Therefore, the aforementioned quality verification module 604 can be used to: determine the data quality verification capability (control capability) of data quality verification rules of the same type in the rule base; and select a data quality verification rule with a high data quality verification capability from the data quality verification rules of the same type in the rule base to verify the quality of newly added data in the data table.

[0188] More specifically, assuming that data quality verification is required for the target field, the aforementioned quality verification module 604 can be used to: determine the data quality verification capability of the first data quality verification rule of the same type in the rule base; select the first data quality verification rule with high data quality verification capability from the first data quality verification rules of the same type in the rule base, and verify the quality of the newly added data under the target field in the data table.

[0189] Figure 6 The data quality verification device 600 provided in the embodiment can automatically mine data quality verification rules for field dimensions from the historical data of the data table to be verified and add them to the rule base. Then, it automatically verifies the quality of newly added data in the data table according to the data quality verification rules in the rule base. Therefore, it can greatly improve the mining efficiency of data quality verification rules, thereby improving the data quality verification efficiency.

[0190] also, Figure 6 The data quality verification device 600 provided in the embodiment can extract data quality verification rules (such as the stability rules of field dimensions) with business attributes, instead of treating all fields indiscriminately. Therefore, on the one hand, it can improve the accuracy of data quality verification, and on the other hand, it can save computing resources and time resources.

[0191] besides, Figure 6 The illustrated embodiment provides a data quality verification device 600 that, by identifying the business attributes of field dimensions in a data table and mining data quality verification rules, can support numerical characteristic verification of fields whose metadata type is string. The numerical characteristic verification of fields whose metadata type is string refers to the ability of this solution to identify that the business attribute of a field with a value of '123…' is a numeric type.

[0192] Furthermore, the embodiments provided in this specification Figure 6 The data quality verification device 600 provided in the embodiment supports batch operations on data tables (including batch mining of data quality verification rules and batch data quality verification). Specifically, it can support batch submission and calculation operations of thousands or more data tables, which further improves the efficiency of data quality verification.

[0193] Optional, such as Figure 7 As shown, one embodiment of this specification provides a data quality verification device 600. In one software implementation, the device 600 may include: a data acquisition module 601, a first determination module 602, a first mining module 603, a second determination module 605, a second mining module 606, and a quality verification module 604.

[0194] The data acquisition module 601 acquires the data table to be verified and the business attribute information of the fields in the data table. Each field corresponds to a business attribute, and the business attributes of different fields may be the same or different.

[0195] The first determining module 602 determines a first mining strategy based on the business attribute information of the target field in the data table. The first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute. The target field is any field in the data table, and the target business attribute is the business attribute of the target field.

[0196] The first mining module 603, according to the first mining strategy, mines the first data quality verification rule for the target field from the historical data corresponding to the target field in the data table and adds it to the rule base.

[0197] The second determining module 605 determines the second mining strategy, wherein the second mining strategy is a data quality verification rule mining strategy for the table dimension.

[0198] As an example, a second mining strategy can be determined based on a second correspondence, wherein the second correspondence is a correspondence between data quality verification rule types for at least one table dimension and data quality verification rule mining strategies for at least one table dimension. The data quality verification rule types for the at least one table dimension may include, but are not limited to, at least one of row count detection rules and idempotency rules.

[0199] The second mining module 606 mines the second data quality verification rules of the table dimension from the historical data in the data table according to the second mining strategy and adds them to the rule base.

[0200] As can be seen, this embodiment can uncover not only data quality verification rules at the field level but also at the table level, thereby enabling quality verification of newly added data in the data table from these two aspects. Specifically, the table-level data quality verification rules primarily prevent sudden changes in data volume between different partitions in the data table (implemented using row count detection rules) and the occurrence of duplicate data (implemented using idempotency rules).

[0201] The quality verification module 604 verifies the quality of newly added data in the data table according to the data quality verification rules in the rule base.

[0202] Specifically, in this embodiment, the quality verification module 604 can verify the quality of newly added data under the target field in the data table according to the first data quality verification rule in the rule base; and verify the quality of newly added data in the table dimension of the data table according to the second data quality verification rule in the rule base.

[0203] In a rule base, there may be multiple data quality verification rules for the same type. For example, there may be data quality verification rules for the same field mined at different time points. In this case, it may be necessary to select one of the data quality verification rules to perform the corresponding data quality verification.

[0204] Therefore, the quality verification module 604 can specifically determine the data quality verification capability of a first data quality verification rule of the same type in the rule base; select a first data quality verification rule with a high data quality verification capability from the first data quality verification rules of the same type in the rule base, and verify the quality of the newly added data under the target field in the data table; and / or, the quality verification module 604 can specifically determine the data quality verification capability of a second data quality verification rule of the same type in the rule base; select a second data quality verification rule with a high data quality verification capability from the second data quality verification rules of the same type in the rule base, and verify the quality of the newly added data in the table dimension of the data table.

[0205] In practical applications, a rule recommendation system can be set up based on the rule base. This system selects data quality verification rules with high data quality verification capabilities from the same type of data quality verification rules in the rule base, and uses these rules to verify the quality of newly added data in the data table. In other words, data quality verification rules with strong control capabilities will be given priority.

[0206] Figure 7 The illustrated embodiment provides a data quality verification device that, in addition to being able to obtain... Figure 6 The technical effect achieved by the data quality verification device provided in the illustrated embodiment is that it can automatically extract data quality verification rules from the historical data of the data table to be verified in terms of both table dimensions and field dimensions. Therefore, it can automatically realize the quality verification of newly added data in the data table in these two aspects, making the data quality verification more comprehensive.

[0207] Optional, in Figure 6 or Figure 7Based on any of the embodiments shown, the data quality verification device 600 provided in this specification may further include: an alarm module, which issues an alarm and generates alarm information when the quality of newly added data in the data table is abnormal. The abnormality here may include data abnormalities at the table level or data abnormalities at the field level.

[0208] It is understandable that issuing an alarm when the quality of newly added data in the data table is abnormal can remind the data table maintenance personnel to investigate or perform other operations on the abnormalities in the data table, so as to avoid using problematic data for the development of new technologies and new products.

[0209] Furthermore, the data quality verification device 600 provided in this specification may further include: a third determining module, which determines a high-risk alarm judgment strategy according to the data quality verification rules for generating the alarm information; and a judging module, which determines whether the alarm corresponding to the alarm information is a high-risk alarm based on the high-risk alarm judgment strategy, and if so, outputs the high-risk alarm.

[0210] The high-risk alarm judgment strategy may differ depending on the type of data quality verification rule.

[0211] Furthermore, the data quality verification device 600 provided in this specification may also include:

[0212] The fourth determining module determines whether there is any anomaly in the newly added data corresponding to the alarm information;

[0213] If the newly added data corresponding to the alarm information does not have any abnormalities, the triggering module will re-trigger the first mining module or the second mining module to mine the data quality verification rules for generating the alarm information from the data table.

[0214] The purpose of this embodiment is to verify whether the newly added data corresponding to the alarm information is truly abnormal. If an abnormality is indeed found, the data table maintenance personnel are reminded to investigate or perform other operations on the abnormality in the data table. If no abnormality is found, it indicates that the data quality verification rules corresponding to the alarm information may have expired or are no longer applicable, and need to be re-mined to find data quality verification rules that can detect the actual abnormality. This step can be regarded as the process of updating or keeping the data quality verification rules fresh, so as to ensure that the data quality verification rules are effective (not expired).

[0215] It should be noted that the data quality verification device 600 is capable of achieving... Figure 1 or Figure 3 A data quality verification method is provided, which can achieve the same technical effect. For details, please refer to the description of the method implementation section above, which will not be repeated here.

[0216] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0217] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0218] In summary, the above description is merely a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

[0219] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0220] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0221] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0222] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A data quality verification method, comprising: Obtain the business attribute information of the data table to be verified and the fields in the data table, wherein one field corresponds to one business attribute, and different fields may have the same or different business attributes; A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field; A second mining strategy is determined, and based on the second mining strategy, a second data quality verification rule for the table dimension is mined from the historical data in the data table and added to the rule base. The second mining strategy is a data quality verification rule mining strategy for the table dimension. According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base; The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base, wherein the second data quality verification rule is used to verify the quality of newly added data in the table dimension of the data table.

2. The method according to claim 1, further comprising, before obtaining the business attribute information of the fields in the data table: The business attributes of the fields in the data table are identified to obtain the business attribute information of the fields in the data table.

3. The method according to claim 2, wherein, The step of identifying the business attributes of the fields in the data table includes: Based on the hierarchical business attribute identification rules, the fields in the data table are filtered layer by layer, and each layer of filtered fields is identified with the business attribute corresponding to that layer. Each layer corresponds to one business attribute.

4. The method according to claim 1, wherein, The step of determining the first mining strategy based on the business attribute information of the target field in the data table includes: A first mining strategy is determined based on the first correspondence and the business attribute information of the target field in the data table. The first correspondence is a correspondence between at least one business attribute, at least one data quality verification rule type, and at least one data quality verification rule mining strategy. The at least one business attribute includes the target business attribute.

5. The method according to claim 4, wherein the at least one business attribute includes at least one of the following: Type of funds; Status type; Flow type; Currency type; Time type.

6. The method according to claim 5, If the business attribute is a fund type, the data quality verification rule type corresponding to the business attribute includes at least one of the following: group stability rule for numeric fields, threshold rule for numeric fields, mean rule for numeric fields, and null value rate detection rule. If the business attribute is a status type or currency type, the data quality verification rule type corresponding to the business attribute includes at least one of the following: enumeration field group stability rule, enumeration range rule, and null value rate detection rule; If the business attribute is a transaction type, the data quality verification rule type corresponding to the business attribute includes at least one of the following: field length detection rule, null value rate detection rule, and duplicate data detection rule; If the business attribute is of type time, the data quality verification rule type corresponding to the business attribute includes at least one of the following: field length detection rule, field format detection rule, and null value rate detection rule.

7. The method according to claim 1, wherein, The determination of the second mining strategy includes: A second mining strategy is determined based on a second correspondence, wherein the second correspondence is the correspondence between at least one table dimension's data quality verification rule type and at least one table dimension's data quality verification rule mining strategy.

8. The method according to claim 7, wherein the data quality verification rule type for the at least one table dimension includes at least one of the following: Row count detection rules; Idempotency rule.

9. The method according to claim 1, wherein the first data quality verification rule is used to verify the quality of newly added data under the target field in the data table.

10. The method according to claim 9, in, The step of verifying the quality of newly added data under the target field in the data table according to the first data quality verification rule in the rule base includes: Determine the data quality verification capability of the first data quality verification rules of the same type in the rule base; From the first data quality verification rules of the same type in the rule base, select the first data quality verification rule with high data quality verification capability, and verify the quality of the newly added data under the target field in the data table. The step of verifying the quality of newly added data in the table dimension of the data table according to the second data quality verification rule in the rule base includes: Determine the data quality verification capability of the second data quality verification rules of the same type in the rule base; From the second data quality verification rules of the same type in the rule base, select the second data quality verification rule with high data quality verification capability, and verify the quality of newly added data in the table dimension of the data table.

11. The method according to any one of claims 1-10, further comprising: An alarm is triggered and alarm information is generated when the quality of newly added data in the data table is abnormal.

12. The method of claim 11, further comprising: Based on the data quality verification rules for generating the alarm information, a high-risk alarm judgment strategy is determined; Based on the high-risk alarm judgment strategy, determine whether the alarm corresponding to the alarm information is a high-risk alarm; If so, output the high-risk warning.

13. The method of claim 11, further comprising: Determine whether the newly added data corresponding to the alarm information is abnormal; If no anomalies are found, the data quality verification rules for generating the alarm information are re-mined from the data table.

14. A data quality verification device, comprising: The data acquisition module acquires the data table to be verified and the business attribute information of the fields in the data table. Each field corresponds to a business attribute, and different fields may have the same or different business attributes. The first determining module determines a first mining strategy based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field; and determines a second mining strategy, and mines a second data quality verification rule of the table dimension from the historical data in the data table according to the second mining strategy and adds it to the rule base, wherein the second mining strategy is a data quality verification rule mining strategy of the table dimension; The first mining module extracts a first data quality verification rule for the target field from the historical data corresponding to the target field in the data table according to the first mining strategy and adds it to the rule base. The quality verification module verifies the quality of newly added data in the data table according to the data quality verification rules in the rule base. The second data quality verification rule is used to verify the quality of newly added data in the table dimension of the data table.

15. An electronic device comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: Obtain the business attribute information of the data table to be verified and the fields in the data table, wherein one field corresponds to one business attribute, and different fields may have the same or different business attributes; A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field; A second mining strategy is determined, and based on the second mining strategy, a second data quality verification rule for the table dimension is mined from the historical data in the data table and added to the rule base. The second mining strategy is a data quality verification rule mining strategy for the table dimension. According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base; The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base, wherein the second data quality verification rule is used to verify the quality of newly added data in the table dimension of the data table.

16. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations: Obtain the data table to be verified and the business attribute information of the fields in the data table, wherein, One field corresponds to one business attribute; different fields may have the same or different business attributes. A first mining strategy is determined based on the business attribute information of the target field in the data table, wherein the first mining strategy is a data quality verification rule mining strategy corresponding to the target business attribute, the target field is any field in the data table, and the target business attribute is the business attribute of the target field; A second mining strategy is determined, and based on the second mining strategy, a second data quality verification rule for the table dimension is mined from the historical data in the data table and added to the rule base. The second mining strategy is a data quality verification rule mining strategy for the table dimension. According to the first mining strategy, the first data quality verification rule for the target field is mined from the historical data corresponding to the target field in the data table and added to the rule base; The quality of newly added data in the data table is verified according to the data quality verification rules in the rule base, wherein the second data quality verification rule is used to verify the quality of newly added data in the table dimension of the data table.

Images