Data quality checking method and apparatus

Abstract

Embodiments of the present specification provide a data quality checking method and device, the method comprising: obtaining a plurality of sets of field descriptions and corresponding field value data sets; extracting field descriptions in the data sets to obtain a field description list, and extracting field values in the data sets to obtain a field value list; calculating a first similarity between any two field descriptions in the field description list; calculating a second similarity between any two field values in the field value list; and according to the first similarity between any two field descriptions and the second similarity between the two field values corresponding to the any two field descriptions, locating a data quality problem based on whether the two satisfy a pre-set attribution condition. The method can improve checking efficiency, reduce errors and reduce uncertainty.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computers, and more particularly to data quality inspection methods and apparatus. Background Technology

[0002] With the advent of the data era, data information is developing towards massive volumes and diversity, which may include private data. Since data anomalies can lead to serious losses or accidents, data quality testing is a crucial component of the data research and development system.

[0003] Traditional methods for data quality inspection require testers to construct comprehensive test cases for each field based on business logic. This involves testers writing the test case code and execution logic, and then judging whether the execution results contain anomalies and meet business requirements. Furthermore, after manual testing, if the repair of abnormal data is involved, manual regression testing is required again after data repair, repeatedly writing and executing test cases. Because this approach involves too much human intervention, there is a possibility of missing abnormal data, resulting in low test coverage. Moreover, test cases need to be executed repeatedly for different fields and for repaired data, leading to low efficiency, large errors, and high uncertainty. Summary of the Invention

[0004] This specification describes one or more embodiments of a data quality inspection method and apparatus that can improve inspection efficiency, reduce errors, and lower uncertainty.

[0005] Firstly, a data quality inspection method is provided, the method including:

[0006] Retrieve a data set containing multiple sets of field descriptions and their corresponding field values;

[0007] Extract field descriptions from the dataset to obtain a list of field descriptions, and extract field values ​​from the dataset to obtain a list of field values;

[0008] For any two field descriptions in the field description list, calculate their first similarity.

[0009] For any two field values ​​in the field value list, calculate their second similarity.

[0010] Based on the first similarity between any two field descriptions and the second similarity between the values ​​of the two fields corresponding to those two field descriptions, the data quality problem is located based on whether the two meet the pre-defined attribution conditions.

[0011] In one possible implementation, obtaining the data set of multiple sets of field descriptions and corresponding field values ​​includes:

[0012] From the display data of a front-end page, obtain a first subset of field descriptions and corresponding field values ​​from a first uniform resource locator (URL), and obtain a second subset of field descriptions and corresponding field values ​​from a second URL; the first subset and the second subset constitute the data set.

[0013] In one possible implementation, the data set is associated with the display data of the front-end page;

[0014] The data set for obtaining multiple sets of field descriptions and corresponding field values ​​includes:

[0015] By embedding tracking points in the front-end page or back-end control layer, message data with a preset structure can be obtained;

[0016] Extract several key-value pairs from the message data with a preset structure. In each key-value pair, the key is used as a field description and the value is used as the corresponding field value.

[0017] In one possible implementation, the dataset is associated with an offline database;

[0018] The data set for obtaining multiple sets of field descriptions and corresponding field values ​​includes:

[0019] Extract the field descriptions and corresponding values ​​of each field contained in the data table of the offline database.

[0020] In one possible implementation, calculating the first similarity between any two field descriptions in the field description list includes:

[0021] Encode any field description in the field description list using a language model to obtain the embedding vector corresponding to that field description;

[0022] Based on the embedding vectors corresponding to any two fields, the first similarity between the two is calculated using cosine similarity.

[0023] In one possible implementation, the field value is formed by concatenating the original values ​​of the same field corresponding to multiple business objects.

[0024] In one possible implementation, calculating the second similarity between any two field values ​​from the list of field values ​​includes:

[0025] For each of the two field values, calculate the corresponding first hash value and second hash value respectively;

[0026] The second similarity is obtained based on the XOR operation result of the first hash value and the second hash value.

[0027] In one possible implementation, locating data quality issues based on whether the two conditions satisfy pre-defined attribution criteria includes:

[0028] If the first similarity between any two field descriptions indicates that the two field descriptions are similar, and the second similarity between the two field values ​​corresponding to the two field descriptions indicates that the two field values ​​are not similar, then it is determined that there is a data consistency problem with the field values.

[0029] Output a prompt message to remind you to check the field value retrieval logic of the relevant business object.

[0030] In one possible implementation, locating data quality issues based on whether the two conditions satisfy pre-defined attribution criteria includes:

[0031] If the first similarity between any two field descriptions indicates that the two field descriptions are not similar, and the second similarity between the two field values ​​corresponding to the two field descriptions indicates that the two field values ​​are similar, then it is determined that there is a data consistency problem with redundant fields.

[0032] Output a prompt message to suggest checking the query fields.

[0033] In one possible implementation, locating data quality issues based on whether the two conditions satisfy pre-defined attribution criteria includes:

[0034] Based on the first similarity, a first matrix is ​​constructed; each row of the first matrix corresponds to the description of each field, and each column also corresponds to the description of each field.

[0035] Based on the second similarity, a second matrix is ​​constructed; each row of the second matrix corresponds to the value of each field, and each column also corresponds to the value of each field.

[0036] The first similarity between two field descriptions at any position in the first matrix is ​​obtained, and the second similarity between two field values ​​at the same position in the second matrix is ​​obtained, so as to obtain the corresponding first similarity and second similarity.

[0037] Secondly, a data quality inspection device is provided, the device comprising:

[0038] The retrieval unit is used to retrieve a data set containing multiple sets of field descriptions and their corresponding field values.

[0039] The extraction unit is used to extract field descriptions from the data set acquired by the acquisition unit to obtain a list of field descriptions, and to extract field values ​​from the data set to obtain a list of field values.

[0040] The first calculation unit is used to calculate the first similarity between any two field descriptions in the field description list obtained by the extraction unit.

[0041] The second calculation unit is used to calculate the second similarity between any two field values ​​in the field value list obtained by the extraction unit.

[0042] The localization unit is used to locate data quality problems based on whether the first similarity between any two field descriptions obtained by the first calculation unit and the second similarity between the values ​​of two fields corresponding to the arbitrary two field descriptions obtained by the second calculation unit meet the pre-set attribution conditions.

[0043] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0044] Fourthly, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0045] The method and apparatus provided in the embodiments of this specification first acquire a data set of multiple sets of field descriptions and corresponding field values; then, the field descriptions are extracted from the data set to obtain a list of field descriptions, and the field values ​​are extracted from the data set to obtain a list of field values; next, for any two field descriptions in the list of field descriptions, a first similarity is calculated between them; for any two field values ​​in the list of field values, a second similarity is calculated between them; finally, based on the first similarity between any two field descriptions and the second similarity between the two field values ​​corresponding to those two field descriptions, and based on whether they meet pre-set attribution conditions, data quality problems are located. As can be seen from the above, the embodiments of this specification, based on field value features, add field description features, reducing the need for manual intervention, automatically performing feature cross-validation between field description features and field value features, and combining the semantics of the data itself for analysis, making it easier to screen out potential data quality problems, reducing the work of large-scale manual data point verification, thereby improving inspection efficiency, reducing errors, and lowering uncertainty. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification;

[0048] Figure 2 A flowchart of a data quality inspection method according to one embodiment is shown;

[0049] Figure 3 This diagram illustrates a combination matrix of field descriptions according to one embodiment.

[0050] Figure 4 A schematic diagram of a first matrix according to one embodiment is shown;

[0051] Figure 5 This diagram illustrates a field value combination matrix according to one embodiment.

[0052] Figure 6 A schematic diagram of a second matrix according to one embodiment is shown;

[0053] Figure 7 This diagram illustrates a combination matrix based on field descriptions for a specific example.

[0054] Figure 8 This diagram illustrates a matrix of field value combinations based on a specific example.

[0055] Figure 9 A schematic diagram of the first matrix is ​​shown according to a specific example;

[0056] Figure 10 This illustrates a second matrix diagram based on a specific example;

[0057] Figure 11 A schematic block diagram of a data quality inspection apparatus according to one embodiment is shown;

[0058] Figure 12 This diagram illustrates the overall flowchart of a data quality inspection method according to one embodiment.

[0059] Figure 13 A functional block diagram of a data quality check according to one embodiment is shown. Detailed Implementation

[0060] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0061] Figure 1 This is a schematic diagram illustrating an implementation scenario of one embodiment disclosed in this specification. This implementation scenario involves data quality checking. Data quality refers to the quality of information within a dataset, and its evaluation dimensions mainly include standardization, accuracy, completeness, consistency, timeliness, and accessibility. Data consistency refers to the degree to which data does not contradict data used in other specific contexts. In this embodiment, data consistency is detected by exploring whether semantic relationships exist between different fields. (Refer to...) Figure 1 The image shows the front-end page of a corporate information display platform. Due to the variety of data storage media and the complex data processing algorithms, the following two data quality issues may exist: Issue 1: The data displayed on the platform has highly similar characteristics, resulting in content duplication and redundancy; Issue 2: Data from different modules displayed on the platform points to the same meaning but has different values, indicating data inconsistency. For example, Figure 1 The information displayed above the dotted line is "Registered Capital: 20 million RMB", while the information displayed below the dotted line is "Registered Capital: 1 million RMB". It can be understood that these two pieces of information point to the same meaning, but the values ​​are different, that is, there is a data inconsistency.

[0062] Typically, manually writing test code to correlate relevant fields and verify the aforementioned data quality issues requires complete validation across all interfaces, and human intervention often leads to omissions. This specification's embodiments propose corresponding solutions to more efficiently detect these data quality issues, thereby improving data management efficiency and reducing errors and uncertainties caused by manual intervention.

[0063] It should be noted that, Figure 1 Only one typical scenario corresponding to the aforementioned problem 2 is shown in the figure. The scenario corresponding to the aforementioned problem 1 is relatively easy to imagine, so it is not shown in the figure.

[0064] In addition, data quality checks can be performed on online data or on offline data.

[0065] Figure 2 This diagram illustrates a data quality inspection method according to one embodiment, which can be based on... Figure 1 The implementation scenario is shown. For example... Figure 2As shown, the data quality inspection method in this embodiment includes the following steps: Step 21, obtaining a data set of multiple sets of field descriptions and corresponding field values; Step 22, extracting the field descriptions from the data set to obtain a field description list, and extracting the field values ​​from the data set to obtain a field value list; Step 23, calculating a first similarity between any two field descriptions in the field description list; Step 24, calculating a second similarity between any two field values ​​in the field value list; Step 25, based on the first similarity between any two field descriptions and the second similarity between the two field values ​​corresponding to those two field descriptions, and based on whether they meet pre-set attribution conditions, locating data quality problems. The specific execution method of each of the above steps is described below.

[0066] First, in step 21, a data set of multiple sets of field descriptions and their corresponding field values ​​is obtained. It can be understood that there is a correspondence between field descriptions and field values ​​within the same set. For example, the data information is "Registered Capital 1 million RMB," where "Registered Capital" is the field description and "1 million RMB" is the field value; these belong to a corresponding set of field descriptions and field values. Another example is the data information "Registered Location: Shanghai," where "Registered Location" is the field description and "Shanghai" is the field value; these belong to another corresponding set of field descriptions and field values.

[0067] In one example, obtaining a data set of multiple sets of field descriptions and corresponding field values ​​includes:

[0068] From the display data of a front-end page, obtain a first subset of field descriptions and corresponding field values ​​from a first uniform resource locator (URL), and obtain a second subset of field descriptions and corresponding field values ​​from a second URL; the first subset and the second subset constitute the data set.

[0069] In this example, a quality check is performed on data from different URLs on the same page to detect duplicate or inconsistent data.

[0070] In one example, the dataset is associated with the display data of the front-end page;

[0071] The data set for obtaining multiple sets of field descriptions and corresponding field values ​​includes:

[0072] By embedding tracking points in the front-end page or back-end control layer, message data with a preset structure can be obtained;

[0073] Extract several key-value pairs from the message data with a preset structure. In each key-value pair, the key is used as a field description and the value is used as the corresponding field value.

[0074] For example, by placing tracking points in the front-end page or back-end controller layer through which the data passes, we can obtain JSON message data D. Based on the mapping relationship between the front-end and back-end, we can flatten the JSON message data D to obtain n key-value pairs (K,V). n That is, (key1-v1,key2-v2,key3-v3,...), where key1 is the field description, v1 is the field value corresponding to key1, key2 is the field description, v2 is the field value corresponding to key2, key3 is the field description, and v3 is the field value corresponding to key3.

[0075] In one example, the dataset is associated with an offline database;

[0076] The data set for obtaining multiple sets of field descriptions and corresponding field values ​​includes:

[0077] Extract the field descriptions and corresponding values ​​of each field contained in the data table of the offline database.

[0078] For example, in a MySQL database, you can obtain a field description by using the column and its corresponding description. The column can also be called a field.

[0079] In the embodiments of this specification, the field value can be the original value of a field corresponding to a business object, or it can be formed by concatenating the original values ​​of the same field corresponding to multiple business objects respectively.

[0080] For example, if the business object is a company, "registered location" is the field description, and "Shanghai" is the original value for company 1, then "Shanghai" can be directly used as the field value; or, if the business object is a company, "registered location" is the field description, and "Shanghai" is the original value for company 1, and "Beijing" is the original value for company 2, then "Shanghai Beijing" can be directly used as the field value.

[0081] For fields whose values ​​are composed of concatenated original values, we can collect m different requests for aggregation. This can ensure the semantic universality of field values ​​and address the situation where some fields of some business objects have empty values, thus ensuring coverage.

[0082] Then, in step 22, the field descriptions are extracted from the data set to obtain a list of field descriptions, and the field values ​​are extracted from the data set to obtain a list of field values. It is understood that since the field descriptions and field values ​​obtained in step 21 have a corresponding relationship, when forming the list of field descriptions and the list of field values, the corresponding field descriptions and field values ​​can be placed in the same position in the list, so as to distinguish whether there is a corresponding relationship between the field descriptions and field values ​​based on their position.

[0083] For example, iterating through and extracting n key-value pairs (KV). n From the key, we get a list K of key values. n (key1,key2,key3,...), K n This is a list of field descriptions; iterate through and extract n key-value pairs (KV). n From the value in the input, we get a list of values ​​V. n (v1,v2,v3,...), V n This is the list of possible values ​​for the field.

[0084] Next, in step 23, for any two field descriptions in the field description list, the first similarity between them is calculated. It is understood that a pre-defined algorithm or a pre-trained model can be used to calculate this first similarity.

[0085] In one example, calculating the first similarity between any two field descriptions in the field description list includes:

[0086] Encode any field description in the field description list using a language model to obtain the embedding vector corresponding to that field description;

[0087] Based on the embedding vectors corresponding to any two fields, the first similarity between the two is calculated using cosine similarity.

[0088] For example, the language model described above can be a multilingual model for multiple languages. Based on the multilingual model's encoded field descriptions, the corresponding embedding vector vec is obtained. ki For the two vec ki By calculating the cosine similarity, the first similarity between the two can be obtained, where the range of the first similarity can be 0 to 1.

[0089] Among these methods, comparative learning can be used to train the model.

[0090] In step 24, for any two field values ​​in the field value list, a second similarity is calculated between them. It is understood that a pre-defined algorithm or a pre-trained model can be used to calculate this second similarity.

[0091] In the embodiments of this specification, the first similarity and the second similarity can be calculated using different methods.

[0092] In one example, calculating the second similarity between any two field values ​​from the list of field values ​​includes:

[0093] For each of the two field values, calculate the corresponding first hash value and second hash value respectively;

[0094] The second similarity is obtained based on the XOR operation result of the first hash value and the second hash value.

[0095] For example, based on the SimHash algorithm, the semantic similarity of the values ​​of two fields is calculated and normalized to obtain the second similarity between the two, where the range of the second similarity can be 0 to 1.

[0096] To address the issues of large amounts of dirty data in the scenario, data preprocessing can be performed before similarity calculation. For example, text cleaning operations can be performed on API return results to remove JSON tags, special characters, and stop words, thereby improving the accuracy and efficiency of similarity calculation.

[0097] Furthermore, the Simhash algorithm can be combined with feature engineering to extract more text features as input. For example, bag-of-words models, TF-IDF features, and n-gram features can be used to transform text data into richer feature representations.

[0098] Finally, in step 25, based on the first similarity between any two field descriptions and the second similarity between the two field values ​​corresponding to those descriptions, and whether they meet pre-defined attribution conditions, the data quality problem is located. It can be understood that the first similarity reflects whether the two field descriptions are similar, and the second similarity reflects whether the two field values ​​are similar.

[0099] In the embodiments described in this specification, a similarity threshold can be set according to specific needs and actual conditions to determine whether the descriptions of two fields are similar or whether the values ​​of two fields are similar. The similarity threshold can be flexibly adjusted according to different business requirements to meet the requirements of data quality inspection.

[0100] In one example, locating data quality issues based on whether the two conditions meet pre-defined attribution criteria includes:

[0101] If the first similarity between any two field descriptions indicates that the two field descriptions are similar, and the second similarity between the two field values ​​corresponding to the two field descriptions indicates that the two field values ​​are not similar, then it is determined that there is a data consistency problem with the field values.

[0102] Output a prompt message to remind you to check the field value retrieval logic of the relevant business object.

[0103] For example, if there are two corresponding keys and two values, and the two keys are similar but the two values ​​are not similar, it can be inferred that there may be a data consistency problem, and the focus should be on checking the value retrieval logic of the relevant business objects.

[0104] In one example, locating data quality issues based on whether the two conditions meet pre-defined attribution criteria includes:

[0105] If the first similarity between any two field descriptions indicates that the two field descriptions are not similar, and the second similarity between the two field values ​​corresponding to the two field descriptions indicates that the two field values ​​are similar, then it is determined that there is a data consistency problem with redundant fields.

[0106] Output a prompt message to suggest checking the query fields.

[0107] For example, given two corresponding keys and two values, if the two keys are dissimilar but the two values ​​are similar, it can be inferred that there may be redundant fields or links, and the architectural design of the query fields or query links should be carefully examined.

[0108] In the embodiments described in this manual, technicians locate data quality issues based on the prompts provided. To complete basic manual verification of data quality, semantic analysis of the data itself is incorporated to pre-screen interfaces more likely to have data quality problems, reducing the workload of large-scale manual data point verification.

[0109] In one example, locating data quality issues based on whether the two conditions meet pre-defined attribution criteria includes:

[0110] Based on the first similarity, a first matrix is ​​constructed; each row of the first matrix corresponds to the description of each field, and each column also corresponds to the description of each field.

[0111] Based on the second similarity, a second matrix is ​​constructed; each row of the second matrix corresponds to the value of each field, and each column also corresponds to the value of each field.

[0112] The first similarity between two field descriptions at any position in the first matrix is ​​obtained, and the second similarity between two field values ​​at the same position in the second matrix is ​​obtained, so as to obtain the corresponding first similarity and second similarity.

[0113] In this example, by constructing a first matrix and a second matrix, it is easy to find the corresponding first similarity and second similarity.

[0114] In the embodiments of this specification, a field description combination matrix can be constructed first, in which each element corresponds to two field descriptions. After calculating the first similarity between the two field descriptions, the elements of the field description combination matrix are replaced with the first similarity to obtain the aforementioned first matrix.

[0115] Figure 3 A schematic diagram of a field description combination matrix according to one embodiment is shown. (Refer to...) Figure 3 A list of key values ​​K can be used. n Construct a matrix K n×n , where matrix K n×n Each row of the matrix K corresponds to a field description, and each column also corresponds to a field description. n×n The row and column of an element constitute a field description combination. For example, the row of element (key1, key1) corresponds to field description key1, and the column of element (key1, key2) corresponds to field description key1, which together constitute a field description combination. The row of element (key1, key2) corresponds to field description key1, and the column of element (key1, key2) corresponds to field description key2, which together constitute another field description combination.

[0116] Figure 4 A schematic diagram of a first matrix according to one embodiment is shown. (Refer to...) Figure 4 For matrix K n×n For each element, calculate the first similarity between the two field descriptions corresponding to that element, and replace the elements of the field description combination matrix with the first similarity to obtain the aforementioned first matrix. In this case, “?” represents any decimal number from 0 to 1.

[0117] In the embodiments of this specification, a field value combination matrix can be constructed first, in which each element corresponds to two field values. After calculating the second similarity between the two field values, the elements of the field value combination matrix are replaced with the second similarity to obtain the aforementioned second matrix.

[0118] Figure 5 This diagram illustrates a field value combination matrix according to one embodiment. (Refer to...) Figure 5 A list of values ​​V can be used. nConstruct a matrix V n×n , where matrix V n×n Each row of the matrix V corresponds to a value in a field, and each column also corresponds to a value in a field. n×n The row and column of an element constitute a combination of field values. For example, the row of element (v1, v1) corresponds to the field value v1, and the column of element (v1, v2) corresponds to the field value v2, which together constitute a combination of field values.

[0119] Figure 6 A schematic diagram of a second matrix according to one embodiment is shown. (Refer to...) Figure 6 For matrix V n×n For each element, calculate the second similarity between the two field values ​​corresponding to that element, and replace the elements of the field description combination matrix with the second similarity to obtain the aforementioned second matrix. In this case, “?” represents any decimal number from 0 to 1.

[0120] Wherein, matrix V n×n A matrix that can be aggregated from m different requests can also be denoted as V. n×n×m .

[0121] The method provided in this specification first obtains a dataset of multiple sets of field descriptions and corresponding field values; then, it extracts the field descriptions from the dataset to obtain a list of field descriptions, and extracts the field values ​​from the dataset to obtain a list of field values; next, it calculates a first similarity between any two field descriptions in the field description list; and it calculates a second similarity between any two field values ​​in the field value list; finally, based on the first similarity between any two field descriptions and the second similarity between the two field values ​​corresponding to those two field descriptions, it locates data quality problems based on whether the two meet pre-set attribution conditions. As can be seen from the above, this specification's embodiment adds field description features to the field value features, reducing the need for manual intervention. It automatically performs feature cross-validation between field description features and field value features, and analyzes the semantics of the data itself, making it easier to screen out potential data quality problems, reducing the work of large-scale manual data entry verification, thereby improving inspection efficiency, reducing errors, and lowering uncertainty.

[0122] Below is a comparison Figure 1 The scenario shown provides a specific example of the data quality inspection process.

[0123] Assuming a platform page consists of data from only two controllers, message D is as follows:

[0124]

[0125]

[0126] From message D, we can obtain the following key-value pairs (K,V). n Actual controller - xx, registered capital - 20 million RMB, telephone - 123xxxxxxxxx, registered address - Shanghai, registered capital - 1 million RMB, establishment date - 2004-02-26, company status - active.

[0127] From key-value pairs (K,V) n Extract the key to obtain a list of key values ​​K. n (url1-actual controller, url1-registered capital, url1-telephone, url1-registered address, url2-registered capital, url2-date of establishment, url2-company status).

[0128] From key-value pairs (K,V) n Extract the value to obtain a list of value V. n (xx, 20 million RMB, 123xxxxxxxxx, Shanghai, 1 million RMB, 2004-02-26, in operation).

[0129] Construct a field description combination matrix K nxn The combination matrix V of field values nxn .

[0130] Figure 7 This diagram illustrates a combination matrix based on field descriptions for a specific example. (See reference...) Figure 7 key1 represents url1 - actual controller, key2 represents url1 - registered capital, and key3 represents url2 - registered capital. It should be understood that the diagram is for illustration only and does not show all the keys.

[0131] Figure 8 This diagram illustrates a matrix of field value combinations based on a specific example. (See reference...) Figure 8 v1 represents "xx", v2 represents "20 million RMB", and v3 represents "1 million RMB". It should be understood that the diagram is for illustration only and does not show all the values.

[0132] In this embodiment, the interface name and URL are concatenated to ensure the uniqueness of the backend output. These are then used as input, along with the special characters [CLS] and [SEP], and fed into the similarity model to obtain the interface embeddings. Subsequently, the cosine similarity of each interface embedding is calculated pairwise to obtain the first matrix. Calculating the similarity matrix returned by the API requires m calls to obtain m company results. These results are then concatenated into strings and subjected to SimHash for similarity matching to obtain the second matrix.

[0133] Figure 9 A schematic diagram of the first matrix is ​​shown based on a specific example. (Refer to...) Figure 9 The first similarity between url1-registered capital and url2-registered capital is 0.99, indicating that they are similar.

[0134] Figure 10 A schematic diagram of a second matrix is ​​shown based on a specific example. (Refer to...) Figure 10 The second similarity score of "20 million RMB" and "1 million RMB" is 0.33, indicating that the two are not similar.

[0135] It should be noted that this can be ignored. Figure 9 and Figure 10 The remaining similarities are shown in the figure for illustrative purposes only.

[0136] The embodiments in this specification, combined with matrix and Based on the premise that "corresponding locations have similar keys but dissimilar values, suggesting a potential data consistency issue," an inconsistency problem arose when the same page displaying the same semantic data was successfully found. Technical personnel can investigate the cause of the data inconsistency and begin resolving it by matching the URL and controller locations.

[0137] According to another embodiment, a data quality inspection apparatus is also provided, which is used to perform the methods provided in the embodiments of this specification. Figure 11 A schematic block diagram of a data quality inspection apparatus according to one embodiment is shown. Figure 11 As shown, the device 1100 includes:

[0138] The acquisition unit 1101 is used to acquire a data set of multiple sets of field descriptions and corresponding field values;

[0139] Extraction unit 1102 is used to extract field descriptions from the data set acquired by acquisition unit 1101 to obtain a list of field descriptions, and to extract field values ​​from the data set to obtain a list of field values;

[0140] The first calculation unit 1103 is used to calculate the first similarity between any two field descriptions in the field description list obtained by the extraction unit 1102.

[0141] The second calculation unit 1104 is used to calculate the second similarity between any two field values ​​in the field value list obtained by the extraction unit 1102.

[0142] The positioning unit 1105 is used to locate data quality problems based on whether the first similarity between any two field descriptions obtained by the first calculation unit 1103 and the second similarity between the values ​​of two fields corresponding to the arbitrary two field descriptions obtained by the second calculation unit 1104 meet the preset attribution conditions.

[0143] Optionally, as an embodiment, the acquisition unit 1101 is specifically used to acquire a first subset of field descriptions and corresponding field values ​​from a first number of fields originating from a first Uniform Resource Locator (URL) and a second subset of field descriptions and corresponding field values ​​from a second URL; the first subset and the second subset constitute the data set.

[0144] Optionally, as an example, the data set is associated with the display data of the front-end page;

[0145] The acquisition unit 1101 includes:

[0146] The event tracking subunit is used to obtain message data with a preset structure by tracking points on the front-end page or the back-end control layer.

[0147] Extraction sub-units are used to extract several key-value pairs from the message data with a preset structure obtained from the embedding sub-units. The key in each key-value pair is used as a field description, and the value is used as the corresponding field value.

[0148] Optionally, as an example, the dataset is associated with an offline database;

[0149] The acquisition unit 1101 is specifically used to extract the field descriptions and corresponding field values ​​of each field contained in the data table of the offline database.

[0150] Optionally, as an embodiment, the first computing unit 1103 includes:

[0151] The encoding subunit is used to encode any field description in the field description list using a language model to obtain the embedding vector corresponding to the field description.

[0152] The first computational subunit is used to calculate the first similarity between any two field descriptions corresponding to the embedding vectors obtained from the encoding subunit using cosine similarity.

[0153] Optionally, as an example, the field value is formed by concatenating the original values ​​of the same field corresponding to multiple business objects.

[0154] Optionally, as one embodiment, the second computing unit 1104 includes:

[0155] The second calculation subunit is used to calculate the corresponding first hash value and second hash value for the two fields respectively.

[0156] The third calculation subunit is used to obtain the second similarity between the first hash value and the second hash value obtained by the second calculation subunit.

[0157] Optionally, as an embodiment, the positioning unit 1105 includes:

[0158] The first positioning subunit is used to determine that there is a data consistency problem in the field values ​​if the first similarity between any two field descriptions shows that the two field descriptions are similar, and the second similarity between the two field values ​​corresponding to the two field descriptions shows that the two field values ​​are not similar.

[0159] The first output subunit is used to output prompt information, which prompts you to check the field value logic of the relevant business object.

[0160] Optionally, as an embodiment, the positioning unit 1105 includes:

[0161] The second positioning subunit is used to determine that there is a data consistency problem with redundant fields if the first similarity between any two field descriptions shows that the two field descriptions are not similar, and the second similarity between the two field values ​​corresponding to the two field descriptions shows that the two field values ​​are similar.

[0162] The second output sub-unit is used to output prompt information, which prompts users to check the query fields.

[0163] Optionally, as an embodiment, the positioning unit 1105 includes:

[0164] The first construction subunit is used to construct a first matrix based on the first similarity; each row of the first matrix corresponds to a field description, and each column also corresponds to a field description.

[0165] The second construction subunit is used to construct a second matrix based on the second similarity; each row of the second matrix corresponds to the value of each field, and each column also corresponds to the value of each field.

[0166] The matching subunit is used to obtain the first similarity between two field descriptions at any position in the first matrix obtained from the first construction subunit, and the second similarity between two field values ​​at the same position in the second matrix obtained from the second construction subunit, so as to obtain the corresponding first similarity and second similarity.

[0167] The apparatus provided in this embodiment first acquires a data set of multiple sets of field descriptions and corresponding field values ​​using an acquisition unit 1101. Then, an extraction unit 1102 extracts the field descriptions from the data set to obtain a list of field descriptions, and extracts the field values ​​from the data set to obtain a list of field values. Next, a first calculation unit 1103 calculates a first similarity between any two field descriptions in the field description list. A second calculation unit 1104 calculates a second similarity between any two field values ​​in the field value list. Finally, a location unit 1105 locates data quality problems based on whether the first similarity between any two field descriptions and the second similarity between the two field values ​​corresponding to those descriptions meet pre-set attribution conditions. As can be seen, this embodiment adds field description features to the field value features, reducing the need for manual intervention. It automatically performs feature cross-validation between field description features and field value features, and analyzes the semantics of the data itself, making it easier to screen out potential data quality problems. This reduces the workload of large-scale manual data entry verification, thereby improving inspection efficiency, reducing errors, and lowering uncertainty.

[0168] Figure 12 A flowchart illustrating the overall process of a data quality inspection method according to one embodiment is shown. (Refer to...) Figure 12 The data undergoes sequential processing including data parsing, algorithm detection, and result analysis to obtain the data quality check results. Specifically, the keyword matrix corresponds to the aforementioned field description combination matrix, the relevance value matrix corresponds to the aforementioned field value combination matrix, the keyword similarity matrix corresponds to the aforementioned first matrix, and the relevance value similarity matrix corresponds to the aforementioned second matrix.

[0169] Figure 13 A functional block diagram of a data quality check according to one embodiment is shown. (Refer to...) Figure 13 The data parsing module is used to execute Figure 12 The data parsing and algorithm detection module is used for execution. Figure 12 The algorithm detection and result analysis module is used to perform the detection. Figure 12The results analysis.

[0170] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.

[0171] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.

[0172] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0173] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data quality inspection method, the method comprising: Retrieve a data set containing multiple sets of field descriptions and their corresponding field values; Extract field descriptions from the dataset to obtain a list of field descriptions, and extract field values ​​from the dataset to obtain a list of field values; For any two field descriptions in the field description list, the field descriptions are encoded using a language model, and the first similarity between the two is calculated based on the encoding results; For any two field values ​​in the field value list, calculate their second similarity. Construct the first matrix based on the first similarity. Each row of the first matrix corresponds to a field description, and each column also corresponds to a field description. Based on the second similarity, a second matrix is ​​constructed; each row of the second matrix corresponds to the value of each field, and each column also corresponds to the value of each field. Obtain the first similarity between two field descriptions at any position in the first matrix, and obtain the second similarity between two field values ​​at the same position in the second matrix, so as to obtain the corresponding first similarity and second similarity. Based on the first similarity between any two field descriptions and the second similarity between the values ​​of the two fields corresponding to those two field descriptions, the data quality problem is located based on whether the two meet the pre-set attribution conditions. The process of locating data quality issues based on whether the two conditions meet pre-defined attribution criteria includes: If the first similarity between any two field descriptions indicates that the two field descriptions are similar, and the second similarity between the two field values ​​corresponding to these two field descriptions indicates that the two field values ​​are not similar, then it is determined that there is a data consistency problem with the field values; output a prompt message to prompt you to check the field value logic of the relevant business object; If the first similarity between any two field descriptions indicates that the two field descriptions are not similar, and the second similarity between the two field values ​​corresponding to the two field descriptions indicates that the two field values ​​are similar, then a data consistency problem with redundant fields is determined; a prompt message is output to prompt you to check the query fields.

2. The method as described in claim 1, wherein, The data set for obtaining multiple sets of field descriptions and corresponding field values ​​includes: From the display data of a front-end page, obtain a first subset of field descriptions and corresponding field values ​​from a first Uniform Resource Locator URL, and obtain a second subset of field descriptions and corresponding field values ​​from a second URL; the first subset and the second subset constitute the data set.

3. The method as described in claim 1, wherein, The dataset is associated with the data displayed on the front-end page; The data set for obtaining multiple sets of field descriptions and corresponding field values ​​includes: By embedding tracking points in the front-end page or back-end control layer, message data with a preset structure can be obtained; Extract several key-value pairs from the message data with a preset structure. In each key-value pair, the key is used as a field description and the value is used as the corresponding field value.

4. The method of claim 1, wherein, The dataset is associated with an offline database; The data set for obtaining multiple sets of field descriptions and corresponding field values ​​includes: Extract the field descriptions and corresponding values ​​of each field contained in the data table of the offline database.

5. The method of claim 1, wherein, The step of encoding any two field descriptions in the field description list using a language model and calculating a first similarity between them based on the encoding results includes: Encode any field description in the field description list using a language model to obtain the embedding vector corresponding to that field description. Based on the embedding vectors corresponding to any two fields, the first similarity between the two is calculated using cosine similarity.

6. The method of claim 1, wherein, The field value is formed by concatenating the original values ​​of the same field corresponding to multiple business objects.

7. The method of claim 1, wherein, The step of calculating the second similarity between any two field values ​​from the list of field values ​​includes: For each of the two field values, calculate the corresponding first hash value and second hash value respectively; The second similarity is obtained based on the XOR operation result of the first hash value and the second hash value.

8. A data quality inspection device, the device comprising: The retrieval unit is used to retrieve a data set containing multiple sets of field descriptions and their corresponding field values. The extraction unit is used to extract field descriptions from the data set acquired by the acquisition unit to obtain a list of field descriptions, and to extract field values ​​from the data set to obtain a list of field values. The first calculation unit is used to encode any two field descriptions in the field description list obtained by the extraction unit using a language model, and calculate the first similarity between the two based on the encoding result. The second calculation unit is used to calculate the second similarity between any two field values ​​in the field value list obtained by the extraction unit. The localization unit is used to locate data quality problems based on whether the first similarity between any two field descriptions obtained by the first calculation unit and the second similarity between the values ​​of two fields corresponding to the arbitrary two field descriptions obtained by the second calculation unit meet the pre-set attribution conditions. The positioning unit includes: The first construction subunit is used to construct a first matrix based on the first similarity; each row of the first matrix corresponds to a field description, and each column also corresponds to a field description. The second construction subunit is used to construct a second matrix based on the second similarity; each row of the second matrix corresponds to the value of each field, and each column also corresponds to the value of each field. The matching subunit is used to obtain the first similarity between two field descriptions at any position in the first matrix obtained from the first construction subunit, and to obtain the second similarity between two field values ​​at the same position in the second matrix obtained from the second construction subunit, so as to obtain the corresponding first similarity and second similarity. The first positioning subunit is used to determine that there is a data consistency problem in the field values ​​if the first similarity between any two field descriptions shows that the two field descriptions are similar, and the second similarity between the two field values ​​corresponding to the two field descriptions shows that the two field values ​​are not similar. The first output subunit is used to output prompt information, which prompts the checking of the field value logic of the relevant business object. The second positioning subunit is used to determine that there is a data consistency problem with redundant fields if the first similarity between any two field descriptions shows that the two field descriptions are not similar, and the second similarity between the two field values ​​corresponding to the two field descriptions shows that the two field values ​​are similar. The second output sub-unit is used to output prompt information, which prompts users to check the query fields.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-7.

10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-7.

Images