Methods and apparatus for determining data quality

Abstract

This application provides a method and apparatus for determining data quality. The method includes: receiving a query request for the null value rate corresponding to a target field name in a data table, the query request carrying an identifier of the data table and a target field name; searching for a first cardinality object corresponding to a primary key field name and a second cardinality object corresponding to the target field name in the data table based on the identifier, wherein the cardinality objects are pre-calculated based on the null value data corresponding to each field name in the data table; calculating the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object; and determining the quality of the data corresponding to the target field name based on the null value rate. This application can shorten the time for querying the null value rate.

Description

Technical Field

[0001] This application relates to the field of data evaluation technology, and in particular to a method and apparatus for determining data quality. Background Technology

[0002] With the development of society and the economy, my country has gradually entered the era of big data, making the analysis of data quality in databases increasingly important. In practice, the null value rate is often used to evaluate data quality. Specifically, the null value rate is the ratio of the number of data entries with null values ​​to the total number of data entries in a data table.

[0003] In existing technologies, when it is necessary to evaluate data quality, it is generally possible to obtain the null value rate of the data and then use the null value rate to evaluate the data quality. The method for obtaining the null value rate is as follows: first, locate the data table whose null value rate needs to be calculated in the HBase database; then, read the data table into the runtime memory; and finally, use the `count` command in the HBase database to perform statistical calculations on the data table to obtain the corresponding null value rate.

[0004] In the above process, the larger the amount of data in the HBase database, the longer the lookup time for the data table in the HBase database, which in turn leads to a longer waiting time for users to receive query results. Furthermore, after finding the data table in the HBase database, it needs to be read into the runtime memory. Since the data volume of the table is large, the reading process also takes time, reducing computational efficiency and further contributing to the excessively long waiting time for users to receive query results. Summary of the Invention

[0005] In view of this, this application provides a method and apparatus for determining data quality, which can shorten the time users wait for query results.

[0006] To achieve the above objectives, this application mainly provides the following technical solutions:

[0007] In a first aspect, this application provides a method for determining data quality, the method comprising:

[0008] Receive a query request for the null value rate corresponding to the target field name in the data table. The query request carries the identifier of the data table and the target field name.

[0009] Based on the identifier, find the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name in the data table, wherein the cardinality object is calculated in advance based on the null value data corresponding to each field name in the data table;

[0010] Calculate the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object;

[0011] The quality of the data corresponding to the target field name is determined based on the null value rate.

[0012] Secondly, this application provides a data quality determination apparatus, the apparatus comprising:

[0013] The receiving unit is used to receive a query request for the null value rate corresponding to the target field name in the data table. The query request carries the identifier of the data table and the target field name.

[0014] The lookup unit is used to look up the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name in the data table according to the identifier received by the receiving unit, wherein the cardinality object is calculated in advance based on the null value data corresponding to each field name in the data table;

[0015] The calculation unit is used to calculate the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object found by the search unit;

[0016] The determining unit is used to determine the quality of the data corresponding to the target field name based on the null value rate calculated by the calculation unit.

[0017] Thirdly, this application provides an electronic device, which includes at least one processor, at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the data quality determination method described in the first aspect.

[0018] Fourthly, this application provides a storage medium for storing a computer program, wherein the computer program, when running, controls the device where the storage medium is located to execute the data quality determination method described in the first aspect.

[0019] By employing the above technical solution, this application provides a method and apparatus for determining data quality. Upon receiving a query request for the null value rate corresponding to a target field name in a data table, the method directly determines the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name based on the data table's identifier. Based on the first and second cardinality objects, the null value rate is calculated, and then the quality of the data corresponding to the target field name is determined according to the null value rate. In this process, the cardinality objects corresponding to each field name are pre-calculated, and the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name are directly found among these cardinality objects, thus directly calculating the null value rate. This eliminates the need to search for the data table in the HBase database and load it into runtime memory, improving computational efficiency and shortening the user's query time.

[0020] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a method for determining data quality disclosed in this application;

[0023] Figure 2 This is a flowchart illustrating a method for determining data quality disclosed in this application;

[0024] Figure 3 This is a schematic diagram of the structure of a data quality determination device disclosed in this application;

[0025] Figure 4 This is a schematic diagram of the structure of a data quality determination device disclosed in this application;

[0026] Figure 5 This is a block diagram of a device disclosed in this application. Detailed Implementation

[0027] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0028] With the development of society and the economy, my country has gradually entered the era of big data, making the analysis of data quality in databases increasingly important. In practice, the null value rate is often used to evaluate data quality. Specifically, the null value rate is the ratio of the number of data entries with null values ​​to the total number of data entries in a data table.

[0029] In existing technologies, when it is necessary to evaluate data quality, it is generally possible to obtain the null value rate of the data and then use the null value rate to evaluate the data quality. The method for obtaining the null value rate is as follows: first, locate the data table whose null value rate needs to be calculated in the HBase database; then, read the data table into the runtime memory; and finally, use the `count` command in the HBase database to perform statistical calculations on the data table to obtain the corresponding null value rate.

[0030] In the above process, the larger the amount of data in the HBase database, the longer the lookup time for the data table in the HBase database, which in turn leads to a longer waiting time for users to receive query results. Furthermore, after finding the data table in the HBase database, it needs to be read into the runtime memory. Since the data volume of the table is large, the reading process also takes time, reducing computational efficiency and further contributing to the excessively long waiting time for users to receive query results.

[0031] To address the aforementioned problems, this application provides a method for determining data quality. The method can be executed by a server or other electronic devices, and is not limited thereto. The specific execution steps are as follows: Figure 1 As shown, it includes:

[0032] Step 101: Receive the null value rate query request corresponding to the target field name in the data table.

[0033] The query request carries the table identifier and target field names. The table identifier uniquely identifies the table; this identifier can be the table name or other data used to uniquely identify the table. The table includes primary key field names and non-primary key field names. The target field name is one or more of the non-primary key field names. The row / column data corresponding to the primary key field name is used as the primary key data, and each primary key field is used to uniquely identify a record in the table. For example, in Table 1, Zhang San, Li Si, and Wang Wu each identify a unique record in the table; therefore, Zhang San, Li Si, and Wang Wu are primary key data, and their corresponding field name "Name" is the primary key field name. Thus, the other fields besides "Name" are named "Gender" and "Age," and these are non-primary key field names. Note that a "-" in the cell indicates a null value.

[0034] Table 1

[0035] Name gender age Zhang San - 25 Li Si female 30 Wang Wu male -

[0036] In this specific implementation, a data table is displayed on the user terminal. When the user selects a target field name in the data table, the terminal displays a set of instructions corresponding to that target field name, including a null value rate query instruction. When the user clicks the null value rate query instruction in the instruction set, the terminal receives the null value rate query instruction corresponding to the target field name in the data table, generates a null value rate query request based on the data table identifier and the target field name, and then sends the request to the server so that the server receives the null value rate query request corresponding to the target field name in the data table. Alternatively, a null value rate search box is displayed on the terminal, where the user can enter the data table identifier and the target field name and click the query button. In this way, the terminal responds to the user's click instruction, generates a null value rate query request based on the obtained data table identifier and target field name, and sends the query request to the server so that the server receives the null value rate query request corresponding to the target field name in the data table.

[0037] Step 102: Based on the identifier, find the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name in the data table.

[0038] The cardinality object is pre-calculated based on the null values ​​corresponding to each field name in the data table. Specifically, the data corresponding to each field name in the data table is input into the hll algorithm to obtain the cardinality object for each field name. This data includes null values ​​and non-null values. Furthermore, due to different methods for calculating the null value rate, the cardinality object involved in this embodiment may include cardinality, the sequence corresponding to m buckets, accuracy, etc., or it may only include cardinality. The sequence corresponding to the m buckets in the cardinality object is used to determine the cardinality, and the accuracy is the accuracy of the calculated cardinality. For example, if the method for calculating the null value rate is based on the number of null values ​​corresponding to the target field name and the total number of data corresponding to the target field name, then the cardinality object may only include the cardinality. If the method for calculating the null value rate is based on the number of rows / columns with null values ​​and the total number of rows / columns, then the cardinality object includes the sequence corresponding to the m buckets and may also include the cardinality. Additionally, the meaning of the cardinality corresponding to different field names is different; for example, the cardinality corresponding to the primary key field name is the total number of rows / columns corresponding to the primary key field name. The cardinality corresponding to the target field name is the number of null values ​​in the data corresponding to the target field name.

[0039] In the specific implementation of this step, when the server receives a null value rate query request, it obtains the identifier of the data table and the target field name in the query request. Based on the pre-stored correspondence between the identifier of the data table and the cardinality object set, it determines the cardinality object set corresponding to the identifier of the data table in the PG library, and finds the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name in the cardinality object set.

[0040] Step 103: Calculate the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object.

[0041] In this specific implementation, the null value rate corresponding to the target field name is determined based on the pre-set null value rate calculation rules, the first cardinality object, and the second cardinality object. The null value rate calculation rules can be pre-set by technical personnel or specified by the user.

[0042] This application provides two rules for calculating the null value rate. The first rule is to calculate the ratio between the number of null data corresponding to the target field name and the total number of data corresponding to the target field name, and to determine this ratio as the null value rate. The specific process is as follows: first, determine the sum of the cardinalities included in at least one second cardinal object, and determine the product of the number of cardinalities included in the first cardinal object and the number of second cardinal objects, and then determine the ratio of the sum to the product as the null value rate corresponding to the target field name.

[0043] For example, if "Gender" and "Age" from Table 1 are set as target field names, then according to Table 1, the number of null values ​​corresponding to "Gender" is 1, and the number of null values ​​corresponding to "Age" is 1, resulting in a total of 2 null values ​​corresponding to the target field names. Meanwhile, according to Table 1, the total number of data points corresponding to the target field names is 6. Thus, the ratio of the number of null values ​​corresponding to the target field names to the total number of data points corresponding to the target field names is one-third, meaning the null value rate for the target field names is one-third.

[0044] The second method is to determine the null value rate of the target field name by the ratio of the number of rows / columns containing null values ​​to the total number of rows / columns in the primary key data. The specific process is as follows: First, determine the number of rows / columns containing null values ​​corresponding to the target field name based on the second cardinality object. Then, determine the ratio of the number of rows / columns containing null values ​​corresponding to the target field name to the cardinality in the first cardinality object, and use this ratio as the null value rate of the target field.

[0045] For example, if "Gender" and "Age" in Table 1 are set as the target field names, then according to Table 1, the rows with null values ​​are the second and fourth rows, resulting in a total of 2 rows with null values. Meanwhile, according to Table 1, the total number of rows corresponding to the primary key data is 3. Therefore, the ratio of the number of rows with null values ​​corresponding to the target field names to the total number of rows corresponding to the primary key data is two-thirds, meaning the null value rate for the target field is two-thirds.

[0046] Step 104: Determine the quality of the data corresponding to the target field name based on the null value rate.

[0047] In this specific implementation, a higher null value rate indicates better data quality for the target field name, and a lower null value rate indicates lower data quality. Thus, the quality of the data corresponding to the target field name can be determined based on its null value rate.

[0048] Upon receiving a query request for the null value rate corresponding to the target field name in the data table, the system directly determines the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name based on the table's identifier. Based on these two cardinality objects, the null value rate is calculated, and the quality of the data corresponding to the target field name is then determined. In this process, the cardinality objects corresponding to each field name are pre-calculated, and the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name are directly found within these cardinality objects. This allows for direct null value rate calculation without needing to search the HBase database and load it into memory, improving computational efficiency and reducing user query time.

[0049] Furthermore, this application embodiment provides a detailed description of the second method for calculating the null value rate in step 103, the specific steps of which are as follows: Figure 2 As shown, it includes:

[0050] Step 201: Determine the total number of rows or columns corresponding to the primary key field name based on the first cardinality object.

[0051] In the specific implementation of this step, to shorten the time it takes for users to query the null value rate, the cardinality in the first cardinality object can be directly obtained and used to determine the number of data corresponding to the primary key field name. Alternatively, the cardinality can be recalculated based on the sequences corresponding to multiple buckets in the first cardinality object, thereby obtaining the total number of rows / columns in the data table.

[0052] Step 202: Based on the second cardinality object, determine the number of rows or columns containing null values ​​corresponding to the target field name.

[0053] This step can be divided into two cases. The first case is that there is only one target field name and its corresponding second cardinality object is also a single object. In this case, when the server receives a query request, it first checks the number of target field names in the query request. When it detects that there is only one target field name, in order to shorten the time for users to query for null values, it can directly determine the number of rows / columns with null values ​​corresponding to the target field name based on the cardinality in the second cardinality object.

[0054] The second scenario involves multiple target field names, resulting in multiple second cardinality objects. In this case, when the server receives a query request, it first checks the number of target field names. If multiple target field names are detected, it first determines a third cardinality object based on the multiple second cardinality objects. Then, based on the third cardinality object, it determines the number of rows / columns containing null values.

[0055] The second cardinality object includes sequences corresponding to m buckets, where m is a positive integer.

[0056] Furthermore, this step provides a method for determining a third cardinal object based on a second cardinal object. Specifically, the method involves inputting the sequence corresponding to the m-th bucket in different second cardinal objects into a preset merging algorithm, outputting the sequence corresponding to the m-th bucket in the third cardinal object, and obtaining the third cardinal object.

[0057] Here, the sequence corresponding to the aforementioned bucket is a sequence of values ​​composed of 0s and / or 1s. The merging algorithm can be XOR calculation or other calculation methods; this is not limited here.

[0058] Meanwhile, this application also provides a method for calculating the number of rows / columns containing null values ​​based on the sequences corresponding to the m buckets in the third cardinal object. The specific steps are as follows: determine the amount of data corresponding to the m bucket based on the sequence corresponding to the m bucket in the third cardinal object; determine the number of rows / columns containing null values ​​based on the sum of the amounts of data corresponding to the m buckets and the first preset formula.

[0059] The first preset formula is E = m 2 *a, E represents the number of rows / columns containing null values, m represents the number of buckets, and a represents the sum of the data volume corresponding to the m buckets.

[0060] In the above process, the specific steps for calculating the data volume corresponding to the m-th bucket are as follows: determine the position of the first preset value in the sequence corresponding to the m-th bucket; determine the data volume corresponding to the m-th bucket based on the position and the second preset formula; wherein, the second preset formula is C m =2 M C m Let M be the data volume corresponding to the m-th bucket, where M is the aforementioned position.

[0061] Specifically, determine the position of the first 1 in the sequence corresponding to the m-th bucket, and substitute this position into the second preset formula to obtain the data volume corresponding to the m-th bucket.

[0062] Step 203: The ratio of the number of rows or columns containing null values ​​to the total number of rows or columns is determined as the null value rate corresponding to the target field name.

[0063] In this specific implementation, the ratio of the number of rows / columns containing null values ​​to the total number of rows / columns is calculated. For example, if the target field name corresponds to row data, the ratio of the number of rows containing null values ​​to the total number of rows is calculated. If the target field name corresponds to column data, the ratio of the number of columns containing null values ​​to the total number of columns is calculated. The calculated ratio is determined as the null value rate corresponding to the target field name.

[0064] In this embodiment, it is necessary to pre-calculate the cardinality objects corresponding to each field name in each data table, and store each field name and its corresponding cardinality object in the PG library to obtain the cardinality object set corresponding to each data table. Thus, when a null value rate query request is received, the cardinality object for which the null value rate is to be calculated can be directly found in the cardinality object set. Therefore, in order to pre-calculate the cardinality objects corresponding to each field name in each data table, this embodiment provides a method for determining the cardinality object set corresponding to each data table, the specific steps of which are as follows:

[0065] Upon receiving a preset request from another device, the system determines the data corresponding to the first field name in the data table it carries, inputs the data corresponding to the first field name into the preset hll algorithm, obtains the cardinality object corresponding to the first field name, and stores the first field name and the corresponding cardinality object accordingly.

[0066] The preset request is used to instruct the data table to be placed into the HBase database. The data corresponding to the first field name includes null and non-null values.

[0067] In this specific implementation, upon receiving a preset request from another device, the preset request is parsed to obtain the data table carried in the preset request. The field names for which the null rate is to be calculated are determined within this data table. These field names can be all field names in the data table or only some of them; there is no limitation here. Thus, after determining the first field names, these field names can be used as the first field name. In the data table, the data corresponding to the first field name is determined, and the data corresponding to each first field name is input into a preset hllHyperLogLog algorithm to obtain the cardinality object corresponding to the first field name. The first field name and its corresponding cardinality object are then stored in the cardinality object set corresponding to the data table.

[0068] The above steps use the HLL algorithm to estimate the cardinality to obtain the cardinality object. The HLL algorithm is the hyperLogLog algorithm. Loglog is a method for estimating cardinality. It sacrifices the accuracy of cardinality statistics in exchange for very low memory consumption. In practice, 100 million data points only require 1K of memory. Compared with other cardinality counting methods such as B-number and bitmap, it greatly saves storage space. At the same time, it is suitable for cardinality counting scenarios with large data volumes that do not require precise counting or absolute accuracy. It saves storage space while keeping the error within a certain range.

[0069] Specifically, the basic idea of ​​the HLL algorithm is to estimate the overall cardinality by using the maximum value of the position of the first 1 in the bit string of the numbers in the set. However, this estimation method has a large error. To improve the error situation, HLL introduces the concept of bucket averaging. For example, the statistical data is divided into m buckets, and each bucket calculates its own Kmax and obtains its own cardinality estimate. Finally, these are averaged to obtain the overall cardinality estimate. The bucket array is used to reduce the error caused by randomness and improve the accuracy of the estimation. In LLC, the geometric mean is used to estimate the overall cardinality, but the error is large when the amount of statistical data is small. Therefore, the HLL algorithm improves on the LLC by using the harmonic mean. The advantage of the harmonic mean is that it can filter out unhealthy statistical values.

[0070] This step provides a method for calculating the number of buckets, specifically: determining the amount of data to be calculated using the HLL algorithm; when this amount of data is less than a preset value, according to... Calculate the number of buckets, where m is the number of buckets, x is the data volume, and e is a constant. When the data volume exceeds a preset value, set the number of buckets to 16834.

[0071] It should be noted that in the above steps, the preset request is used to instruct the data table to be put into the HBase database. Therefore, after using the data table, the data table carried by the preset request needs to be put into the HBase database.

[0072] Furthermore, if some field names in the data table are used as the first field name, the first field name can be determined based on the query frequency corresponding to each field name. The specific method includes: besides the primary key field name, determining the second field name in the data table whose corresponding query frequency is greater than a preset value, and preprocessing the data corresponding to the second field name to obtain the preprocessed data corresponding to the second field name; determining the primary key field name and the second field name as the first field name, and determining the data corresponding to the primary key field name and the preprocessed data corresponding to the second field name as the data corresponding to the first field name.

[0073] In the specific implementation of this step, the preset request also includes the query frequency corresponding to each field name. These query frequencies can be set by technical personnel based on the query performance of each field name. Thus, for all field names except the primary key field name, the query frequency corresponding to each other field name is compared with a preset value to determine the second field name whose query frequency is greater than the preset value. The data corresponding to each second field name is preprocessed to obtain preprocessed data corresponding to the second field name. In this way, the primary key field name and the second field name are determined as the first field name, and the data corresponding to the primary key field name and the preprocessed data corresponding to the second field name are determined as the data corresponding to the first field name.

[0074] The preprocessing steps involved in the above steps include discarding data that does not meet preset conditions. For example, the data table includes employee IDs for multiple employees, with the numerical range of the employee IDs being [1, 100]. If an employee's employee ID is detected to be outside this numerical range, it is considered incorrect, and the employee ID is discarded.

[0075] The preprocessing steps described above also include: converting data corresponding to the same field name into data of the same data type, and discarding data that cannot be converted. In this step, converting data corresponding to the same field name into data of the same data type includes two methods. The first method is to determine the target data type corresponding to a field name based on a pre-stored correspondence between field names and data types, and then convert all data corresponding to that field name into data of that target data type. For example, if the data type corresponding to field names such as "employment date" and "birthday" is pre-set to be date type, then when field names such as "employment date" and "birthday" are detected, the data corresponding to these field names will be converted into date type data. Similarly, if the data type corresponding to field names such as "employee ID," "student ID," and "item number" is pre-set to be long type, then when field names such as "employee ID," "student ID," and "item number" are detected, the data corresponding to these field names will be converted into data corresponding to the long type.

[0076] The second method involves counting the data types of each data corresponding to the same field name, determining the number of data corresponding to each data type, taking the data type with the largest corresponding number as the target data type, and then converting the data corresponding to other data types of the same field name to the target data type, thereby converting the data corresponding to the same field name into data of the same data type.

[0077] In this embodiment, the source data of the data table can also be divided into two types: incremental and full. The distinction between incremental and full data is configured by the data collection rules in data governance. In this embodiment, incremental data arrives in batches, requiring batch processing, while full data arrives as a whole and is processed directly. In the HBase database, incremental data is partitioned and stored according to the incremental field. For example, for data incrementally processed by time field, an incremental update is performed monthly. The actual physical table in HBase is a new physical table created each month, maintaining a consistent table model. This table model refers to the table's metadata, including the number of column families in HBase, the number of columns in each column family, the number of fields in each column, and the format in which the fields are loaded, etc.

[0078] For data tables sourced from the full dataset, the cardinality object set can be directly created using the method described above. However, for data tables sourced incrementally, although the server receives multiple data tables, these tables are essentially formed by dividing a batch of data into multiple batches, meaning multiple data tables share common field names. Therefore, when creating the cardinality object set corresponding to the same batch of data, this cardinality object set stores multiple cardinality objects corresponding to each data table included in that batch of data, as well as the corresponding time point of the receiving data table. In other words, the cardinality object set stores multiple cardinality objects corresponding to the same field name, with each cardinality object corresponding to a different time point. Thus, when a user queries the null value rate corresponding to a target field name in a batch of data, the cardinality object set corresponding to that batch of data can be determined first. Within this cardinality object set, the cardinality objects within their lifecycle can be identified, and then, based on the cardinality objects within their lifecycle, the null value rate corresponding to the target field name can be determined.

[0079] In the above situation, when calculating the null value rate, multiple cardinality objects corresponding to the same field name will be identified, namely, multiple cardinality objects corresponding to the primary key field name and multiple cardinality objects corresponding to the target field name. If the first method in step 103 is used to calculate the null value rate, the cardinality objects corresponding to the same field name can be added together to merge the cardinality objects corresponding to each field name. If the second method in step 103 is used to calculate the null value rate, the cardinality objects corresponding to the primary key field name can be directly added together to determine the total number of rows / columns corresponding to the primary key field name. For the multiple cardinality objects corresponding to the target field name, the number of rows / columns with null values ​​at that time point can be determined first based on the cardinality objects at the same time point, and then the number of rows / columns with null values ​​at each time point can be added together to obtain the total number of rows / columns with null values.

[0080] Furthermore, as a response to the above Figure 1-2The implementation of the method embodiment shown in this application provides a data quality determination device that can shorten the time for users to query the null value rate. The embodiment of this device corresponds to the foregoing method embodiments. For ease of reading, this embodiment will not repeat the details of the foregoing method embodiments one by one, but it should be clear that the device in this embodiment can implement all the contents of the foregoing method embodiments. Specifically, as shown... Figure 3 As shown, the device includes:

[0081] The receiving unit 301 is used to receive a query request for the null value rate corresponding to the target field name in the data table. The query request carries the identifier of the data table and the target field name.

[0082] The lookup unit 302 is used to look up the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name in the data table according to the identifier received by the receiving unit 301. The cardinality object is calculated in advance based on the null value data corresponding to each field name in the target data table.

[0083] The calculation unit 303 is used to calculate the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object found by the search unit 302.

[0084] The determining unit 304 is used to determine the quality of the data corresponding to the target field name based on the null value rate calculated by the calculation unit 303.

[0085] Furthermore, such as Figure 4 As shown, when the second cardinal object is one, the calculation unit 303 includes:

[0086] The first determining module 3031 is used to determine the total number of rows / columns corresponding to the primary key field name based on the first cardinality object;

[0087] The second determining module 3032 is used to determine the number of rows / columns containing null values ​​corresponding to the target field name based on the second cardinality object;

[0088] The third determining module 3033 is used to determine the null value rate corresponding to the target field name by the ratio of the number of rows / columns with null values ​​determined by the second determining module 3032 to the total number of rows / columns determined by the first determining module 3031.

[0089] Furthermore, such as Figure 4 As shown, when there are multiple second cardinal objects, the second determining module 3032 is further configured to:

[0090] Determine the third cardinal object based on multiple second cardinal objects;

[0091] Based on the third cardinality object, determine the number of rows / columns containing null values ​​corresponding to the target field name.

[0092] Furthermore, such as Figure 4 As shown, the second cardinality object includes sequences corresponding to m buckets, where m is a positive integer;

[0093] The second determining module 3032 is further configured to input the sequence corresponding to the m-th bucket in different second cardinal objects into a preset merging algorithm, output the sequence corresponding to the m-th bucket in the third cardinal object, and obtain the third cardinal object.

[0094] Furthermore, such as Figure 4 As shown, the second determining module 3032 is further configured to determine the position of the first preset value appearing in the sequence corresponding to the m-th bucket; and to determine the amount of data corresponding to the m-th bucket based on the position and the second preset formula; wherein, the second preset formula is C m =2 M C m Let M be the data volume corresponding to the m-th bucket, and M be the position; based on the sum of the data volumes corresponding to the m buckets and the first preset formula, determine the number of rows / columns containing null values; wherein, the first preset formula is E = m 2 *a, E represents the number of rows / columns containing null values, m represents the number of buckets, and a represents the sum of the data volume corresponding to the m buckets.

[0095] Furthermore, such as Figure 4 As shown, the device also includes a storage unit 305;

[0096] The storage unit is used to determine the data corresponding to the first field name in the data table after receiving a preset request from another device, input the data corresponding to the first field name into a preset HLL algorithm to obtain the cardinality object corresponding to the first field name, and store the first field name and the corresponding cardinality object accordingly. The preset request is used to instruct the data table to be put into the HBase database. The data corresponding to the first field name includes null value data and non-null value data.

[0097] Furthermore, the device also includes a preprocessing unit 306;

[0098] The preprocessing unit 306 is used to determine, among the field names other than the primary key field name, the second field name whose corresponding query frequency is greater than a preset value, and to preprocess the data corresponding to the second field name to obtain the preprocessed data corresponding to the second field name; to determine the primary key field name and the second field name as the first field name, and to determine the data corresponding to the primary key field name and the data corresponding to the second field name as the data corresponding to the first field name.

[0099] Furthermore, embodiments of this application also provide an electronic device, which includes at least one processor, and at least one memory and bus connected to the processor; wherein the processor and memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the above-mentioned... Figure 1-2 The method for determining data quality described herein.

[0100] Furthermore, embodiments of this application also provide a storage medium for storing a computer program, wherein the computer program, when running, controls the device where the storage medium is located to execute the above-described... Figure 1-2 The method for determining data quality described herein.

[0101] Figure 5 This is a block diagram illustrating a device 50 according to an embodiment of this application. The device 30 includes at least one processor 501, at least one memory 502 connected to the processor 501, and a bus 503; wherein the processor 501 and the memory 502 communicate with each other via the bus 503. The processor 501 is used to call program instructions in the memory 502 to execute the aforementioned data sharing method. The device described herein can be a server (e.g., a local server or a cloud server), a smartphone, tablet computer, PDA, portable computer, or a fixed terminal such as a desktop computer.

[0102] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0103] It is understood that the relevant features in the above methods and apparatus can be referenced interchangeably. Furthermore, the terms "first," "second," etc., in the above embodiments are used to distinguish between embodiments and do not represent the superiority or inferiority of any particular embodiment.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0105] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0106] In addition, the memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0107] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0112] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0114] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining data quality, characterized in that, The method includes: Receive a query request for the null value rate corresponding to the target field name in the data table. The query request carries the identifier of the data table and the target field name. The first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name in the data table are found according to the identifier. The cardinality object is calculated in advance based on the null value data corresponding to each field name in the data table. The second cardinality object includes a sequence corresponding to m buckets, where m is a positive integer. Calculate the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object; Based on the null value rate, determine the quality of the data corresponding to the target field name; When the second cardinality object is one, the step of calculating the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object includes: Based on the first cardinality object, determine the total number of rows / columns corresponding to the primary key field name; Based on the second cardinality object, determine the number of rows / columns containing null values ​​corresponding to the target field name; The ratio of the number of rows / columns containing null values ​​to the total number of rows / columns is determined as the null value rate corresponding to the target field name; When there are multiple second cardinality objects, determining the number of rows / columns containing null values ​​corresponding to the target field name based on the second cardinality objects includes: Input the sequence corresponding to the m-th bucket in different second cardinal objects into the preset merging algorithm, and output the sequence corresponding to the m-th bucket in the third cardinal object to obtain the third cardinal object; Based on the third cardinality object, determine the number of rows / columns containing null values ​​corresponding to the target field name.

2. The method according to claim 1, characterized in that, The step of determining the number of rows / columns containing null values ​​corresponding to the target field name based on the third cardinality object includes: Determine the position of the first preset value in the sequence corresponding to the m-th bucket; Based on the location and the second preset formula, the amount of data corresponding to the m-th bucket is determined, where the second preset formula is C. m =2 M C m Let M be the data volume corresponding to the m-th bucket, and M be the position. Based on the sum of the data volume corresponding to the m buckets and the first preset formula, determine the number of rows / columns containing null values; Wherein, the first preset formula is E=m 2 *a, E is the number of rows / columns containing null values, m is the number of buckets, and a is the sum of the data volume corresponding to the m buckets.

3. The method according to claim 1, characterized in that, The method further includes: After receiving a preset request from another device, the system determines the data corresponding to the first field name in the data table it carries, inputs the data corresponding to the first field name into a preset HLL algorithm to obtain the cardinality object corresponding to the first field name, and stores the first field name and the corresponding cardinality object accordingly. The preset request is used to instruct the system to put the data table it carries into the HBase database. The data corresponding to the first field name includes null value data and non-null value data.

4. The method according to claim 3, characterized in that, Before determining the data corresponding to the first field name in the carried data table, the method further includes: Among the field names other than the primary key field name, determine the second field name whose corresponding query frequency is greater than a preset value, and preprocess the data corresponding to the second field name to obtain the preprocessed data corresponding to the second field name; The primary key field name and the second field name are determined as the first field name, and the data corresponding to the primary key field name and the data corresponding to the second field name are determined as the data corresponding to the first field name.

5. A device for determining data quality, characterized in that, The device includes: The receiving unit is used to receive a query request for the null value rate corresponding to the target field name in the data table. The query request carries the identifier of the data table and the target field name. The lookup unit is used to look up the first cardinality object corresponding to the primary key field name and the second cardinality object corresponding to the target field name in the data table according to the identifier received by the receiving unit. The cardinality object is calculated in advance based on the null value data corresponding to each field name in the data table. The second cardinality object includes a sequence corresponding to m buckets, where m is a positive integer. The calculation unit is used to calculate the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object; When the second cardinality object is one, the step of calculating the null value rate corresponding to the target field name based on the first cardinality object and the second cardinality object includes: Based on the first cardinality object, determine the total number of rows / columns corresponding to the primary key field name; Based on the second cardinality object, determine the number of rows / columns containing null values ​​corresponding to the target field name; The ratio of the number of rows / columns containing null values ​​to the total number of rows / columns is determined as the null value rate corresponding to the target field name; When there are multiple second cardinality objects, determining the number of rows / columns containing null values ​​corresponding to the target field name based on the second cardinality objects includes: Input the sequence corresponding to the m-th bucket in different second cardinal objects into the preset merging algorithm, and output the sequence corresponding to the m-th bucket in the third cardinal object to obtain the third cardinal object; Based on the third cardinality object, determine the number of rows / columns containing null values ​​corresponding to the target field name; The determining unit is used to determine the quality of the data corresponding to the target field name based on the null value rate calculated by the calculation unit.

6. An electronic device, the electronic device comprising at least one processor, and at least one memory and bus connected to the processor; in, The processor and memory communicate with each other via a bus; The processor is used to invoke program instructions in memory to execute the data quality determination method according to any one of claims 1-4.

7. A storage medium, characterized in that, The storage medium is used to store a computer program, wherein the computer program, when running, controls the device where the storage medium is located to execute the data quality determination method according to any one of claims 1-4.

Images