A data quality detection method and apparatus

Abstract

Embodiments of the present application provide a data quality detection method and device, comprising: vectorizing each piece of data in a to-be-detected data set into a vector to obtain a to-be-detected vector set corresponding to the to-be-detected data set; using a vector retrieval engine to perform distance retrieval on the to-be-detected vector set to obtain a distance value corresponding to each vector in the to-be-detected vector set; and determining data quality of the to-be-detected data set according to distance values corresponding to all vectors in the to-be-detected vector set and a preset distance threshold.

Description

Technical Field

[0001] This invention relates to the field of data quality analysis, and more particularly to a data quality detection method and apparatus. Background Technology

[0002] Data quality plays a crucial role in guiding business operations. Poor data quality can distort the various metrics calculated from big data analytics, failing to reflect the true state of the business. For example, missing data in fields related to success rates can lower the calculated success rate, leading to incorrect business decisions. For internet companies, with their numerous systems constantly generating data, the ability to monitor data quality and promptly detect changes is invaluable. Currently, common data quality monitoring systems like Griffin require manual configuration of rules, such as SQL statement fragments or regular expressions. Configuring rules for each field is time-consuming, labor-intensive, and prone to errors.

[0003] In the process of developing this invention, the applicant discovered at least the following problems in the prior art:

[0004] Existing data quality inspection systems require manual configuration of inspection rules, which is time-consuming, labor-intensive, and prone to errors. Summary of the Invention

[0005] This invention provides a data quality detection method and apparatus, which is also a data quality detection method and apparatus based on vector retrieval. It solves the problem that existing data quality detection systems require manual configuration of detection rules, which is time-consuming, labor-intensive, and prone to errors.

[0006] To achieve the above objectives, in one aspect, embodiments of the present invention provide a data quality detection method, comprising:

[0007] Each data point in the dataset to be detected is vectorized into a vector, resulting in the set of vectors to be detected corresponding to the dataset to be detected.

[0008] A vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected.

[0009] The data quality of the dataset to be detected is determined based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold.

[0010] Furthermore, before vectorizing each data point in the dataset to be detected into a vector to obtain the set of vectors to be detected corresponding to the dataset, the method further includes:

[0011] The dataset to be tested is formed by periodically extracting data from the data source at a preset time interval and according to a preset ratio.

[0012] Further, the step of vectorizing each data point in the dataset to be detected into a vector to obtain the set of vectors to be detected corresponding to the dataset includes:

[0013] For each piece of data in the dataset to be detected, each field in the current data is taken as a dimension, and the current data is converted into a multidimensional vector to obtain the multidimensional vector set corresponding to the dataset to be detected.

[0014] For each vector in the multidimensional vector set, PCA is used to reduce the dimension of the current vector to a specified dimension. All the vectors obtained after dimensionality reduction to the specified dimension constitute the set of vectors to be detected.

[0015] Further, the step of using a vector retrieval engine to perform distance retrieval on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected includes:

[0016] Determine whether this is the first time the vector retrieval engine has been used to perform distance retrieval on the set of vectors to be detected, and whether the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have changed;

[0017] If this is the first time the vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected, or if the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected change, then the vector index generated by the vector retrieval engine corresponding to the set of vectors to be detected is saved, and the distance value corresponding to each vector in the set of vectors to be detected is retrieved from the vector retrieval engine.

[0018] If this is not the first time the vector retrieval engine has been used to perform distance retrieval on the set of vectors to be detected, and the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have not changed, then the distance value corresponding to each vector in the set of vectors to be detected is retrieved from the vector retrieval engine according to the vector index.

[0019] Further, determining the data quality of the dataset to be detected based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold includes:

[0020] For each vector in the set of vectors to be detected, if the distance value corresponding to the current vector is greater than or equal to the preset distance threshold, the abnormal data counter is incremented.

[0021] The anomaly rate is determined based on the value of the anomaly data counter and the total number of vectors in the set of vectors to be detected, and the data quality of the dataset to be detected is determined based on the anomaly rate.

[0022] On the other hand, embodiments of the present invention provide a data quality detection device, comprising:

[0023] The vector set preparation unit is used to vectorize each data point in the dataset to be detected into a vector, thereby obtaining the vector set to be detected corresponding to the dataset to be detected.

[0024] The retrieval unit is used to perform distance retrieval using a vector retrieval engine on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected.

[0025] The data quality assessment unit is used to determine the data quality of the dataset to be detected based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold.

[0026] Furthermore, the device also includes:

[0027] The data acquisition unit is used to periodically extract data from the data source at a preset time interval and according to a preset ratio to form the dataset to be detected.

[0028] Furthermore, the vector set preparation unit includes:

[0029] The vector generation module is used to take each field in the current data as a dimension for each data in the dataset to be detected, and convert the current data into a multi-dimensional vector to obtain the multi-dimensional vector set corresponding to the dataset to be detected.

[0030] The vector dimensionality reduction module is used to reduce the dimensionality of each vector in the multidimensional vector set by PCA to obtain a vector of a specified dimension corresponding to the current vector, and the obtained vectors of the specified dimension constitute the set of vectors to be detected.

[0031] Furthermore, the retrieval unit includes:

[0032] The initial detection and recognition module is used to determine whether it is the first time that the vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected, and whether the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have changed.

[0033] The vector set retrieval module is used to save the vector index generated by the vector retrieval engine corresponding to the vector set to be detected if the vector retrieval engine is used for distance retrieval for the first time on the vector set to be detected, or if the number of fields or field attributes of the dataset to be detected corresponding to the vector set to be detected changes, and retrieve the distance value corresponding to each vector in the vector set to be detected from the vector retrieval engine.

[0034] The vector index retrieval module is used to retrieve the distance value corresponding to each vector in the set of vectors to be detected from the vector retrieval engine according to the vector index, provided that the distance retrieval is not performed for the first time using the vector retrieval engine on the set of vectors to be detected and the number or field attributes of the fields in the dataset to be detected corresponding to the set of vectors to be detected have not changed.

[0035] Furthermore, the data quality assessment unit includes:

[0036] The abnormal vector statistics module is used to increment the abnormal data counter for each vector in the set of vectors to be detected if the distance value corresponding to the current vector is greater than or equal to the preset distance threshold.

[0037] The data quality assessment module is used to determine the anomaly rate based on the value of the abnormal data counter and the total number of vectors in the vector set to be detected, and to determine the data quality of the dataset to be detected based on the anomaly rate.

[0038] The above technical solution has the following beneficial effects: By vectorizing the dataset to be detected and performing vector similarity retrieval, the data quality is calculated based on the distance values ​​obtained from the retrieval, which is used to evaluate the data quality of the dataset to be detected. It can automatically perform data quality detection on massive datasets without the need for manually setting various complex detection rules. It can automatically detect abnormal data. Through the vector retrieval engine, data anomalies can be quantified after sampling and retrieval. It enables intelligent, automatic, and long-term monitoring of data quality trends. It is more practical in real-world applications. Furthermore, through innovative vectorization of multi-dimensional data and PCA dimensionality reduction, it makes big data suitable for the technical specifications of vector retrieval engines. No manual setting of data quality rules is required. It is original in the field of data quality detection. It can automatically perform data quality detection without the need for manually setting data quality detection rules. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a data quality detection method according to one embodiment of the present invention;

[0041] Figure 2 This is an architectural diagram of a data quality detection device according to one embodiment of the present invention;

[0042] Figure 3 This is another flowchart of a data quality detection method according to one embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Vector retrieval: Finding a vector that approximates a target vector within a large dataset is called vector retrieval. Vector retrieval is widely used in computer vision, such as in face recognition and object recognition.

[0045] Data quality refers to the degree to which data, within a business environment, meets the intended use by data consumers and satisfies the specific needs of the business scenario. Typically, various reasons can lead to issues with data integrity or consistency, resulting in a decline in data quality (e.g., null values ​​or outliers in certain fields).

[0046] Faiss (or FAISS), short for Facebook AI Similarity Search, provides an efficient and reliable retrieval method for massive amounts of data in high-dimensional space.

[0047] On the one hand, such as Figure 1 As shown, this embodiment of the invention provides a data quality detection method, including:

[0048] Step S1: Vectorize each data point in the dataset to be detected into a vector to obtain the set of vectors to be detected corresponding to the dataset to be detected;

[0049] Step S2: Use a vector retrieval engine to perform distance retrieval on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected;

[0050] Step S3: Determine the data quality of the dataset to be detected based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold.

[0051] In some embodiments, the vector retrieval engine may specifically be the FAISS vector retrieval engine.

[0052] Furthermore, before vectorizing each data point in the dataset to be detected into a vector to obtain the set of vectors to be detected corresponding to the dataset, the method further includes:

[0053] The dataset to be tested is formed by periodically extracting data from the data source at a preset time interval and according to a preset ratio.

[0054] In some embodiments, random sampling is achieved by randomly drawing data from a data source, and real-time monitoring is performed by periodically drawing data at preset time intervals. This avoids calculating the entire data source each time, improving the efficiency of monitoring calculations.

[0055] Further, the step of vectorizing each data point in the dataset to be detected into a vector to obtain the set of vectors to be detected corresponding to the dataset includes:

[0056] For each piece of data in the dataset to be detected, each field in the current data is taken as a dimension, and the current data is converted into a multidimensional vector to obtain the multidimensional vector set corresponding to the dataset to be detected.

[0057] For each vector in the multidimensional vector set, PCA is used to reduce the dimension of the current vector to a specified dimension. All the vectors obtained after dimensionality reduction to the specified dimension constitute the set of vectors to be detected.

[0058] In some implementations, each field is treated as a dimension, and the entire data is converted into a multidimensional vector to facilitate dimensionality reduction. During dimensionality reduction, the importance of each dimension can be more accurately determined based on the specific application scenario.

[0059] Further, the step of using a vector retrieval engine to perform distance retrieval on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected includes:

[0060] Determine whether this is the first time the vector retrieval engine has been used to perform distance retrieval on the set of vectors to be detected, and whether the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have changed;

[0061] If this is the first time the vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected, or if the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected change, then the vector index generated by the vector retrieval engine corresponding to the set of vectors to be detected is saved, and the distance value corresponding to each vector in the set of vectors to be detected is retrieved from the vector retrieval engine.

[0062] If this is not the first time the vector retrieval engine has been used to perform distance retrieval on the set of vectors to be detected, and the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have not changed, then the distance value corresponding to each vector in the set of vectors to be detected is retrieved from the vector retrieval engine according to the vector index.

[0063] In some embodiments, false alarms may occur when the number of fields or field attributes change. Therefore, the vector index can be regenerated and saved only when the number of fields or field attributes change. If the number of fields or field attributes do not change, the previously saved vector index can be used directly for retrieval, thereby improving retrieval efficiency.

[0064] Further, determining the data quality of the dataset to be detected based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold includes:

[0065] For each vector in the set of vectors to be detected, if the distance value corresponding to the current vector is greater than or equal to the preset distance threshold, the abnormal data counter is incremented.

[0066] The anomaly rate is determined based on the value of the anomaly data counter and the total number of vectors in the set of vectors to be detected, and the data quality of the dataset to be detected is determined based on the anomaly rate.

[0067] For example, the ratio of the abnormal data counter value to the total number of vectors in the vector set to be detected can be used as the abnormality rate, and the difference between a preset benchmark value (e.g., the benchmark value is set to 1) and the abnormality rate can be used as the data quality of the dataset to be detected.

[0068] In some embodiments, abnormal data is detected, and the data quality is obtained by counting the abnormal data. A clear parameter value of data quality is provided for data evaluation and system maintenance reference. Since abnormal data is detected, the abnormal data can be clearly located and further analysis and processing can be carried out on the abnormal data.

[0069] On the other hand, such as Figure 2As shown, an embodiment of the present invention provides a data quality detection device, comprising:

[0070] The vector set preparation unit 200 is used to vectorize each data in the dataset to be detected into a vector, so as to obtain the set of vectors to be detected corresponding to the dataset to be detected.

[0071] The retrieval unit 201 is used to perform distance retrieval using a vector retrieval engine on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected;

[0072] The data quality assessment unit 202 is used to determine the data quality of the dataset to be detected based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold.

[0073] Furthermore, the device also includes:

[0074] The data acquisition unit is used to periodically extract data from the data source at a preset time interval and according to a preset ratio to form the dataset to be detected.

[0075] Further, the vector set preparation unit 200 includes:

[0076] The vector generation module is used to take each field in the current data as a dimension for each data in the dataset to be detected, and convert the current data into a multi-dimensional vector to obtain the multi-dimensional vector set corresponding to the dataset to be detected.

[0077] The vector dimensionality reduction module is used to reduce the dimensionality of each vector in the multidimensional vector set by PCA to obtain a vector of a specified dimension corresponding to the current vector, and the obtained vectors of the specified dimension constitute the set of vectors to be detected.

[0078] Further, the retrieval unit 201 includes:

[0079] The initial detection and recognition module is used to determine whether it is the first time that the vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected, and whether the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have changed.

[0080] The vector set retrieval module is used to save the vector index generated by the vector retrieval engine corresponding to the vector set to be detected if the vector retrieval engine is used for distance retrieval for the first time on the vector set to be detected, or if the number of fields or field attributes of the dataset to be detected corresponding to the vector set to be detected changes, and retrieve the distance value corresponding to each vector in the vector set to be detected from the vector retrieval engine.

[0081] The vector index retrieval module is used to retrieve the distance value corresponding to each vector in the set of vectors to be detected from the vector retrieval engine according to the vector index, provided that the distance retrieval is not performed for the first time using the vector retrieval engine on the set of vectors to be detected and the number or field attributes of the fields in the dataset to be detected corresponding to the set of vectors to be detected have not changed.

[0082] Furthermore, the data quality assessment unit 202 includes:

[0083] The abnormal vector statistics module is used to increment the abnormal data counter for each vector in the set of vectors to be detected if the distance value corresponding to the current vector is greater than or equal to the preset distance threshold.

[0084] The data quality assessment module is used to determine the anomaly rate based on the value of the abnormal data counter and the total number of vectors in the vector set to be detected, and to determine the data quality of the dataset to be detected based on the anomaly rate.

[0085] The embodiments of this invention have the following technical effects: By vectorizing the dataset to be detected and performing vector similarity retrieval, the data quality is calculated based on the distance values ​​obtained from the retrieval, which is used to evaluate the data quality of the dataset to be detected. This allows for automatic data quality detection of massive datasets without the need for manually setting various complex detection rules. It can automatically detect abnormal data. Through a vector retrieval engine, data sampling and retrieval can quantify data anomalies. It enables intelligent, automatic, and long-term monitoring of data quality trends. In practical use, it is more practical. Furthermore, through innovative vectorization of multi-dimensional data and PCA dimensionality reduction, it makes large datasets suitable for the technical specifications of vector retrieval engines. No manual setting of data quality rules is required. It is original in the field of data quality detection. It can automatically perform data quality detection without the need for manually setting data quality detection rules.

[0086] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0087] This invention provides a data quality inspection method. First, business data is vectorized. Then, PCA dimensionality reduction is used to reduce the vector dimension. Finally, the sampled data is retrieved using the FAISS vector retrieval engine. Based on the vector retrieval results, data quality inspection can be achieved quickly and accurately.

[0088] The embodiments of the present invention are described below:

[0089] The first part describes the data vectorization scheme.

[0090] The data produced by internet companies often has a very large number of fields, possibly as many as hundreds.

[0091] For example, video player logs record over 200 fields, including "uid", "mediaid", "ip", "video playback duration", and "video encoding type".

[0092] The vectorization scheme used in the technical solution of this invention first treats each field in the data as a dimension and assembles them into a multidimensional vector.

[0093] Each row of this data becomes a multidimensional vector. The resulting multidimensional vectors have a very high number of dimensions, potentially hundreds.

[0094] Commonly used vector search engines are not suitable for searching vector data with so many dimensions. Therefore, it is necessary to reduce the dimensionality of the initial multidimensional vectors.

[0095] The dimensionality reduction algorithm used in the technical solution of this invention is PCA dimensionality reduction.

[0096] PCA, also known as Principal Component Analysis, is a widely used vector dimensionality reduction method in the industry. It has the following three advantages.

[0097] (1) The principal components after dimensionality reduction by PCA are orthogonal, which can eliminate the factors that affect the original data.

[0098] (2) The calculation process of PCA dimensionality reduction is not complicated, because it mainly involves eigenvalue decomposition of a covariance matrix, so it is relatively simple and easy to implement.

[0099] (3) While retaining most of the main information, it achieves a dimensionality reduction effect.

[0100] The technical solution of this invention will reduce the initial vector to 64 dimensions through PCA to make it suitable for retrieval by the FAISS vector retrieval engine.

[0101] Part Two: Constructing Vector Indexes

[0102] The purpose of building a vector index is to facilitate vector retrieval.

[0103] To achieve fast and accurate vector retrieval, a vector index needs to be created.

[0104] There are many types of vector retrieval engines. Preferably, Facebook's open-source vector engine FAISS is used. FAISS is based on the product quantization algorithm and can quickly retrieve massive amounts of vectors.

[0105] Based on FAISS, the vector dataset is created into an index file so that approximate vectors can be retrieved quickly.

[0106] Build a vector index for the dataset. Perform random sampling without replacement on the dataset, ensuring the dataset size is between 50 million and 100 million records. Vector indexes built on datasets of this size offer excellent query performance.

[0107] For example, video player logs contain nearly 10 billion entries daily. We can perform a 1% random, non-replacement sampling on these logs. This reduces the size to under 100 million entries. Then, we can build a vector index from this under 100 million data entries.

[0108] If the data fields do not change significantly, the vector index does not need to be modified; it can be built once and used for a long time.

[0109] If the fields in the data change, such as adding or removing fields, the vector index needs to be rebuilt.

[0110] Part Three: Detecting Data Quality Through Vector Indexing.

[0111] The method for detecting data quality is to take a batch of data at regular intervals and send it to FAISS for vector retrieval.

[0112] For example, if 100 data points are retrieved per minute, they can be converted into 100 vectors using the vectorization method described above, and then used for vector retrieval.

[0113] The FAISS vector engine returns the retrieved vectors and distances.

[0114] If the distance retrieved from a vector exceeds a preset distance threshold, it is considered to be a piece of poor-quality data.

[0115] 100 records are detected per minute, which means 360,000 records are detected per hour.

[0116] Based on the results of vector retrieval, the hourly and daily data quality can be easily calculated.

[0117] For example, if 5000 data entries of poor quality are detected every hour, then the data anomaly rate for that hour is 5000 / 360000 = 1.4%, and the data quality rate is (1 - 5000 / 360000) = 98.6%.

[0118] Continuous data quality monitoring allows for easy tracking of whether data quality remains constant or deteriorates. It also enables the identification of data samples with poor quality.

[0119] It allows for timely data quality checks in response to changes in data quality.

[0120] Figure 3 This is another flowchart of an embodiment of the present invention, and the process is described as follows:

[0121] First, the dataset to be tested is periodically sampled in batches, and then vectorized into vector data.

[0122] The vector data is put into a vector retrieval engine for similarity retrieval.

[0123] If the distance to the nearest retrieved vector exceeds a preset threshold, it is considered abnormal data, and the number of abnormal data is accumulated.

[0124] If the closest retrieved vector is less than a preset threshold, it is considered normal data, and the number of normal data is accumulated.

[0125] Based on the number of normal and abnormal data, the anomaly rate can be calculated, and thus the data quality can be determined.

[0126] Before the embodiments of this invention can function normally, a data index needs to be built in advance. False alarms may also occur in these embodiments. These false alarms are often caused by changes made to data fields by the business side. After a false alarm, simply rebuilding the vector index for the new dataset will prevent further false alarms.

[0127] The embodiments of this invention have the following technical effects: By vectorizing the dataset to be detected and performing vector similarity retrieval, the data quality is calculated based on the distance values ​​obtained from the retrieval, which is used to evaluate the data quality of the dataset to be detected. This allows for automatic data quality detection of massive datasets without the need for manually setting various complex detection rules. It can automatically detect abnormal data. Through a vector retrieval engine, data sampling and retrieval can quantify data anomalies. It enables intelligent, automatic, and long-term monitoring of data quality trends. In practical use, it is more practical. Furthermore, through innovative vectorization of multi-dimensional data and PCA dimensionality reduction, it makes large datasets suitable for the technical specifications of vector retrieval engines. No manual setting of data quality rules is required. It is original in the field of data quality detection. It can automatically perform data quality detection without the need for manually setting data quality detection rules.

[0128] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0129] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0130] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0131] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," just as "including:" is interpreted as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0132] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0133] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0134] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0135] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while disks typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0136] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data quality inspection method, characterized in that, include: Data is periodically extracted from the data source at a preset time interval and at a preset ratio to form the dataset to be tested; Each data point in the dataset to be detected is vectorized into a vector, resulting in the set of vectors to be detected corresponding to the dataset to be detected. A vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected. The data quality of the dataset to be detected is determined based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold. The step of vectorizing each data point in the dataset to be detected into a vector, to obtain the set of vectors to be detected corresponding to the dataset, includes: For each piece of data in the dataset to be detected, each field in the current data is taken as a dimension, and the current data is converted into a multidimensional vector to obtain the multidimensional vector set corresponding to the dataset to be detected. The fields include multiple types. For each vector in the multidimensional vector set, PCA is used to reduce the dimension of the current vector to a specified dimension. All the vectors obtained after dimensionality reduction to a specified dimension constitute the set of vectors to be detected. The step of using a vector retrieval engine to perform distance retrieval on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected includes: Determine whether this is the first time the vector retrieval engine has been used to perform distance retrieval on the set of vectors to be detected, and whether the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have changed; If this is the first time the vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected, or if the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected change, then the vector index generated by the vector retrieval engine corresponding to the set of vectors to be detected is saved, and the distance value corresponding to each vector in the set of vectors to be detected is retrieved from the vector retrieval engine. If this is not the first time the vector retrieval engine has been used to perform distance retrieval on the set of vectors to be detected, and the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have not changed, then the distance value corresponding to each vector in the set of vectors to be detected is retrieved from the vector retrieval engine according to the vector index. The step of determining the data quality of the dataset to be detected based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold includes: For each vector in the set of vectors to be detected, if the distance value corresponding to the current vector is greater than or equal to the preset distance threshold, the abnormal data counter is incremented. The anomaly rate is determined based on the value of the anomaly data counter and the total number of vectors in the set of vectors to be detected, and the data quality of the dataset to be detected is determined based on the anomaly rate.

2. A data quality inspection device, characterized in that, include: The data acquisition unit is used to periodically extract data from the data source at a preset time interval and according to a preset ratio to form a dataset to be tested; The vector set preparation unit is used to vectorize each data point in the dataset to be detected into a vector, thereby obtaining the vector set to be detected corresponding to the dataset to be detected. The retrieval unit is used to perform distance retrieval using a vector retrieval engine on the set of vectors to be detected to obtain the distance value corresponding to each vector in the set of vectors to be detected. The data quality assessment unit is used to determine the data quality of the dataset to be detected based on the distance values ​​corresponding to all vectors in the set of vectors to be detected and a preset distance threshold. The vector set preparation unit includes: The vector generation module is used to convert each data point in the dataset to be detected into a multidimensional vector by taking each field in the current data as a dimension, thereby obtaining a multidimensional vector set corresponding to the dataset to be detected. The fields include various types. The vector dimensionality reduction module is used to reduce the dimensionality of each vector in the multidimensional vector set by PCA to obtain a vector of a specified dimension corresponding to the current vector, and the vector set to be detected is formed by all the vectors of the specified dimension obtained by PCA. The retrieval unit includes: The initial detection and recognition module is used to determine whether it is the first time that the vector retrieval engine is used to perform distance retrieval on the set of vectors to be detected, and whether the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have changed. The vector set retrieval module is used to save the vector index generated by the vector retrieval engine corresponding to the vector set to be detected if the vector retrieval engine is used for distance retrieval for the first time on the vector set to be detected, or if the number of fields or field attributes of the dataset to be detected corresponding to the vector set to be detected changes, and retrieve the distance value corresponding to each vector in the vector set to be detected from the vector retrieval engine. The vector index retrieval module is used to retrieve the distance value corresponding to each vector in the set of vectors to be detected from the vector retrieval engine according to the vector index if the distance retrieval engine is not used for the set of vectors to be detected for the first time and the number of fields or field attributes of the dataset to be detected corresponding to the set of vectors to be detected have not changed. The data quality assessment unit includes: The abnormal vector statistics module is used to increment the abnormal data counter for each vector in the set of vectors to be detected if the distance value corresponding to the current vector is greater than or equal to the preset distance threshold. The data quality assessment module is used to determine the anomaly rate based on the value of the abnormal data counter and the total number of vectors in the vector set to be detected, and to determine the data quality of the dataset to be detected based on the anomaly rate.

Images