A large-scale data quality anomaly detection method based on data features

By constructing a data feature-driven anomaly detection method library and a fuzzing processing mechanism, the problem of low efficiency in large-scale data quality detection is solved, and automated and large-scale detection is achieved.

CN113569006BInactive Publication Date: 2024-11-12STATE GRID CORPORATION OF CHINA +1
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202110671429.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-17
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing technologies are inefficient and limited in scope for detecting large-scale data quality anomalies, and cannot achieve automated and large-scale detection.

Method used

This paper adopts a data feature-driven approach, constructs a data anomaly detection method library, uses a dictionary type to store data features and their anomaly detection methods, and uses word vectors and cosine similarity calculations combined with a fuzzing processing mechanism to perform large-scale data quality detection.

Benefits of technology

It has enabled the automation and scaling of large-scale data quality inspection, improving inspection efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure BDA0003119430860000021
    Figure BDA0003119430860000021
  • Figure BDA0003119430860000031
    Figure BDA0003119430860000031
  • Figure FHA0000006275040000011
    Figure FHA0000006275040000011
Patent Text Reader

Abstract

The present invention discloses a large-scale data quality anomaly detection method based on data features, comprising the following steps: constructing a data anomaly detection method library, setting a corresponding detection method according to each data feature, and summarizing to form a data anomaly detection method library; matching the data features with anomaly detection methods, and detecting according to the anomaly detection methods in the matching results; traversing large-scale data features, matching and detecting each data feature. The substantial effects of the present invention include: transforming the anomaly detection mode driven by detection rules into a mode driven by data features, generating corresponding anomaly value detection methods based on the feature information of the data itself in each field, and setting a special fuzzy processing mechanism for large-scale data, realizing the scale and automation of data quality verification, and improving the efficiency of data quality problem detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data anomaly detection, and in particular to a large-scale data quality anomaly detection method based on data features. Background Art

[0002] With the development of the digital economy, all walks of life are no longer blindly pursuing the scale of data volume. In the process of data application, the requirements for data quality are getting higher and higher. Faced with massive data resources, how to discover and locate data quality problems faster, more accurately and more intelligently, and carry out corresponding governance work, is the focus and core of current enterprise-level data asset management.

[0003] For example, the invention with publication number CN108256074A discloses a verification processing method, including obtaining a model of a data warehouse to be verified, each model including multiple field information, the field information including field definition and field type; verifying the field information according to a pre-stored data dictionary, the data dictionary including multiple standard terms, each standard term including a standard definition and a standard type; if the field definition matches the standard definition and the field type does not match the standard type, then modifying the field type to be consistent with the standard type. The method verifies the model of the data warehouse according to standard terms. When the field definition matches the standard definition and the field type does not match the standard type, the field type is modified to be consistent with the standard type in a targeted manner, thereby obtaining a standard and consistent model. Traditional data quality anomaly detection is rule-driven and targets specific fields of specific tables. Business experts design a set of quality anomaly detection methods based on business specifications and empirical knowledge to carry out corresponding special governance work. The detection methods constructed in this way are specific in detection objects and uses and are not very universal. When large-scale data quality anomaly detection is required, the efficiency is low, the scope is limited, and it needs to be manually specified one by one, and large-scale data quality anomaly detection cannot be achieved. Summary of the invention

[0004] In response to the above problems, the present invention provides a large-scale data quality anomaly detection method based on data features, which transforms the anomaly detection mode driven by detection rules into a mode driven by data features, generates a corresponding outlier detection method based on the characteristic information of the data itself in each field, realizes the scale and automation of data quality verification, expands the scope of data quality detection, and improves the efficiency of data quality problem detection.

[0005] The following is the technical solution of the present invention.

[0006] A large-scale data quality anomaly detection method based on data features comprises the following steps:

[0007] Build a data anomaly detection method library, set the corresponding detection method according to each data feature, and summarize it to form a data anomaly detection method library; match the data features with anomaly detection methods, and perform detection based on the anomaly detection methods in the matching results; traverse large-scale data features to match and detect each data feature.

[0008] The method library of the present invention is set up to design corresponding anomaly detection methods for different data features from the perspectives of statistics, common sense, natural laws, professional general knowledge, etc. For example, the data value feature design reports anomalies when extreme values ​​appear in the field value, and the date feature reports anomalies for field contents that do not conform to the date format. The setting of the method library is specifically determined according to actual usage requirements, and targeted detection is performed after matching.

[0009] Preferably, the data anomaly detection method library is stored in a dictionary type, a tuple consisting of a data feature name and its feature parameters is used as the dictionary key, and anomaly detection methods corresponding to the data features are used as dictionary values. Python's dictionary type is a key-value pair, and Python's dictionary type is used to store data features and their anomaly detection methods. The dictionary key stores a tuple consisting of a data feature name and its feature parameters, and the dictionary value stores anomaly detection methods corresponding to the data feature, wherein the threshold of each anomaly detection method is given by the feature parameters. By storing in a dictionary manner, keys and values ​​can be clearly divided, which is conducive to subsequent matching.

[0010] Preferably, the matching includes the following process: embedding the names of the data features to be processed and the keys in the anomaly detection method library into word vectors obtained by NLP, calculating the cosine similarity between the word vectors, and the keys with similarity greater than the threshold are the potential keys corresponding to the data features, and the anomaly detection methods corresponding to these keys are the matching results. The word vector contains multi-dimensional values, and with the help of cosine similarity, it can be judged and compared more accurately.

[0011] Preferably, the calculation formula of the cosine similarity is as follows:

[0012]

[0013] Where u and v represent two word vectors respectively. This formula is a common formula for calculating cosine similarity.

[0014] As a preferred embodiment, the large-scale data feature traversal process includes: scaling the value of each dimension in the word vector to be matched to a range of 0 to 255, and representing each word vector with an array of n pixels arranged in sequence, wherein n is the dimension of the word vector, and the value of each dimension of the word vector is the gray value of each pixel, so as to copy the image represented by the pixel array to a white background image of m pixels to obtain a replica, wherein m is x^2 times of n, and x is a natural number greater than or equal to 2, reduce the pixels of the replica to n, read the gray value of each pixel, form a new special word vector, and use the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data volume. When facing massive data, if it is still completely consistent with the method of processing single data, although the accuracy is high, the amount of calculation is large, and the overall efficiency is low, so the above method is used to fuzzify the vector, and although there will be deviations between the fuzzy word vector and the original word vector, the originally similar word vectors still retain a suitable similarity, so the calculation results of the similarity are relatively small, and this method can cope with the computational pressure under massive data.

[0015] As an alternative, the large-scale data feature traversal process includes: scaling the value of each dimension in the word vector to be matched to the range of 0 to 255, dividing 0 to 225 into several orders, modifying the value of each dimension to the middle number in the order corresponding to the value, generating a new special word vector, and using the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data. This solution is still based on fuzzy word vectors to reduce the computational intensity under large-scale data.

[0016] The substantial effects of the present invention include: transforming anomaly detection from a detection rule-driven mode to a data feature-driven mode, generating corresponding outlier detection methods based on the characteristic information of the data itself in each field, and setting up a special fuzzy processing mechanism for large-scale data to achieve scale and automation of data quality verification and improve the efficiency of data quality problem detection. DETAILED DESCRIPTION

[0017] The technical scheme of the present application will be described below in conjunction with the embodiments. In addition, in order to better illustrate the present invention, numerous specific details are provided below. It should be understood by those skilled in the art that the present invention can be implemented without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the gist of the present invention.

[0018] Embodiment 1:

[0019] A large-scale data quality anomaly detection method based on data features comprises the following steps:

[0020] Step S1: Build a data anomaly detection method library, set the corresponding detection method according to each data feature, and summarize it to form a data anomaly detection method library.

[0021] The data anomaly detection method library is stored in a dictionary type, with a tuple consisting of a data feature name and its feature parameters as the dictionary key, and anomaly detection methods corresponding to the data feature as the dictionary value. Python's dictionary type is a key-value pair. Python's dictionary type is used to store data features and their anomaly detection methods. The dictionary key stores a tuple consisting of a data feature name and its feature parameters, and the dictionary value stores the anomaly detection method corresponding to the data feature. The threshold of each anomaly detection method is given by the feature parameters. By storing it in a dictionary, the keys and values ​​can be clearly divided, which is conducive to subsequent matching.

[0022] Step S2: Match the data features with anomaly detection methods, and perform detection based on the anomaly detection methods in the matching results.

[0023] The matching process includes the following: embed the name of the data feature to be processed and the key in the anomaly detection method library into the word vector obtained by NLP, calculate the cosine similarity between the word vectors, and the keys with similarity greater than the threshold are the potential keys corresponding to the data feature, and the anomaly detection methods corresponding to these keys are the matching results. The word vector contains multi-dimensional values, and with the help of cosine similarity, it can be judged and compared more accurately.

[0024] The calculation formula of cosine similarity is as follows:

[0025]

[0026] Where u and v represent two word vectors respectively. This formula is a common formula for calculating cosine similarity.

[0027] Step S3: Traverse large-scale data features to match and detect each data feature.

[0028] The large-scale data feature traversal process of this embodiment includes: scaling the value of each dimension in the word vector to be matched to the range of 0 to 255, and representing each word vector with an array of n pixels arranged in sequence, wherein n is the dimension of the word vector, and the value of each dimension of the word vector is the gray value of each pixel, so as to copy the image represented by the pixel array to a white background image of m pixels to obtain a replica, wherein m is x^2 times of n, and x is a natural number greater than or equal to 2, reduce the pixels of the replica to n, read the gray value of each pixel, form a new special word vector, and use the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data volume. When facing massive data, if it is still completely consistent with the method of processing single data, although the accuracy is high, the amount of calculation is large, and the overall efficiency is low. Therefore, the above method is used to fuzzify the vector. Although there will be deviations between the fuzzy word vector and the original word vector, the originally similar word vectors still retain a suitable similarity, so the calculation results of the similarity are relatively small. This method can cope with the computational pressure under massive data.

[0029] The method library of this embodiment is set up to design corresponding anomaly detection methods for different data features from the perspectives of statistics, common sense, natural laws, professional general knowledge, etc. For example, the data value feature is designed to report anomalies when extreme values ​​appear in the field value, and the date feature reports anomalies for field contents that do not conform to the date format. The setting of the method library is specifically determined according to actual usage requirements, and targeted detection is performed after matching.

[0030] Embodiment 2:

[0031] This embodiment is consistent with the embodiment 1 as a whole, except that the large-scale data feature traversal process is different. The large-scale data feature traversal process of this embodiment includes: scaling the value of each dimension in the word vector to be matched to the range of 0 to 255, and dividing 0 to 225 into several orders, modifying the value of each dimension to the middle number in the order corresponding to the value, generating a new special word vector, and using the special word vector to calculate the cosine similarity to reduce the computational intensity under large-scale data. This solution is still based on fuzzy word vectors to reduce the amount of computation under large-scale data.

[0032] The substantial effects of the above embodiments include: transforming anomaly detection from a detection rule-driven mode to a data feature-driven mode, generating corresponding outlier detection methods based on the characteristic information of the data itself in each field, and setting up a special fuzzy processing mechanism for large-scale data to achieve scale and automation of data quality verification and improve the efficiency of data quality problem detection.

[0033] Through the description of the above implementation mode, the technicians in the relevant field can understand that if the present embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0034] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A large-scale data quality anomaly detection method based on data features, characterized in that: The following steps are involved: Build a data anomaly detection method library, set corresponding detection methods according to each data feature, and summarize them to form a data anomaly detection method library; Match the data features with anomaly detection methods, and perform detection based on the anomaly detection methods in the matching results; Traverse large-scale data features to match and detect each data feature; The data anomaly detection method library is stored in a dictionary type, with a tuple consisting of a data feature name and its feature parameters as the dictionary key, and anomaly detection methods corresponding to the data features as the dictionary value; The matching includes the following process: embedding the names of the data features to be processed and the keys in the anomaly detection method library into word vectors obtained by NLP, calculating the cosine similarity between the word vectors, and the keys with similarity higher than the threshold are the potential keys corresponding to the data features, and the anomaly detection methods corresponding to these keys are the matching results; The large-scale data feature traversal process includes: scaling the value of each dimension in the word vector to be matched to the range of 0 to 255, representing each word vector with an array of n pixels arranged in sequence, wherein n is the dimension of the word vector, and the value of each dimension of the word vector is the grayscale value of each pixel, so as to copy the image represented by the pixel array to a white background image of m pixels to obtain a replica image, wherein m is x^2 times of n, and x is a natural number greater than or equal to 2, reducing the pixels of the replica image to n, reading the grayscale value of each pixel, forming a new special word vector, and using the special word vector to calculate cosine similarity to reduce the computational intensity under large-scale data volume.

2. A large-scale data quality anomaly detection method based on data features according to claim 1, characterized in that: The calculation formula of the cosine similarity is as follows: Where u and v represent two word vectors respectively.

Citation Information

Patent Citations

  • Method, device, electronic device and storage medium for verifying and processing

    CN108256074A

  • Detecting method for moving object in video

    CN103971385A

  • Data quality detection method and apparatus

    CN106708909A

  • Abnormal data processing method and device

    CN109766331A

  • Pedestrian image attribute feature quantification method

    CN109993128A