Abnormal data evaluation and analysis method and system for data assets
By introducing data type identification and pattern recognition modules into the data asset analysis system, and combining statistics and machine learning technology to perform multivariate anomaly detection, the problem of difficult to identify complex anomaly patterns in the existing technology is solved, and fully automatic analysis and efficient anomaly recognition of data assets are realized.
Patent Information
- Application Number
- CN202411526911.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-03-25
AI Technical Summary
The prior art has limitations when dealing with data asset exceptions, making it difficult to accurately identify complex anomaly patterns, especially in the analysis of qualitative data and multivariate data sets. It is impossible to independently select appropriate algorithms, and there are technical barriers for non-professional personnel, so it is impossible to quickly establish anomaly detection mechanism.
The data type is automatically identified through the data type identification module, different anomaly data detection algorithms are selected, and multivariate anomaly detection is carried out in combination with statistics and machine learning techniques, and a report is generated. The system includes data loading, type recognition, pattern recognition, abnormality detection and report generation modules to realize fully automatic analysis of data assets.
It realizes fully automatic analysis of data assets, automatically identifying multivariate data patterns, improves the efficiency and accuracy of abnormal data recognition, reduces technical barriers, and is suitable for non-professional personnel to quickly establish an abnormality detection mechanism.
Smart Images
Figure CN119691437A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data assets and abnormal data analysis, and specifically to a method and system for evaluating and analyzing abnormal data of data assets. Background Art
[0002] As one of the "five major production factors", data has become an important asset for enterprises and organizations. However, various abnormalities may occur during the generation, storage, processing and use of data, such as data errors, data missing, abnormal data fluctuations, etc. These anomalies not only affect the quality and availability of data assets, but may also lead to wrong decisions and business losses. How to discover and eliminate these abnormal data is particularly critical.
[0003] Traditional data analysis methods such as statistical methods provide a basis for anomaly detection of data assets. Machine learning technology can select algorithms to set various parameters for training and learning to identify single or similar quantitative data. However, at present, the application of statistical and machine learning methods in the field of data asset anomaly analysis and evaluation is not mature enough. There are limitations in dealing with data asset anomalies. It is difficult to accurately identify complex abnormal patterns. For example, the support for anomaly detection of qualitative data is general, and it is impossible to perform overall identification for multivariate data sets. It is impossible to independently select different algorithms according to different types. There are technical barriers for non-professionals, and it is impossible to quickly establish anomaly detection mechanisms. Summary of the invention
[0004] In view of the deficiencies in the prior art, the present invention provides a method and system for evaluating and analyzing abnormal data of data assets, which solves the problems of limitations in processing data asset anomalies, such as difficulty in accurately identifying complex abnormal patterns, general support for anomaly detection of qualitative data, inability to perform overall identification for multivariate data sets, inability to independently select different algorithms according to different types, technical barriers for non-professionals, and inability to quickly establish anomaly detection mechanisms.
[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for evaluating and analyzing abnormal data of data assets, comprising the following steps:
[0006] Step 1: When performing abnormal data analysis, load and import data through the data loading module;
[0007] Step 2: Based on the data imported in step 1, use the data type identification module to identify the data type so that each column of data has a corresponding type;
[0008] Step 3: According to the data types classified in step 2, different abnormal data detection algorithms are selected through the data pattern recognition module;
[0009] Step 4: Data pattern recognition is required for the multivariate data in step 3, and then anomaly detection of the multivariate is performed through an abnormal data detection module;
[0010] Step 5: Based on the results summarized in step 4, use the report generation module to generate a report to export the abnormal data analysis report of this data set.
[0011] Preferably, the data loading module in step 1 searches and reads files based on Python's os library and uses the pandas library to parse data files.
[0012] Preferably, the data types divided in step 2 are numerical type, coding type and time type.
[0013] Preferably, the anomaly detection in step three adopts an anomaly detection method based on statistical theory and an anomaly detection algorithm based on machine learning.
[0014] A data asset abnormal data evaluation and analysis system, comprising:
[0015] Data loading module, used to load the data set that needs to be analyzed;
[0016] The data type identification module uses language model (LLM) technology to automatically identify the data type of each column in the data set, providing a basis for subsequent abnormal data analysis and processing;
[0017] The data pattern recognition module uses PPMCC technology to identify potential patterns in data sets for numerical data that meet normal distribution. It is mainly used to detect the intrinsic structure of data between multiple variables.
[0018] The abnormal data detection module is designed and implemented using a probabilistic programming model, and combines statistical theory and machine learning techniques for outlier detection;
[0019] The report generation module is used to organize the results of the entire abnormal data detection and processing process into a report format and provide it to users for review. The results can also be exported to PDF format.
[0020] Preferably, the data loading module searches and reads files based on Python's os library, uses the pandas library to parse data files, and converts the data into a data structure for subsequent processing.
[0021] Preferably, the data type identified by the data type identification module is a numerical type, a coding type, or a time type.
[0022] Preferably, the anomaly detection of the abnormal data detection module adopts an anomaly detection method based on statistical theory and an anomaly detection algorithm based on machine learning.
[0023] Preferably, the anomaly detection method based on statistical theory includes interquartile range (IQR), 3-Sigma rule and Grubbs' test.
[0024] Preferably, the method of the anomaly detection algorithm based on machine learning includes Isolation Forests and DBSCAN.
[0025] The present invention provides a method and system for evaluating and analyzing abnormal data of data assets.
[0026] Beneficial effects:
[0027] 1. The present invention realizes fully automatic analysis of input data assets, automatically selects appropriate abnormal analysis algorithms for various types of data, automatically identifies multivariate data patterns to perform multivariate combination abnormal data identification, and finally automatically generates a report to provide analysis results.
[0028] 2. The present invention can select different modes. For inexperienced personnel, abnormal data can be analyzed and identified automatically. For experienced personnel, appropriate determination algorithms or customized determination modes can be selected in a targeted manner to realize automated analysis of abnormal data of data assets, thereby improving the work efficiency of analyzing abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a flow chart of the present invention;
[0030] Figure 2 It is a specific schematic diagram of the present invention. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] Example:
[0033] As one aspect of the present invention, please refer to the attached Figure 1 -Attached Figure 2 The embodiment of the present invention provides a method for evaluating and analyzing abnormal data of data assets, including the following steps:
[0034] Step 1: When analyzing abnormal data, load and import data through the data loading module. The data loading module searches and reads files based on Python's os library and uses the pandas library to parse data files.
[0035] Step 2: Based on the data imported in step 1, use the data type identification module to identify the data type. The data types are numerical type, coding type and time type, so that each column of data has a corresponding type;
[0036] Step 3: According to the data types classified in step 2, different abnormal data detection algorithms are selected through the data pattern recognition module;
[0037] Step 4: Data pattern recognition is required for the multivariate data in step 3, and then multivariate anomaly detection is performed through the abnormal data detection module, where anomaly detection adopts anomaly detection methods based on statistical theory and anomaly detection algorithms based on machine learning.
[0038] Based on the above-mentioned abnormal data evaluation and analysis method for data assets, as another aspect of the present invention, the present invention proposes an abnormal data evaluation and analysis system for data assets, including the following modules:
[0039] The data loading module is used to load the data set to be analyzed. This module uses Python's os library to find and read files, uses the pandas library to parse data files, supports multiple data formats such as CSV, Excel, etc., and converts the data into a data structure that can be processed later.
[0040] The data type identification module uses language model (LLM) technology to automatically identify the data type of each column in the data set, such as numerical type, coding type, time type, etc., providing a basis for subsequent abnormal data analysis and processing.
[0041] The data pattern recognition module uses PPMCC technology to identify potential patterns in the data set for numerical data that satisfies the normal distribution. It is mainly used to detect the intrinsic structure of data between multiple variables, facilitating subsequent multivariate outlier detection.
[0042] The abnormal data detection module is designed and implemented using a probabilistic programming model, combining statistical theory and machine learning technology for outlier detection. This module includes but is not limited to the following methods:
[0043] Anomaly detection methods based on statistical theory, such as interquartile range (IQR), 3-Sigma rule, Grubbs' test, etc.;
[0044] Anomaly detection algorithms based on machine learning, such as Isolation Forests and DBSCAN.
[0045] The report generation module organizes the results of the entire abnormal data detection and processing process into a report format and provides it to users for review. The results can also be exported to PDF for further analysis and sharing.
[0046] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for evaluating and analyzing abnormal data of data assets, characterized in that: The following steps are involved: Step 1: When performing abnormal data analysis, load and import data through the data loading module; Step 2: Based on the data imported in step 1, use the data type identification module to identify the data type so that each column of data has a corresponding type; Step 3: According to the data types classified in step 2, different abnormal data detection algorithms are selected through the data pattern recognition module; Step 4: Data pattern recognition is required for the multivariate data in step 3, and then anomaly detection of the multivariate is performed through an abnormal data detection module; Step 5: Based on the results summarized in step 4, use the report generation module to generate a report to export the abnormal data analysis report of this data set.
2. The abnormal data evaluation and analysis method of data assets according to claim 1 is characterized in that: The data loading module in step 1 searches and reads files based on Python's os library and uses the pandas library to parse data files.
3. The abnormal data evaluation and analysis method of data assets according to claim 1 is characterized in that: The data types divided in step 2 are numerical type, coding type and time type.
4. The abnormal data evaluation and analysis method of data assets according to claim 1 is characterized in that: The anomaly detection in step three adopts an anomaly detection method based on statistical theory and an anomaly detection algorithm based on machine learning.
5. A data asset abnormal data evaluation and analysis system, using a data asset abnormal data evaluation and analysis method according to any one of claims 1 to 4, characterized in that: include: Data loading module, used to load the data set that needs to be analyzed; The data type identification module uses language model (LLM) technology to automatically identify the data type of each column in the data set, providing a basis for subsequent abnormal data analysis and processing; The data pattern recognition module uses PPMCC technology to identify potential patterns in data sets for numerical data that meet normal distribution. It is mainly used to detect the intrinsic structure of data between multiple variables. The abnormal data detection module is designed and implemented using a probabilistic programming model, and combines statistical theory and machine learning techniques for outlier detection; The report generation module is used to organize the results of the entire abnormal data detection and processing process into a report format and provide it to users for review. The results can also be exported to PDF format.
6. The abnormal data evaluation and analysis system for data assets according to claim 5, characterized in that: The data loading module searches and reads files based on Python's os library, uses the pandas library to parse data files, and converts the data into a data structure that can be processed later.
7. The abnormal data evaluation and analysis system for data assets according to claim 5, characterized in that: The data types identified by the data type identification module are numerical type, coding type, and time type.
8. The abnormal data evaluation and analysis system for data assets according to claim 5, characterized in that: The abnormality detection of the abnormal data detection module adopts an abnormality detection method based on statistical theory and an abnormality detection algorithm based on machine learning.
9. The abnormal data evaluation and analysis system for data assets according to claim 8, characterized in that: The anomaly detection methods based on statistical theory include interquartile range (IQR), 3-Sigma rule and Grubbs' test.
10. The abnormal data evaluation and analysis system for data assets according to claim 8, characterized in that: The methods of the anomaly detection algorithms based on machine learning include Isolation Forests and DBSCAN.