Abnormal data identification method and device, equipment and storage medium

Abstract

The invention discloses an abnormal data recognition method, device and equipment and a storage medium, and relates to the technical field of industrial data analysis, and the abnormal data recognition method comprises the steps: collecting a target, a time sequence and historical data, calculating a threshold value, a standard deviation and a normal range based on the data, and training an anomaly recognition model. And marking the data exceeding the threshold value as a first anomaly, the data having the standard deviation anomaly as a second anomaly, and the data not in the normal range as a third anomaly. Fourth abnormal data are recognized through the model, finally the abnormal data are summarized, an abnormal report containing features and reasons is generated, and abnormal data recognition is completed. According to the method, the abnormal data in the industrial data can be accurately identified, stable operation of the industry and equipment is ensured, and the loss risk caused by abnormity is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of industrial data analysis, and in particular to an abnormal data identification method, device, equipment and storage medium. Background Art

[0002] In industrial production, with the widespread use of equipment and sensors, a large amount of data is generated. This data is of great significance for improving production efficiency, reducing failure rates and ensuring product quality. However, the data often contains some abnormal values, which may be caused by equipment failure, sensor errors or changes in the external environment. Timely and accurate identification and processing of these abnormal data not only helps to ensure production safety, but also significantly reduces maintenance costs and improves product quality.

[0003] Currently, the industry generally uses a variety of methods to identify abnormal data. These include statistical methods, such as control charts and z-scores, which are used to detect deviations from data distribution; machine learning methods, such as support vector machines, which train models to identify abnormal patterns; and deep learning methods, such as autoencoders, which can automatically extract features of data. These methods have improved the accuracy and efficiency of anomaly detection to a certain extent and have achieved good results in practical applications.

[0004] Although existing methods have achieved certain results in abnormal data identification, some problems still exist. For example, many statistical methods have strict assumptions about data distribution and may not be able to adapt to complex industrial environments; machine learning and deep learning methods often require a large amount of labeled data for training, but in actual production, it may be very difficult to obtain labeled data. In addition, different types of abnormal data may have different characteristics, making it difficult for a single method to fully cover all scenarios. Therefore, how to accurately identify abnormal data in industrial data has become an urgent problem to be solved.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The purpose of this application is to provide a method, device, equipment and storage medium for identifying abnormal data, aiming to solve the technical problem of how to accurately identify abnormal data in industrial data.

[0007] To achieve the above objectives, the present application proposes a method for identifying abnormal data, the method comprising:

[0008] Obtain target data, time series data, and historical data sets;

[0009] Obtaining an upper threshold, a lower threshold, a standard deviation score, a normal data range, and a target abnormal data identification model according to at least one of the target data, the time series data, and the historical data set;

[0010] Taking data in the target data that is greater than the upper threshold or less than the lower threshold as first abnormal data;

[0011] The data in the target data whose standard deviation score is greater than a preset standard deviation score threshold is taken as second abnormal data;

[0012] taking data in the target data that is not within the normal data range as third abnormal data;

[0013] Identify the target data using the target abnormal data identification model to obtain fourth abnormal data;

[0014] An abnormal data identification report is generated according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data to complete abnormal data identification.

[0015] In one embodiment, the step of obtaining an upper threshold, a lower threshold, a standard deviation score, a normal data range, and a target abnormal data identification model according to at least one of the target data, the time series data, and the historical data set includes:

[0016] Calculate an upper threshold and a lower threshold according to the target data, the time series data and the historical data set;

[0017] Calculate a standard deviation score based on the target data;

[0018] Sorting the target data according to numerical values ​​to obtain ordered data;

[0019] Calculate the first quartile and the second quartile according to the ordered data;

[0020] Calculate the normal data range according to the first quartile and the second quartile;

[0021] A target abnormal data recognition model is constructed based on the historical data set.

[0022] In one embodiment, the step of calculating the upper threshold and the lower threshold according to the target data, the time series data and the historical data set includes:

[0023] Calculate the average value and standard deviation based on the historical data set;

[0024] Smoothing and seasonal decomposing the time series data to obtain seasonal components;

[0025] Setting an initial threshold value according to the mean value and the standard deviation;

[0026] Analyzing the seasonal components to obtain a seasonal variation cycle;

[0027] Adjust the initial threshold according to the seasonal variation cycle to obtain a reference threshold;

[0028] An upper threshold and a lower threshold are calculated based on the reference threshold and the historical data within a preset time window.

[0029] In one embodiment, the step of constructing a target abnormal data recognition model according to the historical data set includes:

[0030] Dividing the historical data set into a training set and a test set;

[0031] Determine the number of sub-models, the sub-model depth threshold, and the split node sample number threshold according to the historical data set;

[0032] An initial abnormal data recognition model is constructed according to the training set, the number of sub-models, the sub-model depth threshold and the split node sample number threshold;

[0033] Inputting the test set into the initial abnormal data recognition model to obtain a test result;

[0034] determining an anomaly score threshold based on the test results;

[0035] A target abnormal data recognition model is obtained according to the initial abnormal data recognition model and the abnormal score threshold.

[0036] In one embodiment, the step of constructing an initial abnormal data recognition model according to the training set, the number of sub-models, the sub-model depth threshold, and the split node sample number threshold comprises:

[0037] Performing sampling with replacement on the training set to obtain out-of-bag data and multiple sample subsets;

[0038] Obtain a submodel according to the sample subset, the number of submodels, the submodel depth threshold, and the split node sample number threshold;

[0039] The sub-model is trained by using the sample subset, and the sub-model is tested by using the out-of-bag data to obtain a target sub-model;

[0040] An initial abnormal data recognition model is constructed based on the multiple target sub-models.

[0041] In one embodiment, the step of obtaining the sub-model according to the sample subset, the number of sub-models, the sub-model depth threshold and the split node sample number threshold comprises:

[0042] According to the sample subset, determining the splitting feature based on information gain, Gini index, and mean square error;

[0043] Creating a plurality of sub-model nodes according to the splitting feature;

[0044] Obtaining a sub-model depth and a split subset according to the sub-model node;

[0045] When the sub-model depth is less than the sub-model depth threshold and the split subset is greater than the split node sample number threshold, the split subset is used as a sample subset, and the step of determining the split feature based on the information gain, the Gini index, and the mean square error according to the sample subset is returned;

[0046] When the sub-model depth is less than the sub-model depth threshold and the split subset is greater than the split node sample number threshold, the sub-model node is taken as a leaf node to obtain a sub-model.

[0047] In one embodiment, the test set includes data labels, and the step of determining the anomaly score threshold according to the test results includes:

[0048] Calculate the true positive rate and the false positive rate according to the test results and the data labels;

[0049] Draw an operating characteristic curve according to the true positive rate and the false positive rate;

[0050] An abnormality score threshold is determined according to the operating characteristic curve.

[0051] In one embodiment, the step of generating an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data, and the fourth abnormal data, after completing the abnormal data identification step, further includes:

[0052] A risk level distribution chart created by a preset visualization tool displays the number of abnormal data at different risk levels in the abnormal data identification report;

[0053] The time series graph created by the preset visualization tool displays the abnormal data trend in the abnormal data identification report in chronological order and is colored according to the risk level;

[0054] The stacked area chart or faceted chart created by the preset visualization tool displays the abnormal data in the abnormal data identification report according to the risk level and the time sequence respectively.

[0055] In one embodiment, after the step of displaying the abnormal data identification report by risk level and time sequence respectively through a preset visualization tool, the method further includes:

[0056] Obtaining user feedback information, and obtaining false positive data and missed negative data according to the user feedback information;

[0057] The number of sub-models, the sub-model depth threshold, and the split node sample number threshold are adjusted according to the false positive data and the missed negative data to obtain a target number of sub-models, a target sub-model depth threshold, and a target split node sample number threshold;

[0058] The target abnormal data recognition model is reconstructed according to the historical data set, the number of target sub-models, the target sub-model depth threshold and the target split node sample number threshold.

[0059] In one embodiment, the step of identifying the target data by using the target abnormal data identification model to obtain fourth abnormal data includes:

[0060] Identify the target data using the target abnormal data identification model to obtain an abnormality score;

[0061] The data in the target data whose abnormality score is greater than or equal to the abnormality score threshold is taken as fourth abnormal data.

[0062] In one embodiment, the step of acquiring target data and time series data includes:

[0063] Obtain industrial sensor data, production equipment data, production plan data, system log data, user behavior data, and environmental data;

[0064] Fusing the industrial sensor data, the production equipment data, the system log data, the user behavior data, and the environment data to obtain initial data;

[0065] Obtaining time series data according to the initial data;

[0066] Converting non-numeric data in the initial data into numeric data to obtain reference data;

[0067] The reference data is standardized to obtain target data.

[0068] In one embodiment, the step of generating an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data, and the fourth abnormal data, and completing abnormal data identification includes:

[0069] taking overlapping data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as high-risk abnormal data;

[0070] taking any two overlapping data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as medium-risk abnormal data;

[0071] taking data other than the high-risk abnormal data and the medium-risk abnormal data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as low-risk abnormal data;

[0072] An abnormal data identification report is generated according to the high-risk abnormal data, the medium-risk abnormal data, and the low-risk abnormal data.

[0073] In addition, to achieve the above purpose, the present application also proposes an abnormal data identification device, the device comprising:

[0074] Data acquisition module, used to acquire target data, time series data and historical data sets;

[0075] A data processing module, used to obtain an upper threshold, a lower threshold, a standard deviation score, a normal data range and a target abnormal data identification model according to at least one of the target data, the time series data and the historical data set;

[0076] A first identification module, configured to take data in the target data that is greater than the upper threshold or less than the lower threshold as first abnormal data;

[0077] A second identification module, configured to take the data in the target data whose standard deviation score is greater than a preset standard deviation score threshold as second abnormal data;

[0078] A third identification module, configured to take data in the target data that is not within the normal data range as third abnormal data;

[0079] a fourth identification module, configured to identify the target data by using the target abnormal data identification model to obtain fourth abnormal data;

[0080] A report generating module is used to generate an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data to complete abnormal data identification.

[0081] In addition, to achieve the above-mentioned purpose, the present application also proposes an abnormal data identification device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the abnormal data identification method described above.

[0082] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the abnormal data identification method described above are implemented.

[0083] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the abnormal data identification method described above are implemented.

[0084] One or more technical solutions proposed in this application have at least the following technical effects:

[0085] Obtain target data, time series data and historical data sets; obtain an upper threshold, a lower threshold, a standard deviation score, a normal data range and a target abnormal data recognition model according to at least one of the target data, the time series data and the historical data sets; take the data in the target data that is greater than the upper threshold or less than the lower threshold as the first abnormal data; take the data in the target data that has a standard deviation score greater than a preset standard deviation score threshold as the second abnormal data; take the data in the target data that is not within the normal data range as the third abnormal data; identify the target data through the target abnormal data recognition model to obtain the fourth abnormal data; generate an abnormal data recognition report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data to complete the abnormal data recognition. First, obtain target data, time series data and historical data sets to ensure the comprehensiveness and diversity of the data and provide a basis for subsequent analysis. These data are used to calculate the upper threshold, the lower threshold, the standard deviation score and the normal data range, and train the target abnormal data recognition model. These indicators and models provide accurate standards and basis for determining abnormalities and enhance the reliability of recognition. Next, the system marks the data in the target data that is greater than the upper threshold or less than the lower threshold as the first abnormal data. This process can quickly identify significantly deviated anomalies and help operation and maintenance personnel to detect potential faults in a timely manner. Subsequently, by calculating the standard deviation score and screening out data points greater than the preset standard deviation score threshold, they are marked as the second abnormal data to ensure that those statistically significant complex anomalies can be captured. After that, the system identifies the data points that are not within the normal range by comparing with the normal data range and marks them as the third abnormal data to confirm relatively small but still important abnormal situations. The trained target abnormal data recognition model is then used to identify the target data, and the fourth abnormal data that meets the model characteristics is automatically detected to improve the efficiency of processing large-scale data and to detect complex abnormal patterns in a timely manner. Finally, all identified abnormal data are summarized to generate a detailed abnormal data recognition report, recording the characteristics and potential causes of various types of anomalies, providing an important basis for subsequent decision-making and maintenance. Through this series of steps, the abnormal data recognition system effectively improves the accuracy and efficiency of anomaly detection, can accurately identify abnormal data in industrial data, ensure the stable operation of industry and equipment, and reduce the risk of losses caused by anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0087] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0088] Figure 1 A flowchart of the first embodiment of the abnormal data identification method of the present application is provided;

[0089] Figure 2 A flowchart of the second embodiment of the abnormal data identification method of the present application is provided;

[0090] Figure 3 This is a schematic diagram of the module structure of the abnormal data identification device according to an embodiment of the present application;

[0091] Figure 4 Schematic diagram of the device structure of the hardware operating environment involved in the abnormal data identification method in the embodiment of the present application.

[0092] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0093] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0094] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0095] In industrial production, the extensive use of equipment and sensors generates a large amount of data, which is crucial to improving efficiency, reducing failure rates and ensuring product quality. However, the presence of outliers in the data may be caused by equipment failure, sensor errors or environmental changes, which has a significant impact on production safety and costs. At present, the industry uses statistics, machine learning and deep learning methods to identify abnormal data. Although this has improved accuracy and efficiency, it still has problems such as adaptability and difficulty in data labeling.

[0096] The main solution of the embodiment of the present application is: by collecting comprehensive and diverse data, calculating thresholds and training anomaly recognition models, the system can mark abnormal data that are significantly deviated, statistically significant and outside the normal range, and automatically identify complex abnormal patterns through the model, and finally generate a detailed report containing abnormal characteristics and causes, providing a basis for decision-making and maintenance.

[0097] It should be noted that the execution subject of the embodiment of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an abnormal data identification system, etc. The following takes the abnormal data identification system as an example to illustrate this embodiment and the following embodiments.

[0098] Based on this, the present application embodiment provides a method for identifying abnormal data. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the abnormal data identification method of the present application.

[0099] In this embodiment, the abnormal data identification method includes steps S10 to S70:

[0100] Step S10, obtaining target data, time series data and historical data sets;

[0101] It should be noted that target data refers to the main data type that needs to be paid attention to and processed in a specific analysis or monitoring task. In an industrial environment, target data usually includes industrial data (such as production volume, energy consumption, temperature, pressure, etc.) and equipment data (such as equipment operating status, fault records, sensor readings, etc.). These data are used to evaluate equipment performance, monitor production processes, identify anomalies, etc. Time series data is a data set obtained by arranging target data in chronological order. Time series data usually contains the target data value at each time point and can be used to analyze the trend, periodicity and emergencies of data over time. In industrial applications, time series data is often used to monitor the real-time status of equipment, analyze the dynamic changes of production processes, and perform predictive maintenance. Historical data sets refer to a collection of data such as industrial data and equipment data collected in a certain period of time in the past. Historical data sets contain the operation records and performance data of equipment in different time periods, which are usually used to train models, perform trend analysis or as a comparison benchmark. By analyzing historical data sets, enterprises can identify long-term trends, periodic patterns and abnormal events, thereby improving the accuracy and effectiveness of decision-making.

[0102] It is understandable that relevant target data, such as operating indicators such as temperature, pressure and vibration of equipment, are first collected through automation systems or sensors to ensure the real-time and accuracy of the data, so as to facilitate the timely detection of potential problems. Next, these target data are organized into time series data. The specific methods include sorting and interpolating the data by timestamp to fill in possible missing values, which can reveal the trend and periodic changes of the data, provide a clear time dimension for subsequent analysis, and make anomaly identification more efficient and reliable. Subsequently, historical data sets are collected and integrated, usually by extracting operating data from a certain period of time in the past from the database, and performing data standardization and normalization to eliminate the impact of data fluctuations in different time periods. In this way, a comparative benchmark can be established to identify which current data points deviate from the normal operating state, thereby improving the accuracy of anomaly detection and ultimately achieving better maintenance decisions and production efficiency.

[0103] As an example, the step of obtaining target data and time series data includes: obtaining industrial sensor data, production equipment data, production plan data, system log data, user behavior data and environmental data; fusing the industrial sensor data, the production equipment data, the system log data, the user behavior data and the environmental data to obtain initial data; obtaining time series data based on the initial data; converting non-numeric data in the initial data into numerical data to obtain reference data; and standardizing the reference data to obtain target data.

[0104] Industrial sensor data refers to real-time data collected by industrial sensors, such as temperature, pressure, flow, vibration, etc., which reflect the status of equipment or production processes. Production equipment data includes information such as equipment operation status, fault records, working hours, production efficiency, etc., which are usually obtained through equipment management systems or monitoring systems. Production planning data refers to information related to production scheduling and resource allocation, such as production goals, process flow, raw material requirements and production progress, etc., which are used to guide actual production. System log data refers to log files generated when the operating system, application or equipment is running, containing system events, error messages and operation records, etc., which are used for monitoring and troubleshooting. User behavior data refers to the collection of user operation records and behavior patterns in the system, such as login time, operation frequency and usage functions, etc., which are usually used to analyze user habits and optimize the system. Environmental data includes external factors that affect the production environment, such as temperature, humidity, noise, air quality, etc., which may affect the performance of the equipment and the quality of production. Initial data refers to the data set obtained by integrating the above different data sources (industrial sensors, production equipment, system logs, user behavior and environmental data), which contains original information from various sources. Non-numeric data refers to data types that cannot be directly subjected to mathematical operations, such as categorical data (such as device status "normal" or "fault") or text data (such as error logs and user feedback), which usually need to be encoded or converted. Numerical data refers to data types that can be subjected to mathematical operations, such as integers and floating-point numbers, and are usually used for statistical analysis and modeling. Reference data refers to processed data sets, including the results obtained after converting non-numeric data to numerical data, and are usually used for subsequent analysis and modeling. Standardization is the process of converting data with different features to a unified scale, usually by subtracting the mean and dividing by the standard deviation to ensure that each feature is in the same range, thereby eliminating the dimensional effects between different features and facilitating comparison and analysis.

[0105] First, industrial sensor data (such as temperature, pressure, flow, etc.), production equipment data (such as operating status, fault records, working hours, etc.), production plan data (such as production goals, process flow, etc.), system log data (such as system events, error messages, etc.), user behavior data (such as login time, operation frequency, etc.) and environmental data (such as temperature, humidity, noise, etc.) are collected through sensors and system interfaces. This step ensures the acquisition of comprehensive and multi-dimensional data, which helps to fully understand the production environment. Then, through data fusion techniques such as data cleaning, merging and alignment, various types of data are integrated together to generate initial data. In this process, timestamps are usually used for data alignment to ensure the consistency of data from different sources in the time dimension, and to ensure information integrity and comparability. Subsequently, time series data is constructed based on the initial data. The specific method is to sort the initial data by time and interpolate or fill in missing values ​​to achieve data continuity in time, which enables subsequent analysis to better capture time trends and cyclical changes. Next, the non-numeric data in the initial data (such as device status, user behavior description, etc.) is converted into numerical data. Common methods include label encoding or one-hot encoding, so that these data can be converted into a format that the model can process and obtain reference data. Finally, the reference data is standardized, usually by calculating the mean and standard deviation, and converting the data into a form with a mean of 0 and a variance of 1. This processing step eliminates the dimensional influence between different data features and improves the comparability of the data, thereby ultimately generating the target data. This improvement in the data structure enables the subsequent anomaly detection model to more effectively identify data points that deviate from the normal pattern.

[0106] Step S20, obtaining an upper threshold, a lower threshold, a standard deviation score, a normal data range, and a target abnormal data recognition model according to at least one of the target data, the time series data, and the historical data set;

[0107] It should be noted that the upper threshold refers to a maximum value determined in data analysis. When a data point exceeds this value, it may be considered an anomaly. This threshold is usually derived from the statistical analysis of historical data sets, such as by calculating the maximum value of normal data or setting it according to a multiple of a standard deviation. The lower threshold corresponds to the upper threshold and refers to a minimum value determined in data analysis. When a data point is lower than this value, it may also be considered an anomaly. The lower threshold can also be based on the statistical analysis of historical data sets, such as calculating the minimum value of normal data or setting it according to the standard deviation. The standard deviation score is an indicator of the relative deviation between a data point and the mean, usually calculated by subtracting the mean of a data point from the data set and then dividing it by the standard deviation. The standard deviation score can be used to identify outliers. Usually, when the standard deviation score is greater than a certain threshold (such as ±2 or ±3), the data point may be considered an anomaly. The normal data range refers to an interval determined after statistical analysis in a given data set, and the data within this interval is considered normal. The normal data range is usually bounded by an upper threshold and a lower threshold, indicating the expected operation or running state. The target anomaly data identification model is a mathematical or machine learning model used to identify and classify anomalies in data. It is built based on the analysis results of target data, time series data, and historical data sets. The model can accurately identify abnormal data points, thereby achieving effective anomaly detection.

[0108] It can be understood that, first, by analyzing the target data, time series data and historical data sets, key statistical features such as mean and standard deviation are extracted to calculate the upper and lower thresholds. Usually, the upper threshold is set as the mean plus several times the standard deviation, while the lower threshold is the mean minus the same multiple of the standard deviation. Such a setting ensures that normal data can be effectively covered. Next, these thresholds are used to define the normal data range, which provides a benchmark for the normality judgment of subsequent data points. Then, for the labeled data in the historical data set, the standard deviation score of each data point is calculated. This score is obtained by subtracting the mean from each data point and then dividing it by the standard deviation. It is used to quantify the degree of deviation of the data point. When the absolute value of the standard deviation score is large, it means that the data point may be an outlier. Finally, based on the historical data set (which is labeled, including normal and abnormal data), the target abnormal data recognition model is trained. The model learns how to perform anomaly detection based on the features and corresponding labels in the historical data, and finally achieves accurate classification of new data so as to identify potential abnormal situations in a timely manner.

[0109] Step S30, taking data in the target data that is greater than the upper threshold or less than the lower threshold as first abnormal data;

[0110] It should be noted that the first abnormal data refers to the data points in the target data set that are identified as abnormal after threshold analysis. These data points are screened according to the set upper and lower thresholds. When the values ​​in the target data are greater than the upper threshold or less than the lower threshold, these data points are marked as first abnormal data, which means that they are significantly deviated from the normal data range and may reflect equipment failure, operational errors or other potential problems.

[0111] It is understandable that the upper and lower thresholds are first calculated by analyzing the time series data and taking into account the seasonal variation cycle, and these thresholds are designed to reflect the normal fluctuation range of the data. Next, the system will traverse each data point in the target data set to check whether its value exceeds the calculated threshold range. Specifically, if the value of a data point is greater than the upper threshold, it means that the data point is significantly higher than the normal range, which may indicate equipment failure or abnormal operation; if the value of the data point is lower than the lower threshold, it means that it is significantly lower than the normal level, which may reflect system problems or external interference. All data points that meet these conditions will be marked as the first abnormal data. This process effectively classifies data that is significantly deviated from the normal state, thereby laying the foundation for subsequent abnormal analysis and processing. Through this screening step, potential faults or problems can be quickly identified, so that corresponding preventive and corrective measures can be taken in a timely manner.

[0112] Step S40, taking the data in the target data whose standard deviation score is greater than a preset standard deviation score threshold as second abnormal data;

[0113] It should be noted that the preset standard deviation score threshold is a pre-set value used to determine whether a data point is considered abnormal. This threshold is usually determined based on field experience or data distribution characteristics, and is commonly set to ±2 or ±3. For example, if the preset standard deviation score threshold is ±2, then data points with a standard deviation score greater than 2 or less than -2 will be considered abnormal. Second abnormal data refers to data points in the target data whose standard deviation score is greater than the preset standard deviation score threshold. These data points are independent of the first abnormal data (i.e., data greater than the upper threshold or less than the lower threshold), represent another different abnormal situation, and emphasize the significant deviation of the data point from normal data. In this way, different types of abnormal data can be identified separately, providing a more comprehensive perspective for subsequent analysis and processing.

[0114] It is understandable that the standard deviation score of each data point in the target data is first calculated. This score is obtained by subtracting the mean of the data set from the value of each data point and then dividing it by the standard deviation of the data set. It is used to measure the relative deviation of the data point from the mean. Next, a preset standard deviation score threshold is set, usually selected as ±2 or ±3, as the criterion for determining abnormalities. The system will traverse the target data set and identify those data points whose standard deviation scores are greater than the preset threshold. Specifically, if the standard deviation score of a data point is greater than the preset upper limit (for example, 3), it means that the data point deviates significantly from the mean of the normal data and may indicate an abnormal situation. All data points that meet this condition will be marked as second abnormal data, and this process provides a basis for further analysis of abnormal data. In this way, points that are extremely deviated in the data distribution can be effectively identified to help monitor and diagnose potential problems, thereby achieving more accurate data management and fault detection.

[0115] Step S50, taking the data in the target data that is not within the normal data range as third abnormal data;

[0116] It should be noted that the third abnormal data refers to those data points in the target data set that are not within the normal data range calculated based on the interquartile range. The normal data range is usually calculated in combination with the interquartile range. For example, the upper and lower limits are determined by calculating the IQR (interquartile range, Q3-Q1) to define the fluctuation range of the normal data. If the value of a data point in the target data exceeds this normal data range, that is, it is higher than Q3 plus a certain multiple of the IQR or lower than Q1 minus the same multiple of the IQR, these data points will be marked as third abnormal data. By identifying these third abnormal data, situations that deviate significantly from the normal state can be effectively captured, helping to promptly detect potential equipment failures or other problems, and then take appropriate corrective measures.

[0117] It can be understood that the quartiles (Q1 and Q3) are first calculated based on the target data set, and then the range of normal data is determined by calculating the interquartile range (IQR, i.e. Q3 minus Q1). Next, the system traverses each data point in the target data set to check whether it exceeds the normal data range calculated based on the quartiles. Specifically, if the value of a data point is higher than Q3 plus a certain multiple of IQR or lower than Q1 minus the same multiple of IQR, then the data point will be marked as the third abnormal data. These third abnormal data represent a significant deviation from the normal data range, which may reflect equipment failure, abnormal operation or external interference. By identifying and recording these abnormal data, system operation and maintenance personnel can promptly discover and solve potential problems, thereby ensuring the stability and reliability of the production environment.

[0118] Step S60, identifying the target data by using the target abnormal data identification model to obtain fourth abnormal data;

[0119] It should be noted that the fourth abnormal data refers to the data points that are determined to be abnormal by the model after the target data is identified by the target abnormal data identification model. The model is trained based on historical data sets, which contain labeled normal and abnormal data, so the model can learn abnormal characteristics and patterns. When the model identifies new target data, it automatically determines whether each data point is abnormal based on these characteristics. The abnormal data points identified by the model, namely the fourth abnormal data, may represent more complex abnormal patterns, which are usually difficult to detect with the aforementioned simple rules (such as thresholds and interquartile ranges). Identifying the fourth abnormal data helps capture complex or hidden anomalies and provides a basis for further analysis and prevention.

[0120] It is understandable that the trained target abnormal data recognition model is first applied to the target data set. The model is trained based on normal and abnormal samples in historical data, and has learned the characteristics of abnormal data. Therefore, in actual use, it can analyze each data point one by one and determine whether it meets the abnormal characteristics. The system automatically identifies each data point in the target data set through the model and filters out data that meets the abnormal pattern. Specifically, if the characteristics of a data point are similar to the pattern in the historical abnormal data, the model will mark the data point as the fourth abnormal data. This recognition method can capture complex abnormal patterns that are difficult to detect with rules, provide more accurate results for abnormal monitoring, and thus assist in timely detection of potential faults or abnormal conditions.

[0121] Step S70, generating an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data, to complete abnormal data identification.

[0122] It should be noted that the abnormal data identification report refers to a report that records in detail all the abnormal data identified. The report summarizes the abnormal data categories obtained based on different identification methods, including the first, second, third and fourth abnormal data. The report content usually includes key information such as the specific number of each abnormal data, the distribution location of the abnormal data points, the timestamp and the degree of deviation from the normal data. In addition, the report will summarize the characteristics of each abnormal type, analyze the possible causes and abnormal patterns of the abnormality, and provide a reference for further fault analysis and processing. Generating this report can comprehensively display all abnormal data found by the system in each identification step, thereby helping operation and maintenance and management personnel to quickly understand the current system status and take corresponding preventive and corrective measures.

[0123] It is understandable that the system first summarizes the abnormal data identified by different methods, including the first abnormal data (based on threshold judgment), the second abnormal data (based on standard deviation score), the third abnormal data (data not within the normal data range) and the fourth abnormal data (data identified by the abnormal data identification model). Then, the system classifies and counts these abnormal data, and analyzes the quantity, time distribution and degree of deviation from normal data of each abnormal data. Next, an abnormal data identification report is generated, which lists in detail the characteristics and potential causes of each abnormal data, and provides visual charts to facilitate intuitive understanding of the abnormal situation. The report may also include comparative analysis, trend analysis and recommended measures to guide subsequent troubleshooting and system optimization. Through this report, operation and maintenance and management personnel can fully understand the current abnormal state of the system, effectively formulate corresponding response strategies, and ensure the normal operation and stability of the system.

[0124] As an example, the abnormal data identification report is generated based on the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data, and the step of completing the abnormal data identification includes: taking the overlapping data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as high-risk abnormal data; taking the overlapping data of any two of the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as medium-risk abnormal data; taking the data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data except the high-risk abnormal data and the medium-risk abnormal data as low-risk abnormal data; and generating the abnormal data identification report based on the high-risk abnormal data, the medium-risk abnormal data and the low-risk abnormal data.

[0125] Overlapping data refers to data points that appear simultaneously in the first, second, third, and fourth abnormal data, which means that a data point meets multiple abnormality determination criteria and shows the multiplicity of its abnormal characteristics. High-risk abnormal data refers to data points that overlap in the first, second, third, and fourth abnormal data, indicating that they are identified as abnormal under multiple determination criteria, and therefore represent a higher potential risk and may indicate a more serious fault or problem. Medium-risk abnormal data refers to data points that overlap in any two of the first, second, third, and fourth abnormal data. Although these data are not as significant as high-risk abnormal data, they still show abnormal characteristics, which means that there is a certain risk and deserves attention. Low-risk abnormal data refers to data points that do not belong to high-risk and medium-risk abnormal data. These data may only meet one of the abnormal identification criteria, representing a lower risk level. Although they also need to be monitored, they have a lower priority than high-risk and medium-risk abnormal data.

[0126] The system first performs overlapping analysis on the first, second, third, and fourth abnormal data previously identified to identify abnormal data of different risk levels. Specifically, the system marks those data points that appear in all four categories of abnormal data as high-risk abnormal data, because these data are identified as abnormal under multiple criteria, indicating that their potential risk is high. Next, the system marks those data points that only appear in any two categories of abnormal data as medium-risk abnormal data, indicating that although these data do not overlap in all categories, they still show certain abnormal characteristics. Finally, the system marks the remaining data points, that is, data that do not belong to the high-risk and medium-risk categories, as low-risk abnormal data, which may only appear in one category of abnormality and have relatively low risks. Subsequently, the system will generate an abnormal data identification report based on these three categories of abnormal data (high risk, medium risk, and low risk), which records in detail the number, characteristics, and possible risk levels of each type of abnormal data, providing an important basis for subsequent analysis, decision-making, and maintenance. This process ensures a comprehensive assessment of abnormal situations, allowing the operation and maintenance team to prioritize high-risk issues, thereby improving the stability and security of the system.

[0127] This embodiment provides an abnormal data identification method, which obtains target data, time series data and historical data sets; obtains an upper threshold, a lower threshold, a standard deviation score, a normal data range and a target abnormal data identification model according to at least one of the target data, the time series data and the historical data sets; takes the data in the target data that is greater than the upper threshold or less than the lower threshold as the first abnormal data; takes the data in the target data that has a standard deviation score greater than a preset standard deviation score threshold as the second abnormal data; takes the data in the target data that is not within the normal data range as the third abnormal data; identifies the target data through the target abnormal data identification model to obtain the fourth abnormal data; generates an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data, and completes the abnormal data identification. First, the target data, time series data and historical data sets are obtained to ensure the comprehensiveness and diversity of the data and provide a basis for subsequent analysis. These data are used to calculate the upper threshold, the lower threshold, the standard deviation score and the normal data range, and train the target abnormal data identification model. These indicators and models provide accurate standards and basis for determining abnormalities and enhance the reliability of identification. Next, the system marks the data in the target data that is greater than the upper threshold or less than the lower threshold as the first abnormal data. This process can quickly identify significantly deviated anomalies and help operation and maintenance personnel to detect potential faults in a timely manner. Subsequently, by calculating the standard deviation score and screening out data points greater than the preset standard deviation score threshold, they are marked as the second abnormal data to ensure that those statistically significant complex anomalies can be captured. After that, the system identifies the data points that are not within the normal range by comparing with the normal data range and marks them as the third abnormal data to confirm relatively small but still important abnormal situations. The trained target abnormal data recognition model is then used to identify the target data, and the fourth abnormal data that meets the model characteristics is automatically detected to improve the efficiency of processing large-scale data and to detect complex abnormal patterns in a timely manner. Finally, all identified abnormal data are summarized to generate a detailed abnormal data recognition report, recording the characteristics and potential causes of various types of anomalies, providing an important basis for subsequent decision-making and maintenance. Through this series of steps, the abnormal data recognition system effectively improves the accuracy and efficiency of anomaly detection, can accurately identify abnormal data in industrial data, ensure the stable operation of industry and equipment, and reduce the risk of losses caused by anomalies.

[0128] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , Figure 2This is a flow chart of the second embodiment of the abnormal data identification method of the present application. Step S20 of the abnormal data identification method includes steps S21 to S26:

[0129] Step S21, calculating an upper threshold and a lower threshold according to the target data, the time series data and the historical data set;

[0130] It is understandable that the target data (such as industrial sensor data or equipment performance data), time series data (arrange the target data in chronological order), and historical data sets (including labeled normal and abnormal data) are first collected to ensure the comprehensiveness and accuracy of the analysis. Then, based on the trend and seasonal changes of the time series data, combined with the characteristics of the historical data set, the upper and lower thresholds are dynamically calculated. This process may involve analyzing the fluctuations of the data in different time periods, identifying the normal range within a specific season or cycle, and then adjusting the thresholds. For example, different upper and lower thresholds may be set for production changes in winter and summer, respectively, so as to more accurately reflect the normal operating status in different time periods. In this way, it is possible to adapt to data changes more flexibly, ensure the rationality and effectiveness of the thresholds, and thus provide a solid foundation for subsequent abnormal data identification, effectively improve the accuracy of detection, discover potential problems in a timely manner, and ensure the stable operation of the system.

[0131] As an example, the step of calculating the upper threshold and the lower threshold based on the target data, the time series data and the historical data set includes: calculating the mean value and the standard deviation based on the historical data set; smoothing and seasonally decomposing the time series data to obtain the seasonal component; setting the initial threshold based on the mean value and the standard deviation; analyzing the seasonal component to obtain the seasonal variation cycle; adjusting the initial threshold based on the seasonal variation cycle to obtain the reference threshold; and calculating the upper threshold and the lower threshold based on the reference threshold and the historical data within a preset time window.

[0132] The mean is the sum of the target data in the historical data set and divided by the number of data points, reflecting the central position of the data; the standard deviation measures the degree of deviation of the data from the mean and provides a quantitative indicator of data volatility. Smoothing is the process of revealing long-term trends by removing noise and short-term fluctuations in the data. Common methods include moving average, weighted average, etc. Smoothed data helps to more clearly identify trends and seasonal changes in the data. Seasonal decomposition is the decomposition of time series data into several components, including trend components, seasonal components, and random components, in order to better understand the change pattern of the data. Through this process, seasonal fluctuations and long-term trends can be analyzed separately. The seasonal component is the part of the time series that shows periodic changes, usually related to seasons, months, or specific time periods. For example, some production data may show different patterns in summer and winter. The initial threshold is a preliminary limit set after calculating the mean and standard deviation. It is used to preliminarily judge the normal range of the data. It can usually be set to the mean plus or minus a certain standard deviation multiple. The seasonal variation cycle refers to the cycle length of the recurring seasonal pattern in the data, such as quarterly changes in a year or cyclical changes within a month. These cycles reflect the cyclical fluctuations of the data over time. The reference threshold is a more accurate threshold that is adjusted based on the initial threshold according to the analysis results of the seasonal component and the seasonal change cycle. This threshold can better reflect the normal data range in different time periods. The preset time window refers to the time period length of the historical data selected when calculating the upper and lower thresholds, such as the past month, three months or one year. The data in this window is used to ensure that the calculation of the threshold reflects the trend and changes in the time period.

[0133] First, extract the target data from the historical data set to calculate its mean and standard deviation, which provides a basis for determining the central trend and fluctuation range of the data. Next, smooth the time series data, use techniques such as moving average to remove short-term fluctuations, and then perform seasonal decomposition to separate the trend component, seasonal component, and random component in the data, so as to clarify the change pattern in different time periods, especially the performance of seasonal components. Based on the previously calculated mean and standard deviation, the initial threshold is preliminarily set to provide a reference standard for the normal range of the data. Subsequently, by analyzing the seasonal components, the seasonal change cycle is identified to understand the law of the data in different seasons or time periods. Then, the initial threshold is adjusted according to these cycles to obtain a reference threshold that is more in line with the actual situation, so that it can better adapt to the seasonal fluctuations of the data. Finally, by combining the reference threshold with the historical data in the preset time window, the final upper and lower thresholds are calculated to ensure that these thresholds can effectively distinguish normal data from potential abnormal data, thus laying a solid foundation for subsequent abnormal data identification.

[0134] Step S22, calculating a standard deviation score according to the target data;

[0135] It is understandable that the target data needs to be acquired first, usually industrial sensor data or equipment performance data collected within a specific time period. Next, the mean of the target data is calculated to determine the central trend of the data. Then, the standard deviation of the target data set is calculated to reflect the volatility of the data. The standard deviation score is calculated by subtracting the mean from each data point and then dividing it by the standard deviation to obtain a standardized value indicating the degree of deviation of the data point from the mean. This calculation can not only reveal the degree of abnormality of the data point, but also compare the volatility between different data sets to determine which data points deviate significantly from the normal range. Through this standardized processing, potential abnormal data can be identified more effectively, and a quantitative basis can be provided for subsequent anomaly detection to ensure the stability and security of system operation.

[0136] Step S23, sorting the target data according to the numerical value to obtain ordered data;

[0137] It should be noted that ordered data refers to a data sequence formed by arranging the target data according to its numerical value. Specifically, this means comparing all data points in the target data set and arranging them in order from small to large or from large to small. For example, if the target data includes a set of numerical values ​​(such as sensor measurements or equipment operating parameters), after sorting, the ordered data obtained will be a new sequence in which the numerical values ​​are arranged in a specific order (usually ascending or descending). This process helps to more clearly identify the distribution characteristics of the data, find extreme values, and conduct more in-depth statistical analysis in subsequent data analysis, such as calculating quantiles and identifying outliers. Sorting can provide a clear foundation for subsequent anomaly detection and data processing, ensuring the accuracy and effectiveness of the analysis.

[0138] It can be understood that, first, each data point in the target data set (for example, the value collected by industrial sensors or equipment performance indicators) is extracted. Then, by comparing the numerical relationship between the data points, they are rearranged into a new sequence, and usually they can be sorted in ascending order (from small to large) or descending order (from large to small). When implementing the sorting, different algorithms, such as quick sort, merge sort, etc., can be used to ensure the efficiency and accuracy of the sorting process. The obtained ordered data sequence can clearly show the overall distribution of the data, which is convenient for subsequent analysis. For example, the sorted data can help identify extreme values ​​or outliers in the data, calculate quartiles, and then provide a basis for subsequent statistical analysis, threshold setting, and abnormal data detection. In addition, the structured characteristics of ordered data also lay the foundation for visual display (such as drawing box plots or histograms), which helps to further understand the law of data change.

[0139] Step S24, calculating the first quartile and the second quartile according to the ordered data;

[0140] It should be noted that the first quartile is the first boundary that divides the ordered data set into four parts. It represents the data points in the first 25% (i.e., the 25th percentile) of the data. When calculating specifically, the ordered data is arranged by numerical value, and then the 25th position in the data set is found. If the total number of data points is N, the position corresponding to Q1 is N multiplied by 0.25. If the position is an integer, the value of Q1 is the data point at that position; if the position is not an integer, the value of Q1 is the average of the data point at that position and the next data point. Q1 can reflect the lower value range in the data set and provide early insights into the distribution of the data. The second quartile refers to the data points in the first 75% (i.e., the 75th percentile) of the data. The calculation method is similar to Q1, dividing the ordered data set into 75% positions. If N is the total number of data points, the position corresponding to Q3 is N multiplied by 0.75. Similarly, if the position is an integer, the value of Q3 is the data point at that position; if it is not an integer, it is the average of the data point at that position and the next data point. Q3 reflects the higher value range of the data set and provides further information about the distribution of the data.

[0141] It can be understood that, first, data is extracted from the sorted data set, and the total number of data N is determined. When calculating Q1, the position of Q1 is determined by the formula N×0.25. If the position is an integer, the value at the position is directly taken as Q1; if it is not an integer, linear interpolation is performed between the value at the position and the value at the next position to calculate the average value. Next, Q3 is calculated in a similar way: the position of Q3 is determined by the formula N×0.75, and it is also determined whether it is an integer and the corresponding processing method is adopted. The calculated Q1 and Q3 represent the 25th and 75th percentile data points in the data distribution, respectively, which helps to understand the upper and lower quartile intervals of the data. The calculation results of these two quartiles not only provide key thresholds for subsequent abnormal data detection, but can also be used to describe the central tendency and dispersion of the data, thereby helping to analyze the morphology and characteristics of the data distribution.

[0142] Step S25, calculating a normal data range according to the first quartile and the second quartile;

[0143] It should be noted that the normal data range refers to the data value interval calculated based on the first quartile (Q1) and the second quartile (Q3), which is usually used to identify and judge the normal and abnormal states of data.

[0144] It can be understood that, first, the interquartile range (IQR) is calculated using Q1 and Q3, and the formula is IQR = Q3-Q1. This value reflects the dispersion of the middle 50% of the data and can help identify the central tendency of the data. Next, the upper and lower limits of the normal data range are set according to the IQR, and the following formula is usually used: lower limit = Q1-1.5×IQR, upper limit = Q3+1.5×IQR. Among them, 1.5 is an empirical value used to identify potential outliers in the data distribution. Ultimately, the normal data range obtained refers to the data values ​​between the upper and lower limits. The establishment of this range helps the subsequent anomaly detection process, so that data points that exceed the normal range can be effectively identified and excluded, thereby ensuring the accuracy of data analysis and the safety of industrial equipment. By defining the normal data range, an important reference basis can be provided for data monitoring and analysis.

[0145] Step S26, constructing a target abnormal data recognition model based on the historical data set.

[0146] It is understandable that, first of all, it is necessary to extract labeled normal and abnormal data from historical data sets, which will serve as the basis for model training. Then, select appropriate machine learning algorithms, such as support vector machines, to model according to the characteristics and distribution of the data. During the model training process, by inputting normal and abnormal data, the algorithm will learn the difference characteristics between the data so that it can effectively distinguish when new data arrives. Model training also includes parameter tuning and cross-validation to ensure its performance and generalization ability on different data sets. In addition, performance indicators such as accuracy, precision, recall, and F1-score can be used to evaluate the effect of the model. By building and optimizing the target abnormal data recognition model, it can provide strong support for subsequent real-time data monitoring, help quickly identify potential abnormal situations, and thus improve the safety and reliability of industrial systems. Ultimately, the model will become an important tool for determining whether data is abnormal in practical applications.

[0147] As an example, the step of constructing a target abnormal data identification model based on the historical data set includes: dividing the historical data set into a training set and a test set; determining the number of sub-models, the sub-model depth threshold, and the split node sample number threshold based on the historical data set; constructing an initial abnormal data identification model based on the training set, the number of sub-models, the sub-model depth threshold, and the split node sample number threshold; inputting the test set into the initial abnormal data identification model to obtain a test result; determining an abnormal score threshold based on the test result; and obtaining a target abnormal data identification model based on the initial abnormal data identification model and the abnormal score threshold.

[0148] The training set refers to a part of the data extracted from the historical data set, usually accounting for 70% to 80% of the total data. It is used to train the model to learn the patterns and features between normal and abnormal data. The test set refers to the remaining part extracted from the historical data set, usually accounting for 20% to 30% of the total data. It is used to evaluate the performance of the model to test its generalization ability on unseen data. The number of sub-models refers to the number of basic models used when building an integrated model, such as the number of decision trees used in a random forest. More sub-models usually improve the stability and accuracy of the model. The sub-model depth threshold refers to the maximum depth of the tree when building each sub-model. Too large a depth may lead to overfitting, and too small a depth may lead to underfitting, so it needs to be set reasonably. The split node sample number threshold refers to the minimum number of samples required to determine the split at the node of the tree. Setting a reasonable threshold can avoid overfitting of small samples, thereby improving the generalization ability of the model. The initial abnormal data recognition model is the first version of the abnormal data recognition model built based on the training set and set parameters (such as the number of sub-models, depth threshold, etc.). After training, this model can perform preliminary abnormal recognition of data, but it still needs to be verified and optimized. The test result is the output obtained after the test set is input into the initial anomaly data identification model, usually including the anomaly score or category label of each test sample. These results are used to evaluate the accuracy and effectiveness of the model. The anomaly score threshold is a value determined based on the test results, which is used to divide the boundary between normal data and abnormal data. If the anomaly score of a data point is higher than this threshold, it is judged as abnormal; otherwise, it is considered normal. The setting of this threshold helps to improve the reliability of the model in practical applications.

[0149] First, the historical data set is divided into a training set and a test set to ensure that the model can be effectively evaluated on unseen data. The data is usually divided into 70% for training and 30% for testing. Next, according to the characteristics of the historical data set, the number of sub-models, the sub-model depth threshold (maximum tree depth) and the split node sample number threshold (minimum node sample number) are determined. These parameters will affect the complexity and performance of the model. Appropriate settings can avoid overfitting or underfitting. Then, the initial abnormal data recognition model is constructed using the training set and the aforementioned parameters. This model will learn the patterns and features in the data during the training process. Next, the test set is input into the initial model to obtain test results. These results will reflect the performance of the model on new data, including the anomaly score or classification of each sample. According to the test results, the anomaly score threshold is further determined. The threshold is used to distinguish between normal and abnormal data, thereby enhancing the judgment ability of the model. Finally, the initial abnormal data recognition model and the determined anomaly score threshold are combined to generate the final target abnormal data recognition model, which has the ability to perform real-time anomaly recognition in practical applications. This series of steps ensures the scientificity and accuracy of the model, enabling it to effectively monitor and identify abnormal data.

[0150] As an example, the step of identifying the target data through the target abnormal data identification model to obtain the fourth abnormal data includes: identifying the target data through the target abnormal data identification model to obtain an abnormality score; and taking the data in the target data whose abnormality score is greater than or equal to the abnormality score threshold as the fourth abnormal data.

[0151] The anomaly score is an indicator calculated by the model for each data point, reflecting the degree of deviation of the data point from normal data. Specifically, the anomaly score is obtained by analyzing the patterns and features learned by the target data during the model training process. Generally, the larger the value, the more likely the data point is to be considered anomaly. The model uses the decision boundary obtained by training normal and abnormal data in the historical data set to evaluate the anomaly score of the target data, and ultimately to distinguish normal data from abnormal data.

[0152] First, the model will be used to analyze each data point in the target data and calculate its anomaly score, which represents the degree of deviation of the data point from the normal data pattern learned during the training process. The calculation of the anomaly score may be based on multiple factors, such as the deviation of data features, the similarity between samples, etc. The higher the value, the more likely the data point is to be an anomaly. Next, define an anomaly score threshold, which is obtained through the test set results during model training and is usually used to distinguish between normal and abnormal data. Subsequently, all data points in the target data with anomaly scores greater than or equal to this threshold are screened out and marked as the fourth anomaly data. The ultimate goal of this process is to effectively identify and classify abnormal data that may affect system performance or data quality, provide a basis for subsequent analysis or processing, and ensure the stability and reliability of the system.

[0153] As an example, the step of constructing an initial abnormal data recognition model based on the training set, the number of sub-models, the sub-model depth threshold and the split node sample number threshold includes: sampling the training set with replacement to obtain out-of-bag data and multiple sample subsets; obtaining a sub-model based on the sample subset, the number of sub-models, the sub-model depth threshold and the split node sample number threshold; training the sub-model with the sample subset and testing the sub-model with the out-of-bag data to obtain a target sub-model; and constructing an initial abnormal data recognition model based on multiple target sub-models.

[0154] Sampling with replacement means that when samples are randomly selected from the training set, the selected samples can be selected again in the next selection. This process allows some samples to appear multiple times in the sampling, while some samples may not be selected, thereby providing diversity for model training. Out-of-bag data refers to samples that are not selected in the sampling process with replacement, which can be used as a test set to evaluate the performance of the model. Multiple sample subsets refer to different data subsets generated in each sampling with replacement, which will be used to train different sub-models. Sub-model refers to the model trained based on each sample subset, and each sub-model may differ in structure and parameters to capture different features and patterns in the training data. The target sub-model refers to the verified and effective sub-model obtained by training with sample subsets during the training process and testing on out-of-bag data. The performance of these models will be used to build the final abnormal data recognition model.

[0155] First, the training set is sampled with replacement, which means that when randomly selecting samples from the training set, the selected samples can be selected again in subsequent sampling, thereby generating multiple sample subsets and obtaining unselected out-of-bag data at the same time. These out-of-bag data can be used for model testing. Next, multiple sub-models are constructed based on the generated sample subsets, the number of sub-models determined according to the historical data set, the sub-model depth threshold (limiting the maximum depth of the model), and the split node sample number threshold (limiting the minimum number of samples required on each split node). Subsequently, these sub-models are trained using sample subsets and tested using out-of-bag data to evaluate the performance of each sub-model, thereby selecting effective target sub-models. Finally, these target sub-models are combined to construct an initial abnormal data recognition model through an integrated method. This model can more comprehensively capture the abnormal features in the data, thereby improving the accuracy and robustness of anomaly detection.

[0156] As an example, the step of obtaining a submodel based on the sample subset, the number of submodels, the submodel depth threshold and the split node sample number threshold includes: determining the split feature based on the information gain, Gini index and mean square error according to the sample subset; creating multiple submodel nodes according to the split feature; obtaining the submodel depth and split subset according to the submodel node; when the submodel depth is less than the submodel depth threshold and the split subset is greater than the split node sample number threshold, taking the split subset as the sample subset and returning the step of determining the split feature based on the information gain, Gini index and mean square error according to the sample subset; when the submodel depth is less than the submodel depth threshold and the split subset is greater than the split node sample number threshold, taking the submodel node as a leaf node to obtain a submodel.

[0157] Information gain refers to the degree to which the uncertainty of information is reduced after data is segmented by a certain feature. It is used to evaluate the importance of a feature in a classification task. The greater the information gain, the more helpful the feature is in improving the classification effect. The Gini index is a method to measure data impurity and is used to evaluate the quality of split features in decision tree algorithms. The lower the Gini index, the higher the purity of the data set, which is suitable for selecting the best split feature. The mean square error is an indicator used to measure the degree to which data points deviate from the mean. It is usually used in regression problems. The smaller its value, the more concentrated the distribution of data points around the target value, which is suitable for evaluating the split quality of regression trees. Split features refer to features used at nodes to divide a data set into multiple subsets. The goal of selecting split features is to maximize information gain, reduce the Gini index or mean square error, and thus improve the purity of the subsets. Specifically, split features help determine the optimal split point at the current node by evaluating the impact of each feature on the target variable. This process usually involves calculating the classification results under different feature values ​​and selecting the feature that makes the best classification effect for data segmentation. Split features are crucial in the construction of decision trees because they directly affect the predictive performance and complexity of the model. A sub-model node refers to an intermediate node or leaf node in a decision tree, representing a certain feature value or category of the data. During the model building process, multiple sub-model nodes are generated by continuously splitting features. The depth of a sub-model refers to the length of the path from the root node to the current node. The greater the depth, the more complex the tree. By limiting the depth, the complexity of the model can be controlled to prevent overfitting. A split subset refers to a sub-dataset obtained by splitting a feature on the current node, which contains data points that meet the feature conditions. A sample subset refers to a small data set extracted from the training set during the sampling with replacement process, which contains data points used to train a specific sub-model. A leaf node is a terminal node in a decision tree, representing the final classification result or predicted value. During the training process, the leaf node no longer splits and directly outputs the prediction result of the model.

[0158] First, using the sample subset, the best split features are determined by calculating the information gain, Gini index or mean square error. These features are used to effectively classify the data at the current node. Then, based on the selected split features, multiple sub-model nodes are created to form the preliminary structure of the sub-model. At this point, it is necessary to evaluate the depth of the sub-model and the corresponding split subset. If the depth of the sub-model has not reached the preset depth threshold, and the number of samples in the split subset is greater than the minimum sample number threshold, the split subset is used as a new sample subset and returns to the step of re-evaluating the split features to continue deepening the sub-model structure. On the contrary, if the sub-model depth is less than the depth threshold and the number of samples in the split subset is less than the minimum threshold, the current node is determined as a leaf node, thereby completing the construction of the sub-model. In this way, multiple sub-models can be generated to form the final abnormal data recognition model, thereby improving the accuracy and robustness of abnormal data recognition.

[0159] As an example, the test set includes data labels, and the step of determining the anomaly score threshold based on the test results includes: calculating the true positive rate and the false positive rate based on the test results and the data labels; drawing an operating characteristic curve based on the true positive rate and the false positive rate; and determining the anomaly score threshold based on the operating characteristic curve.

[0160] Data labels refer to the true category used to identify each data point in the test set, usually including "normal" or "abnormal" labels, so that they can be compared with the predicted results when the model is tested to evaluate the accuracy of the model. The true positive rate (TPR) refers to the proportion of all data points that are actually positive (i.e., true abnormal data) that are correctly identified as positive. The calculation formula is TPR = TP / (TP+FN), where TP (True Positives) is the true positive example and FN (False Negatives) is the false negative example. The false positive rate (FPR) refers to the proportion of all data points that are actually negative (i.e., true normal data) that are incorrectly identified as positive. The calculation formula is FPR = FP / (FP+TN), where FP (False Positives) is a false positive example and TN (True Negatives) is a true negative example. The operating characteristic curve (ROC curve) is a graphical tool used to evaluate the performance of a binary classification model, which depicts the relationship between the true positive rate and the false positive rate. The ROC curve plots different TPR and FPR values ​​by changing the decision threshold. The area under the curve (AUC) can be used to compare the effects of different models. The closer the AUC value is to 1, the better the model performance. In this step, by analyzing the ROC curve, an appropriate anomaly score threshold can be selected to achieve ideal classification performance.

[0161] First, the predictions of the test set are compared with the actual data labels to calculate the true positive rate (TPR) and false positive rate (FPR). The true positive rate (TPR) is calculated by the formula TPR = TP / (TP+FN), where TP is the true positive (data correctly predicted by the model as anomaly) and FN is the false negative (real anomaly data is predicted as normal data). The false positive rate (FPR) is calculated by the formula FPR = FP / (FP+TN), where FP is the false positive (real normal data is predicted as anomaly) and TN is the true negative (data correctly predicted by the model as normal). These calculated TPR and FPR are then plotted into a receiver operating characteristic curve (ROC curve), which helps visualize the performance of the model by showing the changes in TPR and FPR under different thresholds. The process of selecting anomaly score threshold involves identifying the best cut point under the ROC curve, which is usually the location where the best balance between TPR and FPR is achieved, that is, you want to maximize the true positive rate while minimizing the false positive rate. A common approach is to determine the threshold by selecting the point closest to the upper left corner of the curve, which ensures high sensitivity and reduces the false alarm rate. The threshold selected in this way can effectively improve the model's ability to identify abnormal data and optimize the classification performance, making the final model more accurate and reliable in practical applications.

[0162] As an example, the abnormal data identification report is generated based on the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data, and after completing the abnormal data identification step, it also includes: a risk level distribution chart created by a preset visualization tool to display the number of abnormal data at different risk levels in the abnormal data identification report; a time series chart created by the preset visualization tool to display the abnormal data trend in the abnormal data identification report in chronological order and color it according to the risk level; a stacked area chart or facet chart created by the preset visualization tool to display the abnormal data in the abnormal data identification report according to the risk level and the chronological order, respectively.

[0163] Preset visualization tools refer to software tools or libraries that are pre-selected during the data analysis and visualization process. These tools provide a variety of graphical display functions that can effectively present the characteristics and patterns of data. Common ones include Matplotlib, Seaborn, Tableau, etc. The risk level distribution chart is a chart used to show the number distribution of abnormal data at different risk levels (such as high risk, medium risk and low risk). It is usually presented in the form of a bar chart or a pie chart to help users quickly understand the proportion and distribution of various types of abnormal data. Risk level refers to the standard for classifying abnormal data according to the characteristics and historical performance of the data during the abnormal data identification process. It is usually divided into high risk, medium risk and low risk for easy identification and processing. The number of abnormal data refers to the total number of abnormal data at each risk level counted in the abnormal data identification report. This number can help assess the severity of the risk and the necessary countermeasures. Time series chart refers to a chart that shows data changes in chronological order. It is usually used to show trends over time and can clearly reflect the fluctuations of data. Abnormal data trend refers to the change pattern of abnormal data in a specific time period. It may appear as an increase, decrease or fluctuation. Trend analysis can understand the frequency and severity of abnormal data. Coloring refers to using different colors to classify or highlight data in a chart. It is usually used to distinguish different risk levels and make the chart more intuitive and easy to understand. A stacked area chart is a chart that displays the total amount of data and its composition ratio by superimposing the areas of different data series in the same coordinate system. It is often used to observe the changes in multiple types of data over a period of time. A faceted chart refers to a chart that slices and displays the same data set according to one or more dimensions. It uses multiple sub-charts to present the distribution of data under different conditions, helping to deeply analyze the characteristics of various types of abnormal data. Abnormal data refers to data that is identified during the analysis process and does not conform to normal patterns or expectations. This data may appear due to errors, failures, or abnormal events, and usually requires further investigation and processing.

[0164] First, by creating a risk level distribution chart, the number of abnormal data at different risk levels (such as high risk, medium risk and low risk) is displayed, so that users can identify the distribution of various types of abnormal data at a glance. This kind of chart is usually presented in the form of a bar chart or a pie chart, which is convenient for comparing and understanding the relative severity of each risk level. Next, a time series chart is used to display the trend of abnormal data in chronological order, and the data is colored to distinguish different risk levels, which helps users observe the change pattern of abnormal data at different time points and reveal potential time correlations and fluctuation trends. Finally, the abnormal data is further decomposed by stacked area charts or faceted charts, and displayed by risk level and chronological order, so that users can deeply analyze the dynamic changes of different types of abnormal data. The application of these visualization tools not only improves the intuitiveness and readability of the report, but also helps decision makers quickly identify and respond to potential risks.

[0165] As an example, after the step of displaying the abnormal data identification report according to risk level and time sequence respectively through a preset visualization tool, it also includes: obtaining user feedback information, and obtaining false alarm data and missed alarm data according to the user feedback information; adjusting the number of sub-models, the sub-model depth threshold, and the split node sample number threshold according to the false alarm data and the missed alarm data to obtain the target number of sub-models, the target sub-model depth threshold, and the target split node sample number threshold; and reconstructing the target abnormal data identification model according to the historical data set, the target number of sub-models, the target sub-model depth threshold, and the target split node sample number threshold.

[0166] User feedback information refers to the evaluation and opinions of users on the abnormal data identification report, which may include what users consider to be false positives (normal data is incorrectly identified as abnormal data) and false negatives (data that is actually abnormal is not identified). False positives refer to normal data that is incorrectly marked as abnormal in abnormal data identification, while false negatives refer to data that should have been identified as abnormal but was omitted during the identification process. By analyzing these false positives and false negatives, the shortcomings of the existing model in a specific context can be found. The target number of sub-models, target sub-model depth threshold, and target split node sample number threshold refer to the parameter settings required to adjust the optimization of the model after analyzing user feedback. The target number of sub-models refers to the number of sub-models that will be used when rebuilding the model, the target sub-model depth threshold refers to the maximum allowed depth of the sub-model in the new model to avoid overfitting, and the target split node sample number threshold refers to the minimum number of samples required for each split node, which helps to improve the stability and generalization ability of the model. By adjusting these parameters based on user feedback, the accuracy and effectiveness of the target abnormal data identification model can be improved.

[0167] After completing the steps of displaying the abnormal data identification report by risk level and time order through the preset visualization tool, the next process is to obtain user feedback information, which can reflect the user's satisfaction with the identification results and potential problems. Based on the user's feedback, false positive data and missed data can be identified, where false positive data refers to normal data that is incorrectly marked as abnormal, and missed data is data that should be identified as abnormal but is not identified. By analyzing these false positives and missed data, the deficiencies of the model can be determined. Next, the key parameters of the model are adjusted according to the false positive and missed data, including the number of sub-models, the sub-model depth threshold, and the split node sample number threshold, to optimize the performance of the model, and finally obtain the new target number of sub-models, target sub-model depth threshold, and target split node sample number threshold. Finally, based on the adjusted parameters, historical data sets, and new sub-model settings, the target abnormal data identification model is rebuilt to improve the accuracy and recognition ability of the model and ensure that abnormal data can be identified more effectively.

[0168] This embodiment calculates an upper threshold and a lower threshold according to the target data, the time series data, and the historical data set; calculates a standard deviation score according to the target data; sorts the target data by numerical value to obtain ordered data; calculates the first quartile and the second quartile according to the ordered data; calculates the normal data range according to the first quartile and the second quartile; and constructs a target abnormal data recognition model according to the historical data set. First, the upper threshold and the lower threshold are calculated according to the target data, the time series data, and the historical data set. This process provides a dynamic threshold based on historical information by analyzing the changing trend and seasonal fluctuation of the historical data, which helps to accurately identify when the data exceeds the normal range. Next, the standard deviation score is calculated by the target data. This indicator measures the degree of deviation of the data point from the average value, making the identification of potential abnormal data more systematic. The larger the standard deviation score, the more likely the data is abnormal. Subsequently, the target data is sorted by numerical value to generate ordered data, which lays the foundation for subsequent statistical analysis. Through the ordered data, the data distribution can be observed more clearly, and the first quartile (Q1) and the second quartile (Q3) can be calculated more accurately. The first quartile represents 25% of the values ​​in the data, and the second quartile is the median, which reflects the central trend of the data. Using the calculated Q1 and Q3, the normal data range can be further defined, which is usually used to judge the rationality of the data. When the data exceeds this range, it can be preliminarily determined to be abnormal to ensure the effectiveness of the detection process. Finally, a target abnormal data recognition model is constructed based on the historical data set. This step trains the machine learning model so that it can learn to recognize abnormal patterns from historical data, thereby improving the ability to recognize abnormal data in the future. Through these steps, the entire anomaly detection process is not only systematic and scientific, but also can effectively reduce the false alarm rate and improve the accuracy of anomaly detection.

[0169] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the abnormal data identification method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0170] This application also provides an abnormal data identification device, please refer to Figure 3 , the abnormal data identification device comprises:

[0171] The data acquisition module 10 is used to acquire target data, time series data and historical data sets;

[0172] A data processing module 20, configured to obtain an upper threshold, a lower threshold, a standard deviation score, a normal data range, and a target abnormal data identification model according to at least one of the target data, the time series data, and the historical data set;

[0173] A first identification module 30, configured to take data in the target data that is greater than the upper threshold or less than the lower threshold as first abnormal data;

[0174] A second identification module 40 is used to take the data in the target data whose standard deviation score is greater than a preset standard deviation score threshold as second abnormal data;

[0175] A third identification module 50, configured to take data in the target data that is not within the normal data range as third abnormal data;

[0176] A fourth identification module 60, configured to identify the target data by using the target abnormal data identification model to obtain fourth abnormal data;

[0177] The report generating module 70 is used to generate an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data to complete abnormal data identification.

[0178] In one embodiment, the data acquisition module 10 is also used to acquire industrial sensor data, production equipment data, production plan data, system log data, user behavior data and environmental data; fuse the industrial sensor data, the production equipment data, the system log data, the user behavior data and the environmental data to obtain initial data; obtain time series data based on the initial data; convert non-numeric data in the initial data into numerical data to obtain reference data; and standardize the reference data to obtain target data.

[0179] In one embodiment, the data processing module 20 is also used to calculate an upper threshold and a lower threshold based on the target data, the time series data and the historical data set; calculate a standard deviation score based on the target data; sort the target data by numerical value to obtain ordered data; calculate a first quartile and a second quartile based on the ordered data; calculate a normal data range based on the first quartile and the second quartile; and construct a target abnormal data recognition model based on the historical data set.

[0180] In one embodiment, the data processing module 20 is also used to calculate the mean value and standard deviation based on the historical data set; smooth and seasonally decompose the time series data to obtain a seasonal component; set an initial threshold based on the mean value and the standard deviation; analyze the seasonal component to obtain a seasonal variation cycle; adjust the initial threshold based on the seasonal variation cycle to obtain a reference threshold; calculate an upper threshold and a lower threshold based on the reference threshold and the historical data within a preset time window.

[0181] In one embodiment, the data processing module 20 is also used to divide the historical data set into a training set and a test set; determine the number of sub-models, the sub-model depth threshold, and the split node sample number threshold based on the historical data set; construct an initial abnormal data recognition model based on the training set, the number of sub-models, the sub-model depth threshold, and the split node sample number threshold; input the test set into the initial abnormal data recognition model to obtain a test result; determine the abnormal score threshold based on the test result; and obtain a target abnormal data recognition model based on the initial abnormal data recognition model and the abnormal score threshold.

[0182] In one embodiment, the data processing module 20 is also used to perform sampling with replacement on the training set to obtain out-of-bag data and multiple sample subsets; obtain a sub-model based on the sample subset, the number of sub-models, the sub-model depth threshold and the split node sample number threshold; train the sub-model through the sample subset and test the sub-model through the out-of-bag data to obtain a target sub-model; and construct an initial abnormal data recognition model based on multiple target sub-models.

[0183] In one embodiment, the data processing module 20 is also used to determine the split feature based on the sample subset, based on information gain, Gini index, and mean square error; create multiple sub-model nodes based on the split feature; obtain the sub-model depth and split subset based on the sub-model node; when the sub-model depth is less than the sub-model depth threshold and the split subset is greater than the split node sample number threshold, use the split subset as the sample subset and return to the step of determining the split feature based on the sample subset, based on information gain, Gini index, and mean square error; when the sub-model depth is less than the sub-model depth threshold and the split subset is greater than the split node sample number threshold, use the sub-model node as a leaf node to obtain a sub-model.

[0184] In one embodiment, the data processing module 20 is further used to calculate the true positive rate and the false positive rate based on the test results and the data labels; draw an operating characteristic curve based on the true positive rate and the false positive rate; and determine an abnormal score threshold based on the operating characteristic curve.

[0185] In one embodiment, the data processing module 20 is also used to display the number of abnormal data at different risk levels in the abnormal data identification report through a risk level distribution chart created by a preset visualization tool; to display the abnormal data trend in the abnormal data identification report in chronological order through a time series chart created by the preset visualization tool, and to color it according to the risk level; and to display the abnormal data in the abnormal data identification report according to the risk level and the chronological order according to a stacked area chart or a faceted chart created by the preset visualization tool.

[0186] In one embodiment, the data processing module 20 is also used to obtain user feedback information, and obtain false alarm data and missed alarm data based on the user feedback information; adjust the number of sub-models, the sub-model depth threshold, and the split node sample number threshold based on the false alarm data and the missed alarm data to obtain the target number of sub-models, the target sub-model depth threshold, and the target split node sample number threshold; and reconstruct the target abnormal data recognition model based on the historical data set, the target number of sub-models, the target sub-model depth threshold, and the target split node sample number threshold.

[0187] In one embodiment, the fourth identification module 60 is further used to identify the target data through the target abnormal data identification model to obtain an abnormality score; and use the data in the target data whose abnormality score is greater than or equal to the abnormality score threshold as the fourth abnormal data.

[0188] In one embodiment, the report generation module 70 is further used to take the overlapping data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as high-risk abnormal data; take the overlapping data of any two of the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as medium-risk abnormal data; take the data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data other than the high-risk abnormal data and the medium-risk abnormal data as low-risk abnormal data; and generate an abnormal data identification report based on the high-risk abnormal data, the medium-risk abnormal data and the low-risk abnormal data.

[0189] The abnormal data identification device provided by the present application adopts the abnormal data identification method in the above embodiment, which can solve the technical problem of how to accurately identify abnormal data in industrial data. Compared with the prior art, the beneficial effects of the abnormal data identification device provided by the present application are the same as the beneficial effects of the abnormal data identification method provided by the above embodiment, and other technical features in the abnormal data identification device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0190] The present application provides an abnormal data identification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the abnormal data identification method in the above-mentioned embodiment 1.

[0191] Reference below Figure 4 , which shows a schematic diagram of the structure of an abnormal data identification device suitable for implementing an embodiment of the present application. The abnormal data identification device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The abnormal data identification device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0192] like Figure 4As shown, the abnormal data identification device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the abnormal data identification device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the abnormal data identification device to communicate wirelessly or wired with other devices to exchange data. Although the abnormal data identification device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0193] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0194] The abnormal data identification device provided by the present application adopts the abnormal data identification method in the above embodiment, which can solve the technical problem of how to accurately identify abnormal data in industrial data. Compared with the prior art, the beneficial effects of the abnormal data identification device provided by the present application are the same as the beneficial effects of the abnormal data identification method provided by the above embodiment, and the other technical features in the abnormal data identification device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0195] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0197] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the abnormal data identification method in the above-mentioned embodiment.

[0198] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0199] The computer-readable storage medium may be included in the abnormal data identification device; or may exist independently without being assembled into the abnormal data identification device.

[0200] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the abnormal data identification device, the abnormal data identification device: obtains target data, time series data and historical data sets; obtains an upper threshold, a lower threshold, a standard deviation score, a normal data range and a target abnormal data identification model according to at least one of the target data, the time series data and the historical data set; takes data in the target data that is greater than the upper threshold or less than the lower threshold as first abnormal data; takes data in the target data whose standard deviation score is greater than a preset standard deviation score threshold as second abnormal data; takes data in the target data that is not within the normal data range as third abnormal data; identifies the target data through the target abnormal data identification model to obtain fourth abnormal data; generates an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data to complete abnormal data identification.

[0201] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0202] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0203] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0204] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned abnormal data identification method, and can solve the technical problem of how to accurately identify abnormal data in industrial data. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the abnormal data identification method provided by the above-mentioned embodiment, and will not be repeated here.

[0205] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned abnormal data identification method when executed by a processor.

[0206] The computer program product provided by the present application can solve the technical problem of how to accurately identify abnormal data in industrial data. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the abnormal data identification method provided by the above embodiment, which will not be repeated here.

[0207] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for identifying abnormal data, characterized in that: The method comprises: Obtain target data, time series data, and historical data sets; Obtaining an upper threshold, a lower threshold, a standard deviation score, a normal data range, and a target abnormal data identification model according to at least one of the target data, the time series data, and the historical data set; Taking data in the target data that is greater than the upper threshold or less than the lower threshold as first abnormal data; The data in the target data whose standard deviation score is greater than a preset standard deviation score threshold is taken as second abnormal data; taking data in the target data that is not within the normal data range as third abnormal data; Identify the target data using the target abnormal data identification model to obtain fourth abnormal data; An abnormal data identification report is generated according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data to complete abnormal data identification.

2. The method according to claim 1, characterized in that The step of obtaining an upper threshold, a lower threshold, a standard deviation score, a normal data range, and a target abnormal data identification model according to at least one of the target data, the time series data, and the historical data set comprises: Calculate an upper threshold and a lower threshold according to the target data, the time series data and the historical data set; Calculate a standard deviation score based on the target data; Sorting the target data according to numerical values ​​to obtain ordered data; Calculate the first quartile and the second quartile according to the ordered data; Calculate the normal data range according to the first quartile and the second quartile; A target abnormal data recognition model is constructed based on the historical data set.

3. The method according to claim 2, characterized in that The step of calculating the upper threshold and the lower threshold according to the target data, the time series data and the historical data set comprises: Calculate the average value and standard deviation based on the historical data set; Smoothing and seasonal decomposing the time series data to obtain seasonal components; Setting an initial threshold value according to the mean value and the standard deviation; Analyzing the seasonal components to obtain a seasonal variation cycle; Adjust the initial threshold according to the seasonal variation cycle to obtain a reference threshold; An upper threshold and a lower threshold are calculated based on the reference threshold and the historical data within a preset time window.

4. The method according to claim 2, characterized in that The step of constructing a target abnormal data recognition model according to the historical data set comprises: Dividing the historical data set into a training set and a test set; Determine the number of sub-models, the sub-model depth threshold, and the split node sample number threshold according to the historical data set; An initial abnormal data recognition model is constructed according to the training set, the number of sub-models, the sub-model depth threshold and the split node sample number threshold; Inputting the test set into the initial abnormal data recognition model to obtain a test result; determining an anomaly score threshold based on the test results; A target abnormal data recognition model is obtained according to the initial abnormal data recognition model and the abnormal score threshold.

5. The method according to claim 4, characterized in that The step of constructing an initial abnormal data recognition model according to the training set, the number of sub-models, the sub-model depth threshold and the split node sample number threshold comprises: Performing sampling with replacement on the training set to obtain out-of-bag data and multiple sample subsets; Obtain a submodel according to the sample subset, the number of submodels, the submodel depth threshold, and the split node sample number threshold; The sub-model is trained by using the sample subset, and the sub-model is tested by using the out-of-bag data to obtain a target sub-model; An initial abnormal data recognition model is constructed based on the multiple target sub-models.

6. The method according to claim 4, characterized in that The step of obtaining the sub-model according to the sample subset, the number of sub-models, the sub-model depth threshold and the split node sample number threshold comprises: According to the sample subset, determining the splitting feature based on information gain, Gini index, and mean square error; Creating a plurality of sub-model nodes according to the splitting feature; Obtaining a sub-model depth and a split subset according to the sub-model node; When the sub-model depth is less than the sub-model depth threshold and the split subset is greater than the split node sample number threshold, the split subset is used as a sample subset, and the step of determining the split feature based on the information gain, the Gini index, and the mean square error according to the sample subset is returned; When the sub-model depth is less than the sub-model depth threshold and the split subset is greater than the split node sample number threshold, the sub-model node is taken as a leaf node to obtain a sub-model.

7. The method according to claim 4, characterized in that The test set includes data labels, and the step of determining the abnormal score threshold according to the test result includes: Calculate the true positive rate and the false positive rate according to the test results and the data labels; Draw an operating characteristic curve according to the true positive rate and the false positive rate; An abnormality score threshold is determined according to the operating characteristic curve.

8. The method according to claim 4, characterized in that After the step of generating an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data and completing the abnormal data identification, the method further includes: A risk level distribution chart created by a preset visualization tool displays the number of abnormal data at different risk levels in the abnormal data identification report; The time series graph created by the preset visualization tool displays the abnormal data trend in the abnormal data identification report in chronological order and is colored according to the risk level; The stacked area chart or faceted chart created by the preset visualization tool displays the abnormal data in the abnormal data identification report according to the risk level and the time sequence respectively.

9. The method according to claim 8, characterized in that After the step of displaying the abnormal data identification report by risk level and time sequence respectively through a preset visualization tool, the method further includes: Obtaining user feedback information, and obtaining false positive data and missed negative data according to the user feedback information; The number of sub-models, the sub-model depth threshold, and the split node sample number threshold are adjusted according to the false positive data and the missed negative data to obtain a target number of sub-models, a target sub-model depth threshold, and a target split node sample number threshold; The target abnormal data recognition model is reconstructed according to the historical data set, the number of target sub-models, the target sub-model depth threshold and the target split node sample number threshold.

10. The method according to claim 4, characterized in that The step of identifying the target data by using the target abnormal data identification model to obtain fourth abnormal data comprises: Identify the target data using the target abnormal data identification model to obtain an abnormality score; The data in the target data whose abnormality score is greater than or equal to the abnormality score threshold is taken as fourth abnormal data.

11. The method according to claim 1, characterized in that The step of obtaining target data and time series data includes: Obtain industrial sensor data, production equipment data, production plan data, system log data, user behavior data, and environmental data; Fusing the industrial sensor data, the production equipment data, the system log data, the user behavior data, and the environment data to obtain initial data; Obtaining time series data according to the initial data; Converting non-numeric data in the initial data into numeric data to obtain reference data; The reference data is standardized to obtain target data.

12. The method according to claim 1, characterized in that The step of generating an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data, and completing the abnormal data identification comprises: taking overlapping data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as high-risk abnormal data; taking any two overlapping data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as medium-risk abnormal data; taking data other than the high-risk abnormal data and the medium-risk abnormal data among the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data as low-risk abnormal data; An abnormal data identification report is generated according to the high-risk abnormal data, the medium-risk abnormal data, and the low-risk abnormal data.

13. An abnormal data identification device, characterized in that: The device comprises: Data acquisition module, used to acquire target data, time series data and historical data sets; A data processing module, used to obtain an upper threshold, a lower threshold, a standard deviation score, a normal data range, and a target abnormal data identification model according to at least one of the target data, the time series data, and the historical data set; A first identification module, configured to take data in the target data that is greater than the upper threshold or less than the lower threshold as first abnormal data; A second identification module, configured to take the data in the target data whose standard deviation score is greater than a preset standard deviation score threshold as second abnormal data; A third identification module, configured to take data in the target data that is not within the normal data range as third abnormal data; a fourth identification module, configured to identify the target data by using the target abnormal data identification model to obtain fourth abnormal data; A report generating module is used to generate an abnormal data identification report according to the first abnormal data, the second abnormal data, the third abnormal data and the fourth abnormal data to complete abnormal data identification.

14. An abnormal data identification device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the abnormal data identification method according to any one of claims 1 to 12.

15. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the abnormal data identification method according to any one of claims 1 to 12 are implemented.

Images