Abnormal data detection and elimination method and system under complex working conditions

Abstract

The present application relates to the technical field of data transmission, and discloses a method and system for detecting and eliminating abnormal data under complex working conditions, the method comprising: receiving data streams of multiple sensors in a preset complex working condition operation process; extracting features from the data stream of each sensor in the multiple sensors according to multiple preset time points; finding the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as core data points according to the preset neighborhood radius and the minimum number of points, clustering the multiple feature vectors to form multiple feature clusters and noise data points outside the multiple feature clusters; calculating the outlier measure of the noise data point; calculating the probability that the noise data point is an abnormal data point; when the probability that the noise data point is an abnormal data point is higher than a preset threshold, according to the time point corresponding to the abnormal data point, eliminating data from the data streams of the multiple sensors. The present invention effectively improves the data quality and enhances the abnormality detection capability.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and more specifically, to a method and system for detecting and eliminating abnormal data under complex working conditions. Background Art

[0002] In modern industrial production and network data transmission, especially in complex working conditions such as underground mines, the reliability and accuracy of sensor data are crucial for safe production and efficient operation. However, due to the particularity of the underground mine environment, such as unstable signal transmission, electromagnetic interference, equipment aging and other factors, there are often a large number of abnormal data in the data collected by sensors. These abnormal data not only affect the accuracy of data analysis, but may also lead to wrong decisions.

[0003] At present, the existing abnormal data detection methods mainly include statistical methods, machine learning methods, etc. Although these methods can identify abnormal data to a certain extent, under complex working conditions, especially in poor communication environments, traditional abnormal data detection methods often cannot meet the requirements of high accuracy and real-time performance. Therefore, a new technical solution is needed to effectively identify and eliminate abnormal data under complex working conditions and improve the accuracy and reliability of data processing. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed to provide a method and system for detecting and eliminating abnormal data under complex working conditions, which can effectively identify and eliminate abnormal data under complex working conditions and improve the accuracy and reliability of data processing.

[0005] In a first aspect, the present application provides a method for detecting and eliminating abnormal data under complex working conditions, comprising: receiving data streams of multiple sensors in a preset complex working condition operation process; extracting features from the data stream of each of the multiple sensors according to multiple preset time points, wherein the features of the data streams of the multiple sensors at each of the multiple time points constitute a feature vector, and the features of the data streams of the multiple sensors at the multiple time points constitute multiple feature vectors; according to a preset neighborhood radius and a minimum number of points, finding the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as core data points, clustering the multiple feature vectors to form multiple feature clusters and noise data points outside the multiple feature clusters; calculating the outlier measure of the noise data point; calculating the probability that the noise data point is an abnormal data point according to the outlier measure of the noise data point; when the probability that the noise data point is an abnormal data point is higher than a preset threshold, eliminating data from the data streams of the multiple sensors according to the time point corresponding to the abnormal data point.

[0006] Optionally, in the aforementioned method for detecting and eliminating abnormal data under complex working conditions, before finding out the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as core data points according to the preset neighborhood radius and the minimum number of points, it also includes: according to the preset reference radius, counting the number of other feature vectors adjacent to each feature vector in the multiple feature vectors; and calculating the local density of each feature vector according to the number of other feature vectors adjacent to each feature vector, wherein the local density of the kth feature vector is ; Calculate the average local density of the multiple feature vectors ; Calculate the minimum number of points, where the minimum number of points corresponding to the kth eigenvector .

[0007] Optionally, in the aforementioned method for detecting and eliminating abnormal data under complex working conditions, before finding out the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as core data points according to the preset neighborhood radius and the minimum number of points, it also includes: calculating the distance between each feature vector and other feature vectors; according to the minimum number of points corresponding to each feature vector, selecting a typical distance from the distance between each feature vector and other feature vectors in order from near to far, and the typical distance of the kth feature vector is the distance between the kth eigenvector and other eigenvectors. distances; calculating the average typical distance of the plurality of feature vectors ; Calculate the neighborhood radius ,in, is the preset adjustment factor, represents the number of the plurality of feature vectors.

[0008] Optionally, in the aforementioned method for detecting and eliminating abnormal data under complex working conditions, calculating the outlier metric of the noise data point includes: calculating the centers of the multiple feature clusters, wherein the center of the i-th feature cluster is ; Calculate the distance from the noise data point to the center of the multiple feature clusters, where the distance from the noise data point to the center of the i-th feature cluster is , Represents the noise data point; calculates the distance from all feature vectors to the center of the feature cluster where they are located, and takes the average And take the standard deviation ; Calculate the outlier metric of the noise data point , where m is the number of the multiple feature clusters, is the preset weight coefficient.

[0009] Optionally, in the aforementioned method for detecting and eliminating abnormal data under complex working conditions, after calculating the outlier metric of the noise data point, it also includes: querying the previous time point of the time point corresponding to the noise data point; querying the feature vectors of the data streams of the multiple sensors at the previous time point; ; Calculate the fluctuation amplitude of the noise data point ,in, is a preset reference time; based on the fluctuation amplitude of the noise data point , the outlier measure of the noise data point After correction, the outlier measure of the corrected noise data point is ,in, is the preset correction factor.

[0010] Optionally, in the aforementioned method for detecting and eliminating abnormal data under complex working conditions, the probability that the noise data point is an abnormal data point is calculated according to the outlier metric of the noise data point, including: calculating the preliminary abnormal score of the noise data point ,in, and is a preset parameter that controls the initial anomaly score of the noise data point The steepness parameter and the center position parameter of the corresponding curve; Calculate the spatial consistency score of the noise data point ,in, and is a preset parameter that controls the spatial consistency score of the noise data point The steepness parameter and center position parameter of the corresponding curve; Calculate the probability that the noise data point is an abnormal data point .

[0011] Optionally, in the aforementioned method for detecting and eliminating abnormal data under complex working conditions, eliminating data from the data streams of the multiple sensors according to the time points corresponding to the abnormal data points includes: taking the time points corresponding to the abnormal data points as the elimination time points, and selecting a first candidate elimination time point from before the elimination time point and a second candidate elimination time point from after the elimination time point according to a preset time interval; calculating the gap between the feature vector corresponding to the elimination time point and the feature vector corresponding to the first candidate elimination time point, and when the gap exceeds a preset gap value, taking the first candidate elimination time point as a new elimination time point, and continuing to select a new first candidate elimination time point; calculating the gap between the feature vector corresponding to the elimination time point and the feature vector corresponding to the second candidate elimination time point, and when the gap exceeds a preset gap value, taking the second candidate elimination time point as a new elimination time point, and continuing to select a new second candidate elimination time point; eliminating all data between the smallest elimination time point and the largest elimination time point in the data streams of the multiple sensors.

[0012] Optionally, in the aforementioned method for detecting and eliminating abnormal data under complex working conditions, after eliminating all data between the minimum elimination time point and the maximum elimination time point in the data streams of the multiple sensors, it also includes: filling data between the minimum elimination time point and the maximum elimination time point, so that the gap between the feature vectors corresponding to any two points between the minimum elimination time point and the maximum elimination time point does not exceed the preset gap value.

[0013] In a second aspect, the present application provides a system for detecting and eliminating abnormal data under complex working conditions, comprising: a data stream receiving module, receiving data streams of multiple sensors in a preset complex working condition operation process; a feature extraction module, extracting features from the data stream of each of the multiple sensors according to multiple preset time points, wherein the features of the data streams of the multiple sensors at each of the multiple time points constitute a feature vector, and the features of the data streams of the multiple sensors at the multiple time points constitute multiple feature vectors; a noise identification module, according to a preset neighborhood radius and a minimum number of points, finding the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as core data points, clustering the multiple feature vectors to form multiple feature clusters and noise data points outside the multiple feature clusters; an outlier calculation module, calculating the outlier value metric of the noise data point; an abnormality calculation module, calculating the probability that the noise data point is an abnormal data point according to the outlier value metric of the noise data point; a data elimination module, when the probability that the noise data point is an abnormal data point is higher than a preset threshold, according to the time point corresponding to the abnormal data point, eliminates data from the data streams of the multiple sensors.

[0014] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0015] According to the technical solution of the present invention, through the steps of feature extraction, cluster analysis, outlier measurement, probability evaluation, etc., the data quality is effectively improved, the anomaly detection capability is enhanced, the data processing efficiency is improved, the adaptability to the communication environment is improved, the security and reliability of the system are enhanced, and it has wide applicability. These technical effects enable this embodiment to have significant advantages in data processing under complex working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 It is a flow chart of a method for detecting and eliminating abnormal data under complex working conditions according to an embodiment of the present application;

[0018] Figure 2 It is a flow chart of a method for detecting and eliminating abnormal data under another complex working condition according to an embodiment of the present application;

[0019] Figure 3 It is a flowchart of another method for detecting and eliminating abnormal data under complex working conditions according to an embodiment of the present application;

[0020] Figure 4 It is a partial flow chart of the method for detecting and eliminating abnormal data under complex working conditions according to an embodiment of the present application;

[0021] Figure 5 It is a partial flow chart of a method for detecting and eliminating abnormal data under another complex working condition according to an embodiment of the present application;

[0022] Figure 6 It is a block diagram of a system for detecting and eliminating abnormal data under complex working conditions according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] Some embodiments of the present application are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0024] like Figure 1 As shown, an embodiment of the present application provides a method for detecting and eliminating abnormal data under complex working conditions, including:

[0025] Step S110, receiving data streams from multiple sensors during a preset complex working condition operation.

[0026] Step S120, extracting features from the data stream of each sensor among the multiple sensors according to the preset multiple time points, wherein the features of the data streams of the multiple sensors at each time point in the multiple time points constitute a feature vector, and the features of the data streams of the multiple sensors at the multiple time points constitute multiple feature vectors.

[0027] In this embodiment, by extracting features from sensor data to form feature vectors, the essential attributes of the data can be more accurately reflected. This method can filter out irrelevant noise information and retain important feature information, thereby improving the quality of the data.

[0028] Step S130, according to the preset neighborhood radius and minimum number of points, find out the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as the core data point, cluster the multiple feature vectors to form multiple feature clusters and noise data points outside the multiple feature clusters.

[0029] In this embodiment, clustering the feature vectors can effectively identify normal data areas and potential abnormal data areas.

[0030] Step S140, calculating the outlier metric of the noise data point.

[0031] In this embodiment, by calculating the outlier metric of the noise data point, the degree of abnormality of each data point can be quantified. This method can not only identify outliers, but also further evaluate their degree of outliers.

[0032] Step S150, calculating the probability that the noise data point is an abnormal data point according to the outlier metric of the noise data point.

[0033] In this embodiment, the probability of a noise data point being an abnormal data point is calculated by combining the outlier metric and other factors, so as to more accurately determine whether a data point is an abnormal point. This method provides a quantitative evaluation standard and improves the accuracy of anomaly detection.

[0034] Step S160: when the probability that the noise data point is an abnormal data point is higher than a preset threshold, data is removed from the data streams of the multiple sensors according to the time point corresponding to the abnormal data point.

[0035] According to the technical solution of this embodiment, through the steps of feature extraction, cluster analysis, outlier measurement, probability evaluation, etc., the data quality is effectively improved, the anomaly detection capability is enhanced, the data processing efficiency is improved, the adaptability to the communication environment is improved, the security and reliability of the system are enhanced, and it has wide applicability. These technical effects enable this embodiment to have significant advantages in data processing under complex working conditions.

[0036] like Figure 2 As shown, another embodiment of the present application provides a method for detecting and removing abnormal data under complex working conditions. Compared with the above-mentioned embodiment, the method for detecting and removing abnormal data under complex working conditions of this embodiment, before step S130, further includes:

[0037] Step S210: counting the number of other feature vectors adjacent to each feature vector in the plurality of feature vectors according to a preset reference radius.

[0038] Step S220, calculate the local density of each eigenvector according to the number of other eigenvectors adjacent to each eigenvector, where the local density of the kth eigenvector is ; Calculate the average local density of multiple eigenvectors .

[0039] In this embodiment, by counting the number of other eigenvectors adjacent to each eigenvector and calculating the local density of each eigenvector, the local environmental characteristics of each data point can be more carefully characterized, thereby better identifying the importance of the data point and the cluster to which it belongs.

[0040] Step S230, calculating the minimum number of points, where the minimum number of points corresponding to the kth eigenvector is .

[0041] In this embodiment, by calculating the local density mean of the eigenvectors and the minimum number of points corresponding to each eigenvector, key parameters of the clustering algorithm can be dynamically adjusted, so that the clustering process is more adapted to the actual distribution of the data.

[0042] Step S240, calculating the distance between each feature vector and other feature vectors.

[0043] Step S250, based on the minimum number of points corresponding to each eigenvector, select a typical distance from the distance between each eigenvector and other eigenvectors in order from near to far. The typical distance of the kth eigenvector is is the distance between the kth eigenvector and other eigenvectors. distance.

[0044] In this embodiment, by calculating the distance between each eigenvector and all other eigenvectors and selecting typical distances therefrom, the actual distance relationship between data points can be more accurately reflected. This method helps to identify outliers more accurately.

[0045] Step S260, calculating the average typical distance of multiple feature vectors .

[0046] Step S270, calculate the neighborhood radius ,in, is a preset adjustment factor, and n represents the number of multiple feature vectors.

[0047] In this embodiment, the clustering process is made more accurate by calculating the neighborhood radius. This method can better identify core data points and reduce the possibility of misjudgment. By introducing a preset adjustment factor to optimize the selection of the neighborhood radius, the system can automatically adjust the parameters according to the characteristics of different data sets, thereby improving the adaptability of the system.

[0048] According to the technical solution of this embodiment, by adding steps such as local density calculation of feature vectors, dynamically adjusting the minimum number of points, selecting typical distances, optimizing neighborhood radius, etc., the accuracy, adaptability and robustness of data processing are further improved, thereby providing more effective data detection and elimination means in complex working conditions. These improvements are crucial to ensuring the accuracy and reliability of data.

[0049] like Figure 3 As shown, another embodiment of the present application provides a method for detecting and removing abnormal data under complex working conditions. Compared with the above-mentioned embodiment, the method for detecting and removing abnormal data under complex working conditions of this embodiment, step S140 includes:

[0050] Step S310, calculate the centers of multiple feature clusters, where the center of the i-th feature cluster is .

[0051] In this embodiment, by calculating the center point of each feature cluster, the cluster to which the data point belongs can be determined more accurately, which plays a fundamental role in subsequent anomaly detection.

[0052] Step S320, calculating the distances from the noise data points to the centers of multiple feature clusters, where the distance from the noise data points to the center of the i-th feature cluster is , represents a noisy data point.

[0053] Step S330, calculate the distance from all feature vectors to the center of the feature cluster they are in, and take the average And take the standard deviation .

[0054] Step S340, calculating the outlier metric of the noise data point , where m is the number of feature clusters, is the preset weight coefficient.

[0055] In this embodiment, by calculating the distance between the noise data point and the center of each feature cluster, it is possible to more comprehensively evaluate whether the data point deviates from the normal range, thereby improving the recognition accuracy of the noise data point. By calculating the outlier metric of the noise data point and considering the average distance and standard deviation between the feature clusters, the judgment of the outlier is made more scientific and reasonable.

[0056] Step S350, querying the previous time point of the time point corresponding to the noise data point.

[0057] Step S360: query the feature vectors of the data streams of multiple sensors at the previous time point .

[0058] Step S370, calculating the fluctuation amplitude of the noise data point ,in, The preset base time.

[0059] In this embodiment, by querying the data of the previous time point corresponding to the time point of the noise data point and calculating the fluctuation amplitude, the trend of data changes over time can be effectively captured, which is particularly important for identifying instantaneous noise.

[0060] Step S380, based on the fluctuation amplitude of the noise data point , an outlier measure for noisy data points Correction, the outlier measure of the corrected noise data point ,in, is the preset correction factor.

[0061] In this embodiment, the fluctuation amplitude in the time series is incorporated into the calculation of the outlier metric, which can better handle abnormal data that changes rapidly over time.

[0062] According to the technical solution of this embodiment, by introducing steps such as feature cluster center calculation, distance calculation from noise data points to multiple feature cluster centers, fluctuation amplitude calculation, and outlier metric correction, the accuracy of noise data point identification and the robustness of system processing are further improved, ensuring the efficiency and reliability of data processing under complex working conditions.

[0063] like Figure 4 As shown, another embodiment of the present application provides a method for detecting and removing abnormal data under complex working conditions. Compared with the above-mentioned embodiment, the method for detecting and removing abnormal data under complex working conditions of this embodiment, step S150 includes:

[0064] Step S410, calculating the preliminary anomaly score of the noise data point ,in, and is a preset parameter that controls the initial anomaly score of the noisy data point The corresponding steepness parameter and center position parameter of the curve.

[0065] In this embodiment, by calculating the preliminary anomaly score of the noise data point, the outlier metric can be mapped to a probability score. This method can better reflect the degree of anomaly of the data point, making anomaly detection more accurate.

[0066] Step S420, calculating the spatial consistency score of the noise data point ,in, and is a preset parameter that controls the spatial consistency score of the noise data points The corresponding steepness parameter and center position parameter of the curve.

[0067] In this embodiment, by calculating the spatial consistency score of the noise data point, the spatial distribution of the data point can be considered. This method can identify those data points that are outliers but still have a certain spatial consistency, thereby reducing misjudgment.

[0068] Step S430, calculating the probability that the noise data point is an abnormal data point .

[0069] According to the technical solution of this embodiment, by combining the preliminary anomaly score and the spatial consistency score to calculate the comprehensive anomaly probability, the degree of anomaly of the data point can be more comprehensively evaluated. This method not only considers the outlier metric, but also considers the spatial distribution characteristics of the data points, thereby improving the accuracy and robustness of anomaly detection.

[0070] like Figure 5 As shown, another embodiment of the present application provides a method for detecting and removing abnormal data under complex working conditions. Compared with the above-mentioned embodiment, the method for detecting and removing abnormal data under complex working conditions of this embodiment, step S160 includes:

[0071] Step S510, taking the time point corresponding to the abnormal data point as the removal time point, and selecting a first candidate removal time point from before the removal time point and a second candidate removal time point from after the removal time point according to a preset time interval.

[0072] In this embodiment, by taking the time point corresponding to the abnormal data point as the elimination time point and selecting the previous and next candidate elimination time points on this basis, the elimination range can be determined more accurately. This method can avoid over-elimination or under-elimination of data based on a single point of abnormality.

[0073] Step S520, calculate the difference between the feature vector corresponding to the elimination time point and the feature vector corresponding to the first candidate elimination time point. When the difference exceeds the preset difference value, use the first candidate elimination time point as the new elimination time point, and continue to select the new first candidate elimination time point.

[0074] Step S530, calculate the difference between the feature vector corresponding to the elimination time point and the feature vector corresponding to the second candidate elimination time point. When the difference exceeds the preset difference value, use the second candidate elimination time point as the new elimination time point, and continue to select a new second candidate elimination time point backward.

[0075] In this embodiment, when the feature vector difference between the elimination time point and the previous and next candidate time points exceeds the preset difference value, the elimination time point is dynamically adjusted. This method can better capture the influence range of abnormal data points and ensure that the elimination range is neither too large nor too small.

[0076] Step S540: All data between the smallest elimination time point and the largest elimination time point in the data streams of the multiple sensors are eliminated.

[0077] Step S550, data filling is performed between the minimum elimination time point and the maximum elimination time point, so that the difference between the feature vectors corresponding to any two points between the minimum elimination time point and the maximum elimination time point does not exceed a preset difference value.

[0078] In this embodiment, data filling is performed after data is eliminated so that the gap between the feature vectors corresponding to any two points between the eliminated time points does not exceed the preset gap value. This method can ensure the continuity of data and avoid subsequent analysis errors caused by missing data. By eliminating abnormal data and performing reasonable data filling, a more reliable data basis is provided for subsequent data analysis and decision support, reducing erroneous decisions caused by abnormal data.

[0079] According to the technical solution of this embodiment, by introducing the selection of previous and next candidate time points, dynamically adjusting the elimination range, data filling and other steps, the accuracy and robustness of the elimination of abnormal data points are further improved, ensuring the efficiency and reliability of data processing in complex working conditions. These improvements make the system more stable and reliable when processing complex data, thus providing a solid foundation for subsequent data analysis and decision support.

[0080] like Figure 6As shown, an embodiment of the present application provides a system for detecting and eliminating abnormal data under complex working conditions, including:

[0081] The data stream receiving module 610 receives data streams from multiple sensors during a preset complex working condition operation.

[0082] The feature extraction module 620 extracts features from the data stream of each of the multiple sensors according to multiple preset time points, wherein the features of the data streams of the multiple sensors at each of the multiple time points constitute a feature vector, and the features of the data streams of the multiple sensors at multiple time points constitute multiple feature vectors.

[0083] In this embodiment, by extracting features from sensor data to form feature vectors, the essential attributes of the data can be more accurately reflected. This method can filter out irrelevant noise information and retain important feature information, thereby improving the quality of the data.

[0084] The noise identification module 630 finds the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as the core data point according to the preset neighborhood radius and the minimum number of points, and clusters multiple feature vectors to form multiple feature clusters and noise data points outside the multiple feature clusters.

[0085] In this embodiment, clustering the feature vectors can effectively identify normal data areas and potential abnormal data areas.

[0086] The outlier calculation module 640 calculates the outlier metric of the noise data point.

[0087] In this embodiment, by calculating the outlier metric of the noise data point, the degree of abnormality of each data point can be quantified. This method can not only identify outliers, but also further evaluate their degree of outliers.

[0088] The anomaly calculation module 650 calculates the probability that the noise data point is an anomaly data point according to the outlier metric of the noise data point.

[0089] In this embodiment, the probability of a noise data point being an abnormal data point is calculated by combining the outlier metric and other factors, so as to more accurately determine whether a data point is an abnormal point. This method provides a quantitative evaluation standard and improves the accuracy of anomaly detection.

[0090] The data elimination module 660 eliminates data from the data streams of the multiple sensors according to the time points corresponding to the abnormal data points when the probability that the noise data points are abnormal data points is higher than a preset threshold.

[0091] According to the technical solution of this embodiment, through the steps of feature extraction, cluster analysis, outlier measurement, probability evaluation, etc., the data quality is effectively improved, the anomaly detection capability is enhanced, the data processing efficiency is improved, the adaptability to the communication environment is improved, the security and reliability of the system are enhanced, and it has wide applicability. These technical effects enable this embodiment to have significant advantages in data processing under complex working conditions.

[0092] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0093] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0094] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0095] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0096] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. The method for detecting and eliminating abnormal data under complex working conditions is characterized by: include: Receive data streams from multiple sensors during operations under preset complex working conditions; Extracting features from the data stream of each sensor of the multiple sensors at a preset multiple time points, wherein the features of the data streams of the multiple sensors at each time point in the multiple time points constitute a feature vector, and the features of the data streams of the multiple sensors at the multiple time points constitute multiple feature vectors; According to a preset neighborhood radius and a minimum number of points, find out the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as core data points, cluster the multiple feature vectors to form multiple feature clusters and noise data points outside the multiple feature clusters; Calculating an outlier metric for the noisy data point; Calculating the probability that the noise data point is an abnormal data point according to the outlier metric of the noise data point; When the probability that the noise data point is an abnormal data point is higher than a preset threshold, data is removed from the data streams of the multiple sensors according to the time point corresponding to the abnormal data point, Before finding out the feature vectors whose number of feature vectors in the neighborhood reaches the minimum number of points as the core data points according to the preset neighborhood radius and the minimum number of points, the method further includes: According to a preset reference radius, counting the number of other feature vectors adjacent to each feature vector in the plurality of feature vectors; The local density of each eigenvector is calculated according to the number of other eigenvectors adjacent to each eigenvector, where the local density of the kth eigenvector is ρ k ; Calculate the average local density ρ of the multiple eigenvectors avg ; Calculate the minimum number of points, where the minimum number of points corresponding to the kth eigenvector Before finding out the feature vectors whose number of feature vectors in the neighborhood reaches the minimum number of points as the core data points according to the preset neighborhood radius and the minimum number of points, the method further includes: Calculate the distance between each eigenvector and other eigenvectors; According to the minimum number of points corresponding to each eigenvector, a typical distance is selected from the distances between each eigenvector and other eigenvectors in order from near to far. The typical distance td of the kth eigenvector is k is the MPth distance between the kth eigenvector and other eigenvectors k distance; Calculate the average typical distance td of the multiple feature vectors avg ; Calculate the neighborhood radius Wherein, α is a preset adjustment factor, n represents the number of the plurality of feature vectors, Calculating the outlier metric of the noise data point includes: Calculate the centers of the multiple feature clusters, where the center of the i-th feature cluster is C i ; Calculate the distance from the noise data point to the center of the multiple feature clusters, where the distance from the noise data point to the center of the i-th feature cluster is d(x, C i ), x represents the noise data point; Calculate the distance from all feature vectors to the center of the feature cluster, take the mean μ and the standard deviation σ ; Calculate the outlier metric for the noisy data point Wherein, m is the number of the plurality of feature clusters, w1 is a preset weight coefficient, After calculating the outlier metric of the noise data point, the method further includes: Query the previous time point of the time point corresponding to the noise data point; Query the feature vector x of the data stream of the multiple sensors at the previous time point p ; Calculate the fluctuation amplitude of the noise data point Among them, γ is the preset reference time; Based on the fluctuation amplitude A of the noise data point, the outlier metric D of the noise data point is corrected, and the corrected outlier metric D of the noise data point is neo =D+w2A, where w2 is the preset correction coefficient, The step of calculating the probability that the noise data point is an abnormal data point according to the outlier metric of the noise data point includes: Calculate the preliminary anomaly score of the noisy data point Wherein, β1 and β2 are preset parameters, which are the steepness parameter and the center position parameter of the curve corresponding to the preliminary abnormality score S1 of the noise data point; Calculate the spatial consistency score of the noise data points Wherein, β3 and β4 are preset parameters, which are the steepness parameter and the center position parameter of the curve corresponding to the spatial consistency score S2 of the noise data point; Calculate the probability P=S1×S2 that the noise data point is an abnormal data point, Wherein, according to the time point corresponding to the abnormal data point, removing data from the data streams of the multiple sensors includes: The time point corresponding to the abnormal data point is used as a removal time point, and according to a preset time interval, a first candidate removal time point is selected from before the removal time point, and a second candidate removal time point is selected from after the removal time point; Calculate the difference between the feature vector corresponding to the elimination time point and the feature vector corresponding to the first candidate elimination time point, and when the difference exceeds a preset difference value, use the first candidate elimination time point as a new elimination time point, and continue to select a new first candidate elimination time point; Calculate the difference between the feature vector corresponding to the elimination time point and the feature vector corresponding to the second candidate elimination time point, and when the difference exceeds a preset difference value, use the second candidate elimination time point as a new elimination time point, and continue to select a new second candidate elimination time point backward; All data between the smallest elimination time point and the largest elimination time point in the data streams of the multiple sensors are eliminated.

2. The method for detecting and eliminating abnormal data under complex working conditions according to claim 1, characterized in that: After all data between the smallest elimination time point and the largest elimination time point in the data streams of the multiple sensors are eliminated, the method further includes: Data filling is performed between the minimum elimination time point and the maximum elimination time point, so that the gap between the feature vectors corresponding to any two points between the minimum elimination time point and the maximum elimination time point does not exceed the preset gap value.

3. A system for detecting and eliminating abnormal data under complex working conditions, adopting a method for detecting and eliminating abnormal data under complex working conditions according to any one of claims 1-2, characterized in that: include: A data stream receiving module receives data streams from multiple sensors during a preset complex working condition; A feature extraction module, extracting features from a data stream of each of the multiple sensors at a plurality of preset time points, wherein the features of the data streams of the multiple sensors at each of the plurality of time points constitute a feature vector, and the features of the data streams of the multiple sensors at the plurality of time points constitute a plurality of feature vectors; The noise identification module finds out the feature vectors whose number of feature vectors in all neighborhoods reaches the minimum number of points as core data points according to the preset neighborhood radius and the minimum number of points, and clusters the multiple feature vectors to form multiple feature clusters and noise data points outside the multiple feature clusters; An outlier calculation module calculates an outlier metric of the noise data point; An anomaly calculation module, which calculates the probability that the noise data point is an abnormal data point according to the outlier metric of the noise data point; The data elimination module eliminates data from the data streams of the multiple sensors according to the time points corresponding to the abnormal data points when the probability that the noise data points are abnormal data points is higher than a preset threshold.

Images