Abnormal data detection method and device, electronic equipment and computer storage medium

Abstract

The invention provides an abnormal data detection method and device, electronic equipment and a computer storage medium. The method comprises the following steps: receiving first to-be-tested data in a sliding window; performing anomaly detection on the first to-be-detected data to obtain global anomaly data, the global anomaly data being outlier data of the first to-be-detected data in overall distribution; eliminating the global abnormal data in the first to-be-tested data to obtain second to-be-tested data; and performing anomaly detection on the second to-be-detected data to obtain local anomaly data, the local anomaly data being outlier data of the second to-be-detected data in local distribution, and taking the global anomaly data and the local anomaly data as target anomaly data. According to the method and the device, the target abnormal data in the data can be accurately determined in real time.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to an abnormal data detection method, apparatus, electronic device, and computer storage medium. Background Art

[0002] With the continuous development of big data processing technology and the Internet field, a large amount of data is generated every day in different fields. For example, in the financial field, abnormal data detection is performed on data for credit evaluation and other aspects to improve the security of funds.

[0003] For example, in the vehicle field, abnormal data detection can be performed on the data during the driving process of a vehicle. When abnormal data is detected, it is determined that an abnormal situation has occurred during the driving process of the vehicle, so that an early warning can be given to the abnormal situation of the vehicle in a timely manner. Therefore, how to process data to accurately and real-time determine the abnormal data in the data has become an urgent problem to be solved. Summary of the Invention

[0004] In view of the above, embodiments of this application provide an abnormal data detection method, apparatus, electronic device, and computer storage medium, which can accurately and real-time determine the abnormal data in the data.

[0005] A first aspect of this application provides an abnormal data detection method, and the method includes: receiving first data to be measured within a sliding window; performing abnormal detection on the first data to be measured to obtain global abnormal data, where the global abnormal data is the outlier data in the overall distribution of the first data to be measured; removing the global abnormal data from the first data to be measured to obtain second data to be measured; performing abnormal detection on the second data to be measured to obtain local abnormal data, where the local abnormal data is the outlier data in the local distribution of the second data to be measured; using the global abnormal data and the local abnormal data as target abnormal data.

[0006] Compared with the related art, embodiments of this application have at least the following advantages: First, perform anomaly detection on the first data to be measured, and global anomaly data can be determined from the overall distribution of the first data to be measured. Then, perform anomaly detection on the second data to be measured again, and local anomaly data can be determined from the local distribution of the second data to be measured. On the one hand, first determine the global anomaly data, and then determine the local anomaly data from the part of the first data to be measured after removing the global anomaly data, which reduces the time for anomaly data confirmation; on the other hand, for example, in the field of vehicle technology, based on the global anomaly data and local anomaly data, it is convenient to timely warn of the abnormal operation of the vehicle. At the same time, using the sliding window technology to receive the first data to be measured during the driving process of the vehicle can improve the efficiency of anomaly detection of the first data to be measured and quickly determine the target anomaly data, meeting the timeliness of data anomaly detection.

[0007] In some possible implementation manners, the performing anomaly detection on the first data to be measured to obtain global anomaly data includes: calculating the path length of each data in the first data to be measured in a pre-constructed isolation tree, where the isolation tree is a binary tree constructed based on the isolation forest algorithm; calculating the anomaly score of each data in the first data to be measured based on multiple path lengths; and determining the global anomaly data based on multiple anomaly scores.

[0008] In some possible implementation manners, the performing anomaly detection on the second data to be measured to obtain local anomaly data includes: determining a neighborhood data set of a first data in the second data to be measured based on a preset distance threshold, where the first data is any data in the second data to be measured; calculating the local reachability density of the first data based on the neighborhood data set; calculating the local anomaly factor of the first data based on the local reachability density; and when the local anomaly factor is greater than the anomaly threshold, taking the first data as the local anomaly data.

[0009] In some possible implementation manners, the calculating the local reachability density of the first data based on the neighborhood data set includes: calculating the reachable distance between each data in the neighborhood data set and the first data; and taking the reciprocal of the average value of multiple reachable distances as the local reachability density.

[0010] In some possible implementation manners, before performing anomaly detection on the first data to be measured to obtain global anomaly data, the method further includes: detecting whether there is a preset threshold that matches the first data to be measured, where the first data to be measured includes multiple types of signal data, and the multiple types of signal data correspond to the preset threshold one by one; in the case where it is detected that there is no preset threshold, performing anomaly detection on the first data to be measured to obtain the global anomaly data; in the case where it is detected that there is a preset threshold, performing anomaly detection on the first data to be measured based on the preset threshold to obtain the target anomaly data.

[0011] In some possible implementation manners, the first data to be measured includes multiple types of signal data; after receiving the first data to be measured within the reception sliding window, the method further includes: constructing a signal association graph based on the multiple types of signal data, where the signal association graph includes multiple nodes, and the multiple nodes correspond to the multiple types of signal data one by one, and the signal association graph represents the association relationship between the multiple nodes; after determining the global anomaly data based on the multiple anomaly scores, the method includes: determining, from the signal association graph, a first node associated with the node corresponding to each data in the global anomaly data; in the case where the signal data corresponding to the first node is the outlier data in the overall distribution of the first data to be detected, adding the signal data corresponding to the first node to the global anomaly data; after, when the local anomaly factor is greater than the anomaly threshold, taking the first data as the local anomaly data, the method includes: determining, from the signal association graph, a second node associated with the node corresponding to each data in the local anomaly data; in the case where the signal data corresponding to the second node is the outlier data in the local distribution of the second data to be measured, adding the signal data corresponding to the second node to the local anomaly data.

[0012] In some possible implementation manners, each type of signal data is collected by one acquisition device; constructing the signal association graph based on the multiple types of signal data includes: calculating the embedding vector of each acquisition device; obtaining weight data based on the multiple embedding vectors, where the weight data represents the relationship weights between the multiple acquisition devices; extracting the feature data of the multiple types of signal data based on the weight data; constructing the signal association graph based on the multiple types of feature data.

[0013] In some possible implementation manners, constructing the signal association graph based on the multiple types of feature data includes: performing an aggregation process on the multiple types of feature data to obtain aggregation data, where the signal data corresponding to each node in the aggregation data includes the feature information of the signal data corresponding to the neighbor nodes; constructing the signal association graph based on the aggregation data.

[0014] The second aspect of the present application discloses an abnormal data detection device, including: a data receiving module for receiving first data to be measured within a sliding window; a data processing module for performing abnormal detection on the first data to be measured to obtain global abnormal data, where the global abnormal data is outlier data in the overall distribution of the first data to be measured, removing the global abnormal data from the first data to be measured to obtain second data to be measured, and performing abnormal detection on the second data to be measured to obtain local abnormal data, where the local abnormal data is outlier data in the local distribution of the second data to be measured, and taking the global abnormal data and the local abnormal data as target abnormal data.

[0015] The third aspect of the present application discloses an electronic device, which includes a processor and a memory. The memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the abnormal data detection method as described above.

[0016] The fourth aspect of the present application discloses a computer storage medium, including computer instructions. When the computer instructions run on an electronic device, the electronic device is made to execute the abnormal data detection method as described above.

[0017] It can be understood that the abnormal data detection device in the second aspect, the electronic device in the third aspect, and the computer storage medium in the fourth aspect provided above match the method in the first aspect. Therefore, the beneficial effects they can achieve can refer to the beneficial effects in the matching method provided above, and will not be elaborated here. Description of the Drawings

[0018] Figure 1 is a flowchart of steps of an abnormal data detection method according to an embodiment of the present application.

[0019] Figure 2 is another flowchart of steps of an abnormal data detection method according to an embodiment of the present application.

[0020] Figure 3 is a schematic structural diagram of an abnormal data detection device according to an embodiment of the present application.

[0021] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Detailed Embodiments

[0022] In order to more clearly understand the above-mentioned objects, features, and advantages of the present application, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0023] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. The described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs. The terms used in the specification of the present application herein are only for the purpose of describing specific embodiments, and are not intended to limit the present application.

[0025] Furthermore, it should be noted that in this document, the terms "comprising", "including" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element.

[0026] In the present application, "at least one" means one or more, and "a plurality" means two or more than two. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural.

[0027] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0028] With the continuous development of the vehicle technology field and intelligent network connection technology, more and more vehicles have emerged on the road. Currently, in the process of vehicle production, vehicle manufacturers will configure a large number of sensors and intelligent devices on the vehicle to collect first measured data during the vehicle's driving process.

[0029] For example, in the field of vehicles, abnormal data detection can be performed on the first data to be measured. When abnormal data is detected, it is determined that an abnormal situation has occurred during the vehicle's driving, so that the abnormal situation of the vehicle can be warned in a timely manner. Therefore, how to process data to accurately and real-time determine the abnormal data in the data has become an urgent problem to be solved.

[0030] To solve this problem, the present application provides an abnormal data detection method, which is applied to an abnormal data detection device. One or more vehicles are communicatively connected to the abnormal data detection device, so that the abnormal data detection device can detect the abnormal situations of multiple vehicles during driving and give early warnings. Specifically, the on-vehicle terminals of one or more vehicles are communicatively connected to the abnormal data detection device.

[0031] In this embodiment, the abnormal data detection device includes a processor, and the processor may include one or more processing units. For example, the processor may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0032] To facilitate understanding of the technical solutions described in the present application, some terms in the embodiments of the present application are explained below: In this embodiment, the sliding window algorithm is an algorithm technology based on an array or a string. It mainly defines a window and then slides the window on the data structure to gradually process the data located within the window.

[0033] The Flink data stream processing framework is a distributed processing engine mainly used for stateful computing on unbounded and bounded data streams.

[0034] Kafka is an open-source distributed message system mainly used for processing large-scale fact data streams to meet the real-time processing requirements of data.

[0035] The Isolation Forest algorithm is an unsupervised learning algorithm used to mine abnormal data or outliers. This algorithm mainly recursively and randomly divides the dataset until all sample points are isolated. Under this random division strategy, outliers usually have shorter paths. That is to say, those clusters with high density need to be cut many times to be isolated, but those points with low density can be easily isolated. Therefore, when using the Isolation Forest algorithm to mine abnormal data or outliers, it has advantages such as high efficiency and easy parallelization.

[0036] The Local Outlier Factor (LOF) detection algorithm is also an algorithm for data anomaly detection. It mainly identifies outliers by comparing the relative density of data points with their neighboring points. The core idea of the LOF algorithm is to measure the degree of anomaly of a data point by calculating its local reachability density. Specifically, for each data point, the algorithm calculates its distances to all other points and finds its k-nearest neighbor distance. Then, based on the densities of these neighbors, the local outlier factor of this data point is calculated, which is the ratio of the average local reachability density of points within the neighborhood of this point to the local reachability density of this point. If the value of the local outlier factor is larger, it indicates that this point is more abnormal; conversely, if the value of the local outlier factor is smaller, it indicates that this point is more normal.

[0037] The graph neural network model (GNN) is a deep learning model for processing graph data. It mainly defines a neural network model on the nodes and edges of the graph and captures the relationships between nodes through information passing and aggregation. By iteratively updating the representation of nodes, each node can take into account the information of its neighboring nodes.

[0038] The Graph Deviation Network (GDN) is a multivariate time series anomaly detection method based on graph neural networks. This model can learn the different behaviors between different sensors, thereby capturing the dependencies and interactions between sensor data, and then sending the data into other models for learning, which can provide interpretability for anomaly detection.

[0039] Please combine Figure 1 , Figure 1 as the step flow chart of the anomaly data detection method. According to different requirements, the order of steps in the flow chart can be changed, and some steps can be omitted. The anomaly data detection method in this embodiment can be used to detect anomalies in the data of a vehicle during driving, so as to give an early warning to the vehicle in time after detecting abnormal data, thereby improving the driving safety of the vehicle. In other embodiments, this anomaly data detection method can also be used in fields such as finance and banking. This application does not limit this.

[0040] The abnormal data detection method includes the following steps: Step 101: Receive the first data to be measured within the sliding window.

[0041] In some embodiments, one or more vehicles communicatively connected to the abnormal data detection device send their own data during driving to the abnormal data detection device. Meanwhile, the abnormal data detection device maintains a sliding window with a preset length and receives the data sent by one or more vehicles (i.e., the first data to be measured) in real time based on the sliding window. Among them, the preset length of the sliding window can be set to 30 minutes or 40 minutes, and it can be set according to the actual detection requirements.

[0042] In this embodiment, the first data to be measured includes various signal data, and the various signal data includes charge and discharge signal data, vehicle speed signal data, mileage signal data, total voltage signal data, total current signal data, gear signal data, engine signal data, etc. Since the first data to be measured includes discrete data and continuous data, the discrete data and continuous data in the first data to be measured are respectively stored in different data tables, so as to facilitate subsequent abnormal data detection of discrete data and continuous data respectively and improve the efficiency of abnormal detection.

[0043] In this embodiment, since the data sent by one or more vehicles is in the form of a data stream, where the data stream refers to a data sequence that is large, wide-area, fast, continuous, and whose data spatial distribution changes over time. Therefore, using Kafka as the message backbone to be responsible for ingesting and distributing the data stream, and at the same time, using the Flink data stream processing framework to process and analyze the data stream can reduce the latency of abnormal detection and make the processing and storage of the data stream more efficient and flexible. Using Kafka and the Flink data stream processing framework to distribute and process the data stream is only an example. In other embodiments, other data stream processing technologies can also be used to process it, and this application does not limit this.

[0044] Step 102: Perform abnormal detection on the first data to be measured to obtain global abnormal data, where the global abnormal data is the outlier data in the overall distribution of the first data to be measured.

[0045] In this embodiment, the global abnormal data includes data points (such as outlier data) that are significantly far from other points in the data space of the first data to be measured.

[0046] Before performing abnormal detection on the first data to be measured, the abnormal data detection device can first perform data preprocessing on the first data to be measured. Among them, the data preprocessing includes data cleaning, data elimination, etc.

[0047] In this embodiment, the abnormal data detection device can use the Isolation Forest algorithm to detect abnormalities in the first data to be measured and obtain global abnormal data.

[0048] For example, the abnormal data detection device can first use the Isolation Forest algorithm to construct isolation trees, and then calculate the path length of each data in the first data to be measured in the isolation trees. Among them, the isolation tree is a binary tree.

[0049] Then, based on multiple path lengths, calculate the anomaly score of each data in the first data to be measured. Based on multiple anomaly scores, determine the global abnormal data.

[0050] Specifically, the abnormal data detection device can use the training data to construct multiple isolation trees. The training data can be the data received by the abnormal data detection device in the historical period. The training data is divided into multiple data sets by randomly selecting features and split points, and multiple isolation trees are constructed based on the multiple data sets. Based on the first data to be measured, calculate the path length of each data in the first data to be measured in each isolation tree. Take the average value of multiple path lengths as the average path length. Based on the average path length, the Isolation Forest algorithm will calculate the anomaly score of each data in the first data to be measured. Compare multiple anomaly scores with a preset detection threshold respectively, and take the data matching the anomaly score greater than the preset detection threshold as the global abnormal data.

[0051] Among them, the specific data of the preset detection threshold can be set according to the actual detection requirements, and the embodiments of the present application do not limit this.

[0052] In this embodiment, on the one hand, the calculation speed of the Isolation Forest algorithm can be improved by setting the maximum depth of each isolation tree to the logarithm of the sample size. Since the abnormal data in the first data to be measured is usually less, the path length of the abnormal data is shorter, while the path length of the normal data is longer. Therefore, by limiting the maximum depth of the isolation tree, the calculation amount can be reduced and the algorithm efficiency can be improved.

[0053] On the other hand, the calculation speed of the Isolation Forest algorithm can also be improved by subsampling to limit the sample size for constructing the isolation tree. The size of the subsampling is usually much smaller than the size of the original data set, which can limit the size of the isolation tree and reduce the computational complexity.

[0054] Through these two methods, the Isolation Forest algorithm can improve the calculation speed while maintaining a high detection accuracy.

[0055] At the same time, the Isolation Forest algorithm is based on a tree structure and can quickly partition data without the need to model the entire dataset, making it suitable for scenarios with high real-time detection requirements. Moreover, the Isolation Forest algorithm constructs isolation trees by randomly selecting features and split points instead of modeling the entire data space, enabling it to effectively process datasets with a large number of features and achieving high efficiency in data processing and outlier data detection.

[0056] Step 103: Remove the global outlier data from the first data to be measured to obtain the second data to be measured.

[0057] In this embodiment, the global outlier data in the first data to be measured is removed to obtain the second data to be measured. Thus, accurate local outlier data can be obtained based on the second data to be measured in the subsequent process.

[0058] Step 104: Perform outlier detection on the second data to be measured to obtain local outlier data, where the local outlier data are the outlier data in the local distribution of the second data to be measured.

[0059] In some embodiments, the local outlier data includes data corresponding to the farther neighbor points in each density cluster among different density clusters in the data space of the second data to be measured. If the proportion of outlier data in the first data to be measured is relatively high, using only the Isolation Forest algorithm to detect the global outlier data in the first data to be measured will result in inaccurate outlier data detection efficiency. For example, the detection of data points with slightly abnormal features in the layout area of the overall distribution of the first data to be measured is not very sensitive. Therefore, other techniques are needed to detect outlier data in the local area.

[0060] In this embodiment, the LOF algorithm is used to perform outlier detection on the second data to be measured to obtain local outlier data. The specific steps include: performing data preprocessing on the second data to be measured again, where the data preprocessing includes normalization processing to eliminate the influence of the dimension between the features of different data, enabling the LOF algorithm to more accurately compare the distances between data points.

[0061] Further, the abnormal data detection device can, based on a preset distance threshold, determine a neighborhood data set of the first data in the second data to be measured after data preprocessing, where the first data is any one of the data in the second data to be measured. In this embodiment, it is assumed that the preset distance threshold is k, and k is an integer greater than 1. Then, the neighborhood data set of the first data is the k-distance neighborhood of the first data. Before determining the k-distance neighborhood, it is necessary to first calculate the k-distance of the first data, and the k-distance is the distance from the first data to the k-th nearest neighbor point. The data whose distance from the first data is less than or equal to the k-distance belongs to the neighborhood data set. That is to say, the k-distance neighborhood is all the data points whose distance from the first data does not exceed the k-distance.

[0062] The abnormal data detection device can calculate the reachable distance between each data in the neighborhood data set and the first data. Take the reciprocal of the average value of multiple reachable distances as the local reachability density. In this embodiment, the local outlier factor of the first data is the ratio between the mean value of the local reachability densities of all the data in the k-distance neighborhood of the first data and the local reachability density of the first data. Based on the local reachability density, calculate the local outlier factor of the first data. When the local outlier factor is greater than the outlier threshold, the first data is regarded as local abnormal data.

[0063] In this embodiment, the LOF algorithm is used to process each data in the second data to be measured to determine whether each data is local abnormal data.

[0064] Step 105: Take the global abnormal data and the local abnormal data as the target abnormal data.

[0065] In some embodiments, after the abnormal data detection device detects that a certain data point of a certain signal data is abnormal data, it is necessary to determine that a continuous plurality of data points in the signal data are all abnormal data before taking it as the target abnormal data and sending a warning message to the vehicle in a timely manner. Thus, false alarm situations caused by instantaneous fluctuations of the signal are avoided. Among them, the warning message can be sent in various ways such as sending an email, a text message, etc.

[0066] In some embodiments, in order to meet the real-time performance of abnormal data detection, before performing abnormal detection on the first data to be measured to obtain global abnormal data, it is also possible to first detect whether there is a preset threshold that matches the first data to be measured. Among them, there can be multiple preset thresholds, and the first data to be measured includes multiple types of signal data, and the multiple types of signal data correspond to the multiple preset thresholds one by one. When it is detected that there is a preset threshold, perform abnormal detection on the first data to be measured based on the preset threshold to obtain the target abnormal data. When it is detected that there is no preset threshold, use the isolation forest algorithm and the LOF algorithm to perform abnormal detection on the first data to be measured to obtain the target abnormal data.

[0067] Specifically, the user sets corresponding preset thresholds according to the differences in the signal data of the first data to be measured. For example, for vehicle speed signal data, the preset threshold can be set to 0 km / h to 150 km / h, or 0 km / h to 160 km / h. For engine signal data, the preset threshold can be set to 0 and 1, where 0 represents that the engine is in the off state and 1 represents that the engine is in the running state. When detecting the vehicle speed signal data, directly use 0 km / h to 150 km / h, or 0 km / h to 160 km / h to detect whether there is abnormal data in the vehicle speed signal data. When detecting the engine signal data, directly use 0 and 1 to detect whether there is abnormal data in the engine signal data. Assume that the user does not maintain the total voltage signal data, and directly use the Isolation Forest algorithm and the LOF algorithm to detect whether there is abnormal data in the total voltage signal data to adapt to the dynamic changes of the data.

[0068] In some embodiments, after a period of time, if the user maintains the preset threshold corresponding to the total voltage signal data, the abnormal data detection method will detect the data of the total voltage signal based on the preset threshold corresponding to the total voltage signal data. Moreover, the abnormal data detection method will use the Isolation Forest algorithm and the LOF algorithm to process the historical data and the current data simultaneously, and detect the total voltage signal data in the historical period based on the preset threshold corresponding to the total voltage signal data. Based on the detection results, adjust the relevant parameters in the Isolation Forest algorithm and the LOF algorithm. Thus, the abnormal detection efficiency of the Isolation Forest algorithm and the LOF algorithm is improved.

[0069] Compared with the related technology, the embodiments of the present application have at least the following advantages: On the one hand, using the Kafka and Flink data stream processing frameworks to distribute and process the data stream in the sliding window can reduce the delay of abnormal detection, and make the processing and storage of the data stream more efficient and flexible. On the other hand, when there is a preset threshold, directly use the preset threshold to determine the target abnormal data, otherwise, use the Isolation Forest algorithm and the LOF algorithm to determine the target abnormal data, which can meet the real-time requirements of abnormal detection and adapt to the dynamic changes of the data. On the other hand, first use the Isolation Forest algorithm to determine the global abnormal data, and then use the LOF algorithm to detect the first data to be measured after removing the global abnormal data to determine the local abnormal data, which improves the accuracy of determining the target abnormal data and the abnormal detection efficiency.

[0070] Please combine Figure 2 , Figure 2Another flowchart of steps for the abnormal data detection method. This embodiment is a further description of the above embodiment. The main improvement lies in determining the target abnormal data by combining the correlation relationships among multiple signals in the first data to be measured. The abnormal data detection method includes the following steps: Step 201: Receive the first data to be measured within the sliding window. The first data to be measured includes multiple signal data.

[0071] In this embodiment, as described above, the multiple signal data includes charge-discharge signal data, vehicle speed signal data, mileage signal data, total voltage signal data, total current signal data, gear position signal data, engine signal data, etc. Other contents of this step are the same as those of step 101 and will not be elaborated here.

[0072] Step 202: Based on the multiple signal data, construct a signal correlation graph. The signal correlation graph includes multiple nodes, and the multiple nodes correspond one-to-one with the multiple signal data. The signal correlation graph represents the correlation relationships among the multiple nodes.

[0073] In this embodiment, each type of signal data is collected by a collection component. The collection component can be a sensor. To obtain the correlation relationships among the multiple nodes, a signal correlation graph needs to be constructed. The specific steps include: calculating the embedding vector of each collection component using the GDN model to capture the unique features of each collection component. The multiple embedding vectors will be trained after initialization for structure learning and judging the correlation among the collection components. Based on the multiple embedding vectors, the GDN model calculates the similarity among the multiple collection components to determine whether there are edge links among the multiple collection components, thereby constructing an adjacency matrix. The GDN model is made to learn the relationships among the collection components based on the adjacency matrix and encode them as edges in the graph.

[0074] Specifically, the GDN model uses the graph attention mechanism to predict the value of each collection component at the next moment. By aggregating the features of adjacent collection components in the graph, the GDN model can fuse the features of each collection component and predict the value of each collection component. Then, based on the multiple embedding vectors, weight data is obtained, and the weight data represents the relationship weights among the multiple collection components. Finally, since the GDN model can learn the relationship weights among the multiple collection components, the characteristic data of the multiple signal data to be extracted can be determined based on the weight data. Based on the multiple characteristic data, a signal correlation graph is constructed.

[0075] That is, based on the weight data learned by the GDN model, the characteristic data of each type of signal data is determined. Before extracting the characteristic data of multiple types of signal data, it is necessary to construct a feature vector for each type of signal data. For example, the engine signal data represents the operating state of the vehicle engine, and the feature vector of the engine signal data includes data such as engine speed, temperature, oil pressure, and vibration. The vehicle speed signal data represents the real-time speed of the vehicle, and the feature vector of the vehicle speed signal data includes data such as instantaneous speed, acceleration, and deceleration. The brake system signal data reflects the working state of the brake system, and the feature vector of the brake system signal data includes data such as braking force and brake temperature. The tire signal data includes information such as tire pressure and wear degree, and the feature vector of the tire signal data includes data such as air pressure value and temperature. The fuel system signal data involves fuel quantity, fuel pressure, etc., and the feature vector of the fuel system signal data includes data such as remaining fuel quantity and fuel consumption rate. The environmental perception signal data includes information such as the temperature, humidity, and road conditions of the vehicle's surrounding environment, and the feature vector of the environmental perception signal data includes data such as external temperature data and humidity data.

[0076] The characteristic data of each type of signal data may include the mean, variance, skewness, kurtosis, quantiles, autocorrelation, and entropy of the signal data. Among them, the mean is the average value of the signal data, and the variance is mainly used to measure the degree of deviation of the signal value from its mean. Skewness is mainly used to describe the asymmetry of the signal distribution. Kurtosis mainly describes the "tail" degree of the signal distribution, that is, the frequency of extreme values of the signal value. Quantiles are values that divide the data into intervals with specific probability ranges. Autocorrelation mainly describes the correlation degree of the signal with its past values, reflecting the pattern degree of the signal. Entropy mainly describes the complexity or unpredictability of the data. In other embodiments, the characteristic data of each type of signal data may also include other characteristic values, and the present application does not limit this.

[0077] Based on the weight data learned by the GDN model, the characteristic data of the engine signal data is extracted from the feature vector of the engine signal data. For example, the characteristic data is: [mean speed, variance temperature, skewness oil pressure, kurtosis vibration]. The extraction method of the characteristic data of other signal data can refer to the steps of the characteristic data of the engine signal data.

[0078] Furthermore, based on multiple types of characteristic data, a signal association graph is constructed, including: using the GNN model to perform aggregation processing on multiple types of characteristic data to obtain aggregated data, where the signal data corresponding to each node in the aggregated data includes the characteristic information of the signal data corresponding to the neighbor nodes. In this embodiment, the aggregation function in the GNN model combines the information of each node and its neighbor nodes to update the state of each node. The specific steps include: Each node in the GNN model first generates messages based on its own features and the features of its neighbor nodes, and propagates the information of each node to its neighbor nodes. Then, based on the aggregation function, each node aggregates information from its neighbor nodes. The aggregation function can be sum, mean, max, etc. The information aggregated by the GNN model is used to update the state of each node. Based on the aggregated data, a signal correlation graph is constructed.

[0079] Step 203: Perform anomaly detection on the first data to be measured to obtain global anomaly data.

[0080] The content of this step is the same as that of step 102. To avoid repetition, it will not be elaborated here.

[0081] Step 204: Determine the first nodes associated with the nodes corresponding to each data in the global anomaly data from the signal correlation graph.

[0082] In this embodiment, the first nodes are the nodes in the signal correlation graph. Step 203 can obtain global anomaly data. For example, the global anomaly data includes multiple first anomaly data. The first nodes associated with the nodes corresponding to each first anomaly data are determined from the signal correlation graph, and multiple first nodes are obtained.

[0083] Step 205: When the signal data corresponding to the first node is an outlier data in the overall distribution of the first data to be measured, add the signal data corresponding to the first node to the global anomaly data.

[0084] In this embodiment, it is detected whether the signal data corresponding to the first node is an outlier data in the overall distribution of the first data to be measured. When it is detected that the signal data corresponding to the first node is an outlier data in the overall distribution of the first data to be measured, the signal data corresponding to the first node is added to the global anomaly data.

[0085] By detecting whether the data of other nodes associated with the node where the outlier data is located is also outlier data, the comprehensiveness of the detected global anomaly data is improved, and the efficiency of determining the global anomaly data is also improved.

[0086] Step 206: Remove the global anomaly data from the first data to be measured to obtain the second data to be measured.

[0087] Step 207: Perform anomaly detection on the second data to be measured to obtain local anomaly data.

[0088] For the specific content of step 206, reference can be made to step 103, and for the specific content of step 207, reference can be made to step 104. It will not be elaborated here.

[0089] Step 208: Determine a second node associated with each node corresponding to the data in the local abnormal data from the signal association graph.

[0090] In this embodiment, the second node is also a node in the signal association graph. Local abnormal data can be obtained in step 207. For example, the local abnormal data includes multiple second abnormal data. Determine a second node associated with each node corresponding to the second abnormal data from the signal association graph to obtain multiple second nodes.

[0091] Step 209: When the signal data corresponding to the second node is an outlier data in the local distribution of the second data to be measured, add the signal data corresponding to the second node to the local abnormal data.

[0092] In this embodiment, detect whether the signal data corresponding to the second node is an outlier data in the local distribution of the second data to be measured. When it is detected that the signal data corresponding to the second node is an outlier data in the local distribution of the second data to be measured, add the signal data corresponding to the second node to the local abnormal data.

[0093] By detecting whether the data of other nodes associated with the node where the outlier data is located is also outlier data, the comprehensiveness of the detected local abnormal data is improved and the efficiency of determining the local abnormal data is increased.

[0094] It should be noted that based on the GND model, the association weights between multiple acquisition components can be determined. Then, based on the association weights between multiple acquisition components, the characteristic data of various signal data to be extracted can be determined, so that the characteristic data of each signal data can reflect the characteristics of the signal data to a large extent. Then, based on the GNN model, a signal association graph is constructed. Based on the signal association graph, the isolation forest algorithm and the LOF algorithm are used to detect the first data to be measured to determine whether there are also abnormalities in other signal data associated with a certain signal data. Thus, the abnormal detection efficiency and accuracy of the target abnormal data are improved.

[0095] For example, in a signal association graph, there is an edge between the node corresponding to the engine signal data and the node corresponding to the vehicle speed signal data because the engine state directly affects the vehicle speed. Therefore, there is an association between the engine signal data and the vehicle speed signal data. Similarly, the change in vehicle speed will affect the use of brakes, and there is an edge between the node corresponding to the vehicle speed signal data and the node corresponding to the brake system signal data. The state of the tires will affect the driving speed and stability of the vehicle, and there is an edge between the node corresponding to the tire signal and the node corresponding to the vehicle speed signal. Therefore, when the Isolation Forest algorithm and the LOF algorithm detect an anomaly in the engine signal data, they will detect whether there are anomalies in the vehicle speed signal data and the brake system signal data. In this way, when the Isolation Forest algorithm and the LOF algorithm detect an anomaly in a certain type of signal data, they can synchronously detect whether there are anomalies in other signal data associated with the anomalous signal data, thereby improving the accuracy of data anomaly detection.

[0096] Compared with the related art, the embodiments of the present application have at least the following advantages: On the one hand, the GDN model can learn the association relationships between multiple acquisition components corresponding to multiple signal data, thereby obtaining weight data. Based on the weight data, the characteristic data of multiple signal data to be extracted can be determined, so that the characteristic data can accurately reflect the performance characteristics of the signal data, which is convenient for accurately detecting abnormal data when analyzing target abnormal data subsequently. On the other hand, the GNN model can construct a signal association graph. After the Isolation Forest algorithm and the LOF algorithm detect an abnormal data, they can further detect whether there are also anomalies in other signal data associated with the abnormal data, thereby improving the efficiency and accuracy of abnormal data detection.

[0097] Please refer to Figure 3 , Figure 3 which is a schematic structural diagram of the abnormal data detection device provided by the embodiments of the present application. In this embodiment, the abnormal data detection device further includes a data receiving module, a data storage module, and a data processing module, and the processor is communicatively connected to the data receiving module, the data storage module, and the data processing module. The data receiving module is used to receive the first data to be measured within the sliding window. The data storage module is used to store all the data received by the data receiving module.

[0098] In this embodiment, the data processing module includes a data management module and a data detection module. The data management module includes a data configuration management unit, a data classification unit and a monitoring rule configuration unit. Among them, the data configuration management unit is responsible for the basic attribute settings of the signal data, for example, configuring a unique data identifier for each signal data, and the type of data to be saved, etc. The data classification unit divides the signal data into discrete data and continuous data, and establishes corresponding data tables for discrete data and continuous data respectively, and the user can set the range of preset thresholds for the signal data corresponding to the discrete data and the signal data corresponding to the continuous data respectively. The monitoring rule configuration unit is used for the user to configure the correlation between multiple signal data, set the priority of detecting multiple signal data and the detection interval duration.

[0099] The data detection module is used to perform anomaly detection on the first data to be tested to obtain global anomaly data; remove the global anomaly data in the first data to be tested to obtain the second data to be tested; perform anomaly detection on the second data to be tested to obtain local anomaly data. The global anomaly data and the local anomaly data are used as target anomaly data.

[0100] Furthermore, the data detection module is also used to construct a signal association diagram based on multiple signal data, the signal association diagram includes multiple nodes, the multiple nodes correspond to the multiple signal data one by one, and the signal association diagram represents the association relationship between the multiple nodes. After performing anomaly detection on the first data to be tested and obtaining global anomaly data, a first node associated with a node corresponding to each data in the global anomaly data is determined from the signal association diagram. And the signal data corresponding to the first node is added to the global anomaly data; the global anomaly data in the first data to be tested is eliminated to obtain the second data to be tested. And, after performing anomaly detection on the second data to be tested and obtaining local anomaly data, a second node associated with a node corresponding to each data in the local anomaly data is determined from the signal association diagram, and the signal data corresponding to the second node is added to the local anomaly data.

[0101] In some embodiments, the abnormal data detection device further includes an alarm module, and the processor is communicatively connected to the alarm module. The alarm module includes an alarm rule management unit, an alarm distribution unit, and an alarm suppression unit. The alarm rule management unit supports customizing notification methods corresponding to different abnormal levels. For example, when abnormal engine signal data is detected, it is set as a first-level abnormality; when abnormal tire signal data is detected, it is set as a second-level abnormality, and the priority of the first-level abnormality is higher than that of the second-level abnormality. The alarm distribution unit sends early warning strategies through multiple channels such as emails and text messages. For example, in the case of a first-level abnormality, the alarm distribution unit simultaneously uses multiple methods such as emails and text messages for early warning. In the case of a first-level abnormality, the alarm distribution unit simultaneously uses emails and text messages for early warning. The alarm suppression unit processes global abnormal data and local abnormal data to filter out duplicate and cascading alarms, thereby avoiding alarm storms.

[0102] In this embodiment, the abnormal data detection device can be deployed in an abnormal data detection system, and one or more vehicles are communicatively connected to the abnormal data detection system. After the abnormal data detection device determines the global abnormal data and local abnormal data, the processor processes and analyzes the global abnormal data and local abnormal data, and controls the alarm module to give early warnings about the abnormal conditions of the vehicle through corresponding notification methods according to the results of the processing and analysis.

[0103] Please refer to Figure 4 , Figure 4 which is a schematic hardware structure diagram of the electronic device 1000 provided in the embodiment of the present application. As Figure 4 shown, the electronic device 1000 may include a processor 1001 and a memory 1002. The memory 1002 is used to store one or more computer programs 1003. The one or more computer programs 1003 are configured to be executed by the processor 1001. The one or more computer programs 1003 include instructions, and the above instructions can be used to implement the above-mentioned method in the electronic device 1000.

[0104] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 1000. In other embodiments, the electronic device 1000 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements.

[0105] The processor 1001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1001 is a cache memory. This memory can store the instructions or data that the processor 1001 has just used or recycled. If the processor 1001 needs to use the instruction or data again, it can directly call it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 1001, and thus improves the efficiency of the system.

[0106] In some embodiments, the processor 1001 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0107] In some embodiments, the processor 1001 is used to execute acceleration schemes such as single instruction multiple data (SIMD) and very long instruction word (VLIW).

[0108] In some embodiments, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0109] This embodiment also provides a computer-readable storage medium. Computer instructions are stored in the computer-readable storage medium. When the instructions are run on an electronic device, the electronic device is caused to execute the above-related method steps to implement the method in the above embodiment.

[0110] Among them, the electronic device and the storage medium provided in this embodiment are both used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0111] In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0112] In several embodiments provided in the present application, the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are illustrative. For example, the division of the module or unit is a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0113] The unit described as a separate component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it can be located in one place, or it can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0115] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other various media that can store program codes.

[0116] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any change or replacement within the technical scope disclosed in the present application should be covered by the protection scope of the present application.

Claims

1. A method for detecting abnormal data, characterized in that: The method comprises: Receiving first test data in the sliding window; Performing anomaly detection on the first data to be tested to obtain global anomaly data, wherein the global anomaly data is outlier data in the overall distribution of the first data to be tested; Eliminating the global abnormal data in the first data to be tested to obtain second data to be tested; Performing anomaly detection on the second data to be tested to obtain local anomaly data, wherein the local anomaly data is outlier data in the local distribution of the second data to be tested; The global anomaly data and the local anomaly data are taken as the target anomaly data.

2. The abnormal data detection method according to claim 1, characterized in that: The performing anomaly detection on the first data to be tested to obtain global anomaly data includes: Calculating the path length of each data in the first test data in a pre-constructed isolation tree, where the isolation tree is a binary tree constructed based on an isolation forest algorithm; Calculating an anomaly score of each data in the first test data based on the plurality of path lengths; Based on a plurality of the anomaly scores, the global anomaly data is determined.

3. The abnormal data detection method according to claim 2, characterized in that: The performing anomaly detection on the second data to be tested to obtain local anomaly data includes: Based on a preset distance threshold, determine a neighborhood data set of the first data in the second data to be tested, wherein the first data is any one of the second data to be tested; Based on the neighborhood data set, calculating the local reachable density of the first data; Based on the local reachable density, calculating a local anomaly factor of the first data; When the local abnormality factor is greater than an abnormality threshold, the first data is used as the local abnormal data.

4. The abnormal data detection method according to claim 3, characterized in that: The calculating the local reachable density of the first data based on the neighborhood data set includes: Calculate the reachable distance between each data in the neighborhood data set and the first data; The reciprocal of an average value of the plurality of reachable distances is taken as the local reachable density.

5. The abnormal data detection method according to claim 1, characterized in that: Before performing anomaly detection on the first data to be tested to obtain global anomaly data, the method further includes: Detecting whether there is a preset threshold value matching the first data to be tested, wherein the first data to be tested includes a plurality of signal data, and the plurality of signal data and the preset threshold value correspond one to one; The performing anomaly detection on the first data to be tested to obtain global anomaly data includes: When it is detected that there is no abnormality data that is different from the preset threshold, an abnormality detection is performed on the first data to be tested to obtain the global abnormal data; When the preset threshold is detected, anomaly detection is performed on the first data to be tested based on the preset threshold to obtain the target abnormal data.

6. The abnormal data detection method according to claim 3, characterized in that: The first data to be tested includes multiple signal data; After receiving the first data to be tested in the sliding window, the method further includes: Based on the multiple signal data, construct a signal association graph, the signal association graph includes multiple nodes, the multiple nodes correspond to the multiple signal data one by one, and the signal association graph represents the association relationship between the multiple nodes; After determining the global abnormal data based on the plurality of abnormal scores, the method further comprises: Determine, from the signal association graph, a first node associated with the node corresponding to each data in the global abnormal data; In a case where the signal data corresponding to the first node is the outlier data in the overall distribution of the first data to be tested, adding the signal data corresponding to the first node to the global abnormal data; After taking the first data as the local abnormal data when the local abnormal factor is greater than the abnormal threshold, the method further comprises: Determine, from the signal association graph, a second node associated with the node corresponding to each data in the local abnormal data; In a case where the signal data corresponding to the second node is the outlier data in the local distribution of the second data to be tested, the signal data corresponding to the second node is added to the local abnormal data.

7. The abnormal data detection method according to claim 3, characterized in that: The first data to be tested includes multiple signal data; After receiving the first data to be tested in the sliding window, the method further includes: Based on the multiple signal data, construct a signal association graph, the signal association graph includes multiple nodes, the multiple nodes correspond to the multiple signal data one by one, and the signal association graph represents the association relationship between the multiple nodes; After determining the global abnormal data based on the plurality of abnormal scores, the method further comprises: Determine, from the signal association graph, a first node associated with the node corresponding to each data in the global abnormal data; In a case where the signal data corresponding to the first node is the outlier data in the overall distribution of the first data to be tested, adding the signal data corresponding to the first node to the global abnormal data; After taking the first data as the local abnormal data when the local abnormal factor is greater than the abnormal threshold, the method further comprises: Determine, from the signal association graph, a second node associated with the node corresponding to each data in the local abnormal data; In a case where the signal data corresponding to the second node is the outlier data in the local distribution of the second data to be tested, the signal data corresponding to the second node is added to the local abnormal data.

8. The abnormal data detection method according to claim 6, characterized in that: Each type of signal data is collected by a collection component; The constructing the signal association graph based on the multiple signal data includes: Calculating an embedding vector for each of the acquisition components; Based on the multiple embedding vectors, weight data is obtained, wherein the weight data represents the relationship weight between the multiple acquisition components; Extracting characteristic data of a plurality of signal data based on the weight data; Based on the multiple characteristic data, the signal association graph is constructed.

9. The abnormal data detection method according to claim 7, characterized in that: Each type of signal data is collected by a collection component; The constructing the signal association graph based on the multiple signal data includes: Calculating an embedding vector for each of the acquisition components; Based on the multiple embedding vectors, weight data is obtained, wherein the weight data represents the relationship weight between the multiple acquisition components; Extracting characteristic data of a plurality of signal data based on the weight data; Based on the multiple characteristic data, the signal association graph is constructed.

10. The abnormal data detection method according to claim 8, characterized in that: The signal association diagram is constructed based on the multiple characteristic data, including: Aggregate the multiple characteristic data to obtain aggregated data, wherein the signal data corresponding to each node in the aggregated data includes characteristic information of the signal data corresponding to neighboring nodes; Based on the aggregated data, the signal association graph is constructed.

11. The abnormal data detection method according to claim 9, characterized in that: The signal association diagram is constructed based on the multiple characteristic data, including: Aggregate the multiple characteristic data to obtain aggregated data, wherein the signal data corresponding to each node in the aggregated data includes characteristic information of the signal data corresponding to neighboring nodes; Based on the aggregated data, the signal association graph is constructed.

12. An abnormal data detection device, characterized in that: include: A data receiving module, the data receiving module is used to receive the first test data in the sliding window; A data processing module, the data processing module is used to perform anomaly detection on the first data to be tested to obtain global anomaly data, wherein the global anomaly data is the outlier data of the first data to be tested in the overall distribution, remove the global anomaly data in the first data to be tested to obtain the second data to be tested, and perform anomaly detection on the second data to be tested to obtain local anomaly data, wherein the local anomaly data is the outlier data of the second data to be tested in the local distribution, and use the global anomaly data and the local anomaly data as target anomaly data.

13. An electronic device, characterized in that: The electronic device comprises a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the abnormal data detection method according to any one of claims 1 to 11.

14. A computer storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the abnormal data detection method according to any one of claims 1 to 11.

Images