Abnormal data detection method, system, platform, electronic device and storage medium

Abstract

This invention provides an anomaly detection method, system, platform, electronic device, and storage medium. The method includes: acquiring features of a signal to be detected; constructing an isolated forest detector based on the features, the isolated forest detector comprising multiple isolated forests, each corresponding to a preset dimension; combining the features to be detected and traversing the isolated forests of the corresponding preset dimensions to obtain anomaly data for each preset dimension, each anomaly data carrying a corresponding detector identifier; performing an intersection operation on each multidimensional anomaly data with all one-dimensional anomaly data to obtain the one-dimensional anomaly data corresponding to the multidimensional anomaly data; and updating the detector identifier of the multidimensional anomaly data to the detector identifier of the corresponding one-dimensional anomaly data. This method achieves rapid detection of anomaly data and determination of the anomaly source, i.e., the detector identifier can identify the anomaly source with high accuracy.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to an abnormal data detection method, system, platform, electronic device, and storage medium. Background Technology

[0002] With the continuous development of the Internet of Things (IoT), more and more devices are connecting to the IoT, resulting in a massive amount of data. IoT data, such as signal data from vehicle-to-everything (V2X) systems, commonly exhibits abnormal data phenomena such as jumps, default values, and missing values. The vehicles, signal sources, timing, and operating conditions that generate abnormal data vary, and the corresponding anomaly mechanisms are random, sporadic, and widespread.

[0003] Currently, most methods for detecting outliers employ common techniques, such as defining rules or using unsupervised algorithms to identify and filter out anomalous data. The resulting anomalous data is typically a feature set. This feature set includes multiple signal features, and it's uncertain which feature(s) are the source of the anomaly that led to the feature set being detected as abnormal. This makes it difficult to identify the root cause of the anomalous data and prevent its generation in the first place. Summary of the Invention

[0004] This invention provides an anomaly detection method, system, platform, electronic device, and storage medium to solve the problem that existing anomaly detection methods cannot effectively identify the source of anomalies in anomaly data, are not conducive to finding the root cause of anomalies, and prevent the generation of anomalies from the source.

[0005] This invention provides an abnormal data detection method, comprising:

[0006] Acquire the features of the signal to be detected;

[0007] Based on the features of the signal to be detected, an isolated forest detector is constructed. The isolated forest detector includes multiple isolated forests, and each isolated forest corresponds to a preset dimension.

[0008] The detected signal features and combinations thereof are traversed through the isolated forests of corresponding preset dimensions to obtain abnormal data for each preset dimension. Each abnormal data carries a corresponding detector identifier, which is a preset identifier that identifies the detected signal features or combinations thereof as an isolated forest of abnormal data. The detected signal features correspond to a one-dimensional isolated forest, and the combinations thereof are composed of any two or more types of detected signal features. The combinations thereof correspond to a multi-dimensional isolated forest.

[0009] Perform an intersection operation between each multidimensional abnormal data and all one-dimensional abnormal data to obtain the one-dimensional abnormal data corresponding to the multidimensional abnormal data, and update the detector identifier of the multidimensional abnormal data to the detector identifier of the corresponding one-dimensional abnormal data.

[0010] Optionally, the steps for obtaining the features of the signal to be detected include:

[0011] Obtain message data reported by multiple vehicles;

[0012] Based on a preset message parsing protocol, the message data is parsed to obtain the vehicle signal;

[0013] Based on the vehicle type corresponding to the vehicle signal, the multiple vehicle signals are classified.

[0014] For any vehicle type, the vehicle signals of the same vehicle are combined from multiple sources according to a preset merging period to obtain at least one merged signal dataset for each vehicle. The merged signal dataset includes all vehicle signals reported by multiple sources of a vehicle within one merging period.

[0015] The vehicle signals of the preset target information source in the merged signal dataset are identified as the signals to be detected;

[0016] The signal to be detected is normalized to obtain the features of the signal to be detected.

[0017] Optionally, the step of constructing an isolated forest detector based on the features of the signal to be detected includes:

[0018] Based on all the features of the signal to be detected, feature sampling of a preset dimension is performed to obtain multiple sampled feature sets for each preset dimension.

[0019] Based on the sampling feature set, an isolated subtree corresponding one-to-one with the sampling feature set is constructed. Multiple isolated subtrees of each preset dimension form an isolated forest of the corresponding preset dimension. The number of isolated forests of each preset dimension is determined based on the corresponding preset dimension and the number of types of the signal features to be detected.

[0020] All of the isolated forests constitute the isolated forest detector.

[0021] Optionally, the step of performing feature sampling of a preset dimension based on all the features of the signal to be detected, to obtain multiple sampled feature sets for each preset dimension, includes:

[0022] When the preset dimension includes one-dimensional, two-dimensional and target dimensions, one-dimensional feature random sampling is performed based on any type of the signal feature to be detected to obtain multiple one-dimensional sampled feature sets, wherein the one-dimensional sampled feature set includes a first preset number of one-dimensional signal features to be detected.

[0023] Based on any two types of the signal features to be detected, two-dimensional feature random sampling is performed to obtain multiple two-dimensional sampled feature sets. The two-dimensional sampled feature sets include a second preset number of two-dimensional combinations of the signal features to be detected. The two-dimensional signal features to be detected are composed of any two types of the signal features to be detected.

[0024] Based on all the features of the signal to be detected, random sampling of the target dimension features is performed to obtain multiple target dimension sampled feature sets. The target dimension sampled feature sets include a third preset number of combinations of the signal features to be detected in the target dimension, and the combinations of the signal features to be detected in the target dimension are composed of all types of the signal features to be detected.

[0025] Optionally, the step of constructing isolated subtrees corresponding one-to-one with the sampled feature set includes:

[0026] A sampling point is randomly selected from the sampling feature set as the cutting node;

[0027] Based on the cutting node, sampling points with values ​​less than the cutting node are determined as the left child node of the cutting node, and sampling points with values ​​greater than or equal to the cutting node are determined as the right child node of the cutting node.

[0028] Based on the left child node and the right child node, the process is gradually cut until the cutting is completed or the height of the child node has reached the preset limit height, thus completing the construction of the corresponding isolated subtree.

[0029] Optionally, it also includes:

[0030] Based on a preset clustering model, all the abnormal data are clustered to obtain the abnormal data with cluster identifiers.

[0031] The data anomaly type is determined based on the vehicle signals corresponding to the abnormal data with the cluster identifier.

[0032] The present invention also provides an abnormal data detection system, comprising:

[0033] The signal feature acquisition module is used to acquire the features of the signal to be detected;

[0034] An isolated forest detector construction module is used to construct an isolated forest detector based on the features of the signal to be detected. The isolated forest detector includes multiple isolated forests, and each isolated forest corresponds to a preset dimension.

[0035] An abnormal data detection module is used to traverse the isolated forest corresponding to a preset dimension of the signal feature to be detected and the combination of signal features to be detected, respectively, to obtain abnormal data in each preset dimension. Each abnormal data carries a corresponding detector identifier, which is a preset identifier that determines the isolated forest of abnormal data of the signal feature to be detected or the combination of signal features to be detected. The signal feature to be detected corresponds to a one-dimensional isolated forest, and the combination of signal features to be detected is composed of any two or more types of signal features to be detected. The combination of signal features to be detected corresponds to a multi-dimensional isolated forest.

[0036] The anomaly source determination module is used to perform an intersection operation between each multidimensional anomaly data and all one-dimensional anomaly data to obtain the one-dimensional anomaly data corresponding to the multidimensional anomaly data, and update the detector identifier of the multidimensional anomaly data to the detector identifier of the corresponding one-dimensional anomaly data.

[0037] The present invention also provides an abnormal data detection platform, including: a display module, and an abnormal data detection system as described above;

[0038] The display module is used to display one-dimensional abnormal data, multi-dimensional abnormal data, and the source of the abnormality in the multi-dimensional abnormal data.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the abnormal data detection method as described above.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the abnormal data detection method as described above.

[0041] The beneficial effects of this invention are as follows: The abnormal data detection method, system, platform, electronic device, and storage medium provided by this invention acquire the features of the signal to be detected; based on the features of the signal to be detected, an isolated forest detector is constructed, which includes multiple isolated forests, each corresponding to a preset dimension; the features to be detected and combinations of the features to be detected are traversed through the isolated forests of the corresponding preset dimensions to obtain abnormal data for each preset dimension, and each abnormal data carries a corresponding detector identifier, which is a preset identifier that identifies the isolated forest containing the features to be detected or combinations of the features to be detected as abnormal data; the features to be detected correspond to a one-dimensional isolated forest, and the combinations of the features to be detected are composed of any two or more types of features to be detected, thus the combinations of the features to be detected correspond to a multi-dimensional isolated forest; each multi-dimensional abnormal data is intersected with all one-dimensional abnormal data to obtain the one-dimensional abnormal data corresponding to the multi-dimensional abnormal data, and the detector identifier of the multi-dimensional abnormal data is updated to the detector identifier of the corresponding one-dimensional abnormal data. This paper proposes a method to quickly detect anomalous data and determine its source using an isolated forest anomaly detection algorithm with linear time complexity within a distributed computing framework. Specifically, the detector identifier carried by each anomalous data item is used to identify its source of the anomaly, while maintaining high accuracy. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the abnormal data detection method provided by the present invention;

[0044] Figure 2 This is an exemplary flowchart of the abnormal data detection method provided by the present invention;

[0045] Figure 3 This is a schematic diagram of the abnormal data detection system provided by the present invention;

[0046] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] The following examples illustrate this approach. Figures 1-4 This invention describes the abnormal data detection method, system, platform, electronic device, and storage medium provided by the present invention.

[0049] Please refer to Figure 1 The abnormal data detection method provided in this embodiment includes:

[0050] S10: Obtain the features of the signal to be detected. Specifically, the features of the signal to be detected refer to the feature vector obtained by normalizing the vehicle signal to be detected. Obtaining the features of the signal to be detected facilitates the subsequent construction of an isolated forest detector and anomaly data detection.

[0051] S20: Based on the features of the signal to be detected, an isolated forest detector is constructed. The isolated forest detector includes multiple isolated forests, each corresponding to a preset dimension. That is, each preset dimension corresponds to one or more isolated forests, and the number of isolated forests in each preset dimension is determined based on the number of types of features of the signal to be detected. For example, the number of one-dimensional isolated forests is the same as the number of types of features of the signal to be detected, the number of two-dimensional isolated forests is the same as the number of combinations of every two types of features of the signal to be detected, that is, every two types of features of the signal to be detected are combined pairwise, and the number of combinations is the number of two-dimensional isolated forests, and so on.

[0052] This step involves constructing an isolated forest detector with multiple preset dimensions to detect anomalies in the signal features being tested across multiple dimensions. It's understandable that different signal features are often correlated, such as a vehicle's mileage and fuel consumption. If only a one-dimensional isolated forest is used for one-dimensional anomaly detection, some anomalies may be missed. For example, if a vehicle's mileage is 50 kilometers and its fuel consumption is 0 within the same merging period, performing one-dimensional anomaly detection on these two data points alone might fail to detect them. However, if these two data points are combined as a single signal feature and anomaly detection is performed using a two-dimensional isolated forest, this combination can be identified as anomaly data, thus achieving accurate anomaly detection.

[0053] In addition, compared to constructing a fixed-dimensional isolated forest and using it for anomaly detection, this step detects anomalies by examining multiple preset-dimensional isolated forests (one-dimensional isolated forest, two-dimensional isolated forest, ... N-dimensional isolated forest). The isolated forest detector, composed of multiple preset-dimensional isolated forests, can identify anomalies from multiple preset dimensions, which helps improve the accuracy of anomaly detection.

[0054] S30: The detected signal features and combinations thereof are traversed through the corresponding preset-dimensional isolated forests to obtain abnormal data for each preset-dimensional dimension. Each abnormal data carries a corresponding detector identifier, which is a preset identifier that identifies the detected signal features or combinations thereof as an isolated forest of abnormal data. The detected signal features correspond to a one-dimensional isolated forest, and the combinations thereof are composed of any two or more types of detected signal features. The combinations thereof correspond to multi-dimensional isolated forests. Specifically, one-dimensional detected signal features are traversed through all one-dimensional isolated forests, two-dimensional detected signal feature combinations are traversed through all two-dimensional isolated forests, three-dimensional detected signal feature combinations are traversed through all three-dimensional isolated forests, and so on, thereby obtaining abnormal data for each preset-dimensional isolated forest based on each preset-dimensional isolated forest.

[0055] Specifically, during the traversal of the isolated forest of the corresponding dimension for the signal feature or combination of signal features to be detected, it is necessary to traverse multiple isolated subtrees in each corresponding dimension of the isolated forest and record the path length. Based on the path length, it is determined whether the signal feature or combination of signal features to be detected is anomalous data. Generally, the shorter the path length, the greater the degree of anomalousness or the probability of anomality. Therefore, in specific implementation, a path length threshold can be set to determine whether the signal feature or combination of signal features to be detected is anomalous data. The path length threshold can be set according to the actual situation, such as 3, 5, etc. This yields anomalous data for each preset dimension. Each anomalous data carries a corresponding detector identifier, which is used to identify the source of the anomalous data. For example, if a certain anomalous data is determined by a one-dimensional isolated forest (the preset identifier of this isolated forest is mileage anomalous), then a detector identifier is assigned to this anomalous data, and the detector identifier is mileage anomalous, and so on.

[0056] In addition, the preset identifiers for each isolated forest can be set according to the actual situation, such as F01, F02, etc. Each preset identifier represents a different anomaly, such as F01 representing a mileage anomaly.

[0057] S40: Perform an intersection operation between each multidimensional abnormal data and all one-dimensional abnormal data to obtain the one-dimensional abnormal data corresponding to the multidimensional abnormal data. Update the detector identifier of the multidimensional abnormal data to the detector identifier of the corresponding one-dimensional abnormal data. Finally, multiple abnormal data are obtained, each corresponding to a preset dimension. Each abnormal data carries its own detector identifier, which is either an updated detector identifier (if it intersects with the one-dimensional abnormal data) or an unupdated detector identifier (if it does not intersect with the one-dimensional abnormal data, or is simply the one-dimensional abnormal data itself; understandably, the detector identifier of the one-dimensional abnormal data does not need to be updated).

[0058] In this embodiment, "multidimensional" refers to two or more dimensions. Specifically, the intersection operation is performed between the two-dimensional and higher-dimensional abnormal data and all one-dimensional abnormal data to determine the detectable signal features (i.e., one-dimensional abnormal data) causing the data anomalies in the two-dimensional and higher-dimensional abnormal data. The detector identifiers for the multidimensional abnormal data are then updated to the corresponding detector identifiers for the one-dimensional abnormal data. This facilitates the identification of the anomaly source for each piece of abnormal data by relevant personnel based on the detector identifier. Based on the anomaly source, corresponding maintenance and other operations can be performed, resulting in a high degree of automation. The above-described intersection operation step can determine the anomaly source of most multidimensional abnormal data, eliminating the need for manual anomaly source investigation, effectively reducing labor costs, and demonstrating strong feasibility.

[0059] As shown in Table 1 below, the abnormal data obtained in this embodiment includes:

[0060] Table 1 Abnormal Data

[0061]

[0062] In this table, Data1, Data2, etc., refer to the abnormal data corresponding to the preset dimensions, and F01, F04, etc., refer to the detector identifiers corresponding to the abnormal data. For example, F01 can indicate abnormal mileage, and F04 can indicate abnormal fuel consumption, etc. This embodiment ultimately yields multiple abnormal data sets carrying their respective detector identifiers, facilitating the identification of the source of each abnormal data set. It should be noted that Table 1 above is for clarity regarding the final output results in this embodiment. In actual implementation, the final output of abnormal data may not necessarily be in tabular form; it can also be in the form of images, etc., which will not be elaborated upon here.

[0063] In some embodiments, the step of acquiring the features of the signal to be detected includes:

[0064] S101: Obtain message data reported by multiple vehicles. The source of the message data uploaded by each vehicle can be multiple controllers in the vehicle, such as ECU (Electronic Control Unit) and VCU (Vehicle Control Unit).

[0065] S102: Based on a preset message parsing protocol, the message data is parsed to obtain the vehicle signal. The message parsing protocol may be CAN (Controller Area Network, serial communication protocol), etc.

[0066] S103: Based on the vehicle type corresponding to the vehicle signal, classify the multiple vehicle signals. It should be noted that different vehicle types, such as cranes, pile drivers, mixers, excavators, or engineering vehicles such as aerial work platforms, fire trucks, and mixer trucks, upload different types of message data. Therefore, it is necessary to classify the vehicle signals uploaded by multiple vehicles, that is, classify the vehicle signals according to vehicle type. For different types of vehicle signals, different isolated forest anomaly detectors are used to detect abnormal data. There is a one-to-one correspondence between the vehicle signal type and the isolated forest anomaly detector; that is, each vehicle type corresponds to one isolated forest anomaly detector.

[0067] S104: For vehicle signals of any vehicle type, according to a preset merging period, the vehicle signals of the same vehicle are merged from multiple sources to obtain at least one merged signal dataset for each vehicle. The merged signal dataset includes all vehicle signals reported by multiple sources of a vehicle within one merging period. Specifically, since the message data uploaded by multiple sources of the same vehicle within the same merging period (e.g., 30 seconds) may have temporal differences, the message data uploaded by multiple sources within the same merging period may be discretely distributed. To facilitate subsequent anomaly detection, merging vehicle signals of the same vehicle within the same merging period allows for the selection of vehicle signals from preset target sources within the same merging period from the merged signal dataset before anomaly detection. This enables the sampling of corresponding dimension signal features or combinations of signal features to be detected within the same merging period from the vehicle signals of the preset target sources within the same merging period, thus improving the accuracy of subsequent anomaly detection.

[0068] S105: The vehicle signals from the preset target sources in the merged signal dataset are identified as signals to be detected. Specifically, the merged signal dataset includes vehicle signals uploaded by multiple sources for the same vehicle. Typically, only some of the vehicle signals from these multiple sources are needed for anomaly detection. Therefore, by filtering out the vehicle signals uploaded by the target source from the merged signal dataset and identifying them as signals to be detected, the accuracy of subsequent anomaly detection can be improved.

[0069] S106: Normalize the signal to be detected to obtain the signal features. Specifically, the normalization process can employ methods such as Min-Max (linear function normalization). By normalizing the signal to be detected, different types of signals to be detected are converted into signal features of a unified format, which facilitates the subsequent construction of an isolated forest detector based on the signal features.

[0070] In some embodiments, the step of constructing an isolated forest detector based on the features of the signal to be detected includes:

[0071] S201: Based on all the features of the signal to be detected, feature sampling of a preset dimension is performed to obtain multiple sampled feature sets for each preset dimension. That is, feature sampling of all the features of the signal to be detected is performed in multiple dimensions such as one-dimensional and two-dimensional to obtain sampled feature sets in multiple dimensions. Feature sampling can be performed using random sampling.

[0072] S202: Based on the sampled feature set, construct isolated subtrees corresponding one-to-one with the sampled feature set. Multiple isolated subtrees of each preset dimension form an isolated forest of the corresponding preset dimension. The number of isolated forests of each preset dimension is determined based on the corresponding preset dimension and the number of types of the signal features to be detected. All isolated forests form the isolated forest detector. This step, by constructing isolated forests of multiple dimensions and combining all isolated forests into an isolated forest detector for the current type of vehicle, achieves high accuracy.

[0073] In some embodiments, the step of performing feature sampling of a preset dimension based on all the features of the signal to be detected to obtain multiple sampled feature sets for each preset dimension includes:

[0074] S2011: When the preset dimension includes one-dimensional, two-dimensional, and target dimensions, based on any type of the signal feature to be detected, one-dimensional feature random sampling is performed to obtain multiple one-dimensional sampled feature sets. Each one-dimensional sampled feature set includes a first preset number of one-dimensional signal features to be detected. It should be noted that the number of one-dimensional isolated forests is the same as the number of types of signal features to be detected; that is, there is a one-to-one correspondence between one-dimensional isolated forests and the types of signal features to be detected. Each one-dimensional isolated forest includes multiple one-dimensional isolated subtrees (first one-dimensional isolated subtree, second one-dimensional isolated subtree, ..., R-th one-dimensional isolated subtree). Each one-dimensional isolated subtree requires random sampling of a first preset number of signal features to be detected from the corresponding type of signal features to form a one-dimensional sampled feature set. Then, based on the one-dimensional sampled feature sets, the corresponding isolated subtrees are constructed. The above one-dimensional feature random sampling steps are repeated until all one-dimensional isolated forests are constructed.

[0075] For example: From 10,000 target type signal features to be detected, a first random sample is taken, resulting in 1,000 target signal features. These 1,000 features are defined as a one-dimensional sampled feature set. Then, from these 10,000 features, a second random sample is taken, resulting in another 1,000 target signal features. This 1,000 features are defined as another one-dimensional sampled feature set, and so on. The number of one-dimensional sampled feature sets can be set according to actual needs, such as 10, 12, etc. This completes the construction of a one-dimensional isolated forest. Then, using the same method, a first random sample is taken from the target type signal features, followed by a second random sample, and so on, progressively until all one-dimensional isolated forests are constructed.

[0076] S2012: Based on any two types of the signal features to be detected, perform two-dimensional feature random sampling to obtain multiple two-dimensional sampled feature sets. Each two-dimensional sampled feature set includes a second preset number of two-dimensional combinations of the signal features to be detected. Specifically, a two-dimensional feature refers to a feature formed by combining any two signal features from all the signal features to be detected, i.e., a two-dimensional combination of signal features to be detected. The specific steps for obtaining multiple two-dimensional sampled feature sets through two-dimensional feature random sampling are similar to those for one-dimensional feature random sampling, except that each time two-dimensional feature random sampling is performed, it extracts a two-dimensional combination of signal features to be detected. For example, randomly sample 1000 combinations of signal features to be detected from 10000 combinations containing two types of signal features to be detected, and determine these 1000 combinations as a two-dimensional sampled feature set. It should be noted that the number of two-dimensional isolated forests is the same as the number of combinations of the signal features to be detected in pairs. If the features to be detected include three types: A, B, and C, then the types of two-dimensional feature combinations include [A, B], [A, C], and [B, C]. Each combination corresponds to a two-dimensional isolated forest. Each two-dimensional isolated forest includes multiple two-dimensional isolated subtrees (first two-dimensional isolated subtree, second two-dimensional isolated subtree, ..., R-th two-dimensional isolated subtree). The construction of each two-dimensional isolated subtree is completed using the above-mentioned two-dimensional feature random sampling method.

[0077] S2013: Based on all the features of the signal to be detected, random sampling of the target dimension features is performed to obtain multiple target dimension sampled feature sets. Each target dimension sampled feature set includes a third preset number of combinations of the signal features to be detected in the target dimension. These combinations consist of all types of the signal features to be detected. In this embodiment, the number of target dimensions and types of signal features to be detected are the same; that is, all or all types of signal features to be detected constitute the combination of signal features to be detected in the target dimension. The number of target dimension isolated forests is one. A target dimension isolated forest includes multiple target dimension isolated subtrees (first target dimension isolated subtree, second target dimension isolated subtree, ..., Rth target dimension isolated subtree). The construction of the target dimension isolated subtrees uses random sampling of target dimension features. That is, from all the signal features to be detected, one of each type of signal feature is randomly selected to obtain combinations of all types of signal features to be detected. By selecting a third preset number of combinations of all types of signal features to be detected, a target dimension sampled feature set is obtained. Multiple target dimension sampling feature sets are obtained by random sampling of target dimension features multiple times. Then, based on each target dimension sampling feature set, a target dimension isolated subtree corresponding one-to-one with the target dimension sampling feature set is constructed.

[0078] It should be noted that the steps of obtaining the one-dimensional sampling feature set, the two-dimensional sampling feature set, and the target dimension sampling feature set, namely steps S2011, S2012, and S2011, can be performed simultaneously without any time order.

[0079] Furthermore, the step of constructing isolated subtrees corresponding one-to-one with the sampling feature set includes:

[0080] S2021: Randomly select a sampling point from the sampled feature set as a cutting node. Specifically, based on the randomly selected sampling point, generate a hyperplane that divides the data space into two subspaces. Based on these two subspaces, establish left and right child nodes. A sampling point refers to the extracted feature or combination of features to be detected.

[0081] S2022: Based on the cutting node, sampling points with values ​​less than the cutting node are determined as the left child node of the cutting node, and sampling points with values ​​greater than or equal to the cutting node are determined as the right child node of the cutting node.

[0082] S2023: Based on the left and right child nodes, perform progressive segmentation until segmentation is complete or the height of the child nodes reaches a preset limit, thus completing the construction of the corresponding isolated subtree. The limit height can be set according to actual conditions, such as 8 levels. Segmentation completion means that the left and right child nodes can no longer be divided, that is, the left child node is the smallest segmentation unit, such as a single signal feature to be detected or a combination of signal features to be detected. This completes the construction of the isolated subtree.

[0083] To facilitate finding the root cause or origin of anomalous data, this embodiment proposes clustering of anomalous data, specifically including:

[0084] a. Based on a preset clustering model, all the abnormal data are clustered to obtain abnormal data with cluster identifiers. Specifically, the clustering model can be a clustering model such as DBSCAN (an unsupervised clustering model). By using a clustering model to cluster all the abnormal data and obtaining abnormal data with cluster identifiers, it is convenient to determine the total number of abnormal data categories based on the cluster identifiers, and to trace the root cause of the abnormal data for each category, thereby fundamentally solving the cause of the abnormal data. The cluster identifiers can be 01, 02, 03, etc., as long as they can easily identify the category of abnormal data.

[0085] For example, as shown in Table 2 below, after clustering all the abnormal data, the abnormal data finally obtained in this embodiment includes:

[0086] Table 2 Abnormal Data

[0087]

[0088] Among them, D01, D02, etc. refer to different anomaly categories.

[0089] b. Determine the data anomaly type based on the vehicle signals corresponding to the abnormal data bearing the cluster identifier. That is, by identifying the vehicle signals associated with the abnormal data bearing the cluster identifier, the corresponding unnormalized vehicle signals are obtained. Based on these vehicle signals, the data anomaly type is determined, such as abnormal fuel consumption. Furthermore, the vehicle source of the abnormal vehicle signal can be determined based on the Vehicle Identification Number (VIN) in the vehicle signal, facilitating timely adjustment or repair of the corresponding vehicle by relevant personnel.

[0090] To facilitate understanding of the abnormal data detection method provided in the above embodiments, an exemplary embodiment will be used to further illustrate the abnormal data detection method below.

[0091] Please refer to Figure 2 Step 1: First, obtain the message data reported by multiple vehicles (x1, x2, x3, x4, ... x t Where t represents the number of message data. Next, based on a preset message parsing protocol, the message data is parsed to obtain vehicle signals. Then, based on the vehicle type corresponding to the vehicle signal, multiple vehicle signals are classified (vehicle type classification). Afterwards, for vehicle signals of any vehicle type, according to a preset merging period, multi-source signals of the same vehicle are merged to obtain at least one merged signal dataset for each vehicle. The merged signal dataset includes all vehicle signals reported by multiple sources of a vehicle within one merging period. Then, target source vehicle signal identification and normalization processing is performed, that is, the vehicle signals of the preset target sources in the merged signal dataset are determined as signals to be detected, and the signals to be detected are normalized to obtain the features of the signals to be detected.

[0092] Step 2: First, based on the features of the signal to be detected, an isolated forest detector is constructed. The isolated forest detector comprises multiple isolated forests, including multiple one-dimensional isolated forests, two-dimensional isolated forests, and a target-dimensional isolated forest. The number of one-dimensional isolated forests is the same as the number of types of features to be detected, and the number of two-dimensional isolated forests is the same as the number of combinations of pairs of features to be detected. The total number of isolated forests, h = n. (n-1) / 2+n+1, where n represents the number of types of signal features to be detected, and the first term n in the formula... (n-1) / 2 represents the number of two-dimensional isolated forests, the second term n represents the number of one-dimensional isolated forests, and the third term 1 represents the number of isolated forests in the target dimension. For the construction steps of the isolated forest detector, please refer to S2011-S2013 (random sampling of one-dimensional features, random sampling of two-dimensional features, and random sampling of target dimension features) and S2021-S2023, which will not be repeated here. Then, the one-dimensional and two-dimensional features to be detected are combined, and the target dimension features are combined, traversing the isolated forests of the corresponding preset dimensions to obtain one-dimensional abnormal data, two-dimensional abnormal data, and target dimension abnormal data. The one-dimensional, two-dimensional, and target dimension abnormal data are assigned an isolated forest detector identifier to obtain abnormal data with the isolated forest detector identifier, i.e., one-dimensional, two-dimensional, and target dimension abnormal data with the isolated forest detector identifier. Each two-dimensional abnormal data and the target dimension abnormal data are intersected with all the one-dimensional abnormal data to obtain the corresponding one-dimensional abnormal data for each of the two-dimensional abnormal data and the target dimension abnormal data. The corresponding one-dimensional abnormal data is determined as the source of the abnormality of the two-dimensional abnormal data or the target dimension abnormal data.

[0093] Step 3: Based on the preset DBSCAN clustering model, perform cluster detection on all the abnormal data. Assign cluster labels to the classified abnormal data to obtain abnormal data with cluster labels, which facilitates subsequent identification of the abnormal data categories.

[0094] The abnormal data detection system provided by the present invention is described below. The abnormal data detection system described below can be referred to in correspondence with the abnormal data detection method described above.

[0095] Please refer to Figure 3 The abnormal data detection system provided in this embodiment includes:

[0096] The signal feature acquisition module 310 is used to acquire the features of the signal to be detected.

[0097] The isolated forest detector construction module 320 is used to construct an isolated forest detector based on the features of the signal to be detected. The isolated forest detector includes multiple isolated forests, and each isolated forest corresponds to a corresponding preset dimension.

[0098] An anomaly detection module 330 is used to traverse the isolated forest corresponding to a preset dimension by taking the signal feature to be detected and the combination of signal features to be detected, respectively, to obtain anomaly data in each preset dimension. Each anomaly data carries a corresponding detector identifier, which is a preset identifier that determines the isolated forest of anomaly data by the signal feature to be detected or the combination of signal features to be detected. The signal feature to be detected corresponds to a one-dimensional isolated forest, and the combination of signal features to be detected is composed of any two or more types of signal features to be detected. The combination of signal features to be detected corresponds to a multi-dimensional isolated forest.

[0099] The anomaly source determination module 340 is used to perform intersection operations on each multidimensional anomaly data with all one-dimensional anomaly data to obtain the one-dimensional anomaly data corresponding to the multidimensional anomaly data, and update the detector identifier of the multidimensional anomaly data to the detector identifier of the corresponding one-dimensional anomaly data. The signal feature acquisition module 310, the isolated forest detector construction module 320, the anomaly data detection module 330, and the anomaly source determination module 340 are connected. The anomaly data detection system in this embodiment achieves high accuracy in determining the anomaly source of anomaly data, avoids the need for manual screening of anomaly data to determine the anomaly source, and has high accuracy, low cost, and high feasibility.

[0100] In some embodiments, the signal feature acquisition module 310 is specifically used to acquire message data reported by multiple vehicles;

[0101] Based on a preset message parsing protocol, the message data is parsed to obtain the vehicle signal;

[0102] Based on the vehicle type corresponding to the vehicle signal, the multiple vehicle signals are classified.

[0103] For any vehicle type, the vehicle signals of the same vehicle are combined from multiple sources according to a preset merging period to obtain at least one merged signal dataset for each vehicle. The merged signal dataset includes all vehicle signals reported by multiple sources of a vehicle within one merging period.

[0104] The vehicle signals of the preset target information source in the merged signal dataset are identified as the signals to be detected;

[0105] The signal to be detected is normalized to obtain the features of the signal to be detected.

[0106] In some embodiments, the isolated forest detector building module 320 includes:

[0107] The sampling unit is used to perform feature sampling of a preset dimension based on all the features of the signal to be detected, so as to obtain multiple sampled feature sets for each preset dimension.

[0108] The construction unit is used to construct isolated subtrees that correspond one-to-one with the sampled feature set based on the sampled feature set. Multiple isolated subtrees of each preset dimension form an isolated forest of the corresponding preset dimension. The number of isolated forests of each preset dimension is determined based on the corresponding preset dimension and the number of types of the signal features to be detected. All isolated forests form the isolated forest detector.

[0109] In some embodiments, the sampling unit is specifically used to perform one-dimensional feature random sampling based on any type of the signal feature to be detected when the preset dimension includes one-dimensional, two-dimensional and target dimensions, to obtain multiple one-dimensional sampled feature sets, wherein the one-dimensional sampled feature sets include a first preset number of one-dimensional signal features to be detected.

[0110] Based on any two types of the signal features to be detected, two-dimensional feature random sampling is performed to obtain multiple two-dimensional sampled feature sets. The two-dimensional sampled feature sets include a second preset number of two-dimensional combinations of the signal features to be detected. The two-dimensional signal features to be detected are composed of any two types of the signal features to be detected.

[0111] Based on all the features of the signal to be detected, random sampling of the target dimension features is performed to obtain multiple target dimension sampled feature sets. The target dimension sampled feature sets include a third preset number of combinations of the signal features to be detected in the target dimension, and the combinations of the signal features to be detected in the target dimension are composed of all types of the signal features to be detected.

[0112] In some embodiments, the construction unit is specifically used to randomly select a sampling point from the sampling feature set as a cutting node;

[0113] Based on the cutting node, sampling points with values ​​less than the cutting node are determined as the left child node of the cutting node, and sampling points with values ​​greater than or equal to the cutting node are determined as the right child node of the cutting node.

[0114] Based on the left child node and the right child node, the process is gradually cut until the cutting is completed or the height of the child node has reached the preset limit height, thus completing the construction of the corresponding isolated subtree.

[0115] In some embodiments, the system further includes: a clustering module, configured to cluster all the abnormal data based on a preset clustering model to obtain the abnormal data with cluster identifiers;

[0116] The data anomaly type is determined based on the vehicle signals corresponding to the abnormal data with the cluster identifier.

[0117] This embodiment also provides an abnormal data detection platform, including: a display module, and an abnormal data detection system as described above;

[0118] The display module is used to display one-dimensional abnormal data, multi-dimensional abnormal data, and the source of the abnormality in the multi-dimensional abnormal data.

[0119] It should be noted that the steps for the anomaly data detection system to acquire vehicle signals include: the vehicle's onboard TBox (gateway device) transmits the vehicle's message data to the gateway device of the anomaly data detection platform in real time. The gateway device of the anomaly data detection platform parses the binary message of location information uploaded by the TBox into a recognizable vehicle signal and uploads the vehicle signal to the message queue (such as Kafka) of the anomaly data detection platform. Based on this message queue, the vehicle signal is acquired. This allows for subsequent classification and multi-source signal merging processing of the vehicle signal.

[0120] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute an abnormal data detection method. The method includes: acquiring the features of a signal to be detected; constructing an isolated forest detector based on the features of the signal to be detected, wherein the isolated forest detector includes multiple isolated forests, each of which corresponds to a preset dimension; traversing the isolated forests of the corresponding preset dimensions by combining the features of the signal to be detected and combinations thereof to obtain abnormal data of each preset dimension, wherein each abnormal data carries a corresponding detector identifier, the detector identifier being a preset identifier that identifies the isolated forest of the signal to be detected or combinations thereof as abnormal data; the signal to be detected corresponds to a one-dimensional isolated forest, and the combination of the signal to be detected consists of any two or more types of signal to be detected features, the combination of the signal to be detected corresponding to a multi-dimensional isolated forest; performing an intersection operation on each multi-dimensional abnormal data with all one-dimensional abnormal data to obtain the one-dimensional abnormal data corresponding to the multi-dimensional abnormal data; and updating the detector identifier of the multi-dimensional abnormal data to the detector identifier of the corresponding one-dimensional abnormal data.

[0121] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the abnormal data detection method provided by the above methods. The method includes: acquiring a signal feature to be detected; constructing an isolated forest detector based on the signal feature to be detected, the isolated forest detector including multiple isolated forests, each isolated forest corresponding to a corresponding preset dimension; traversing the isolated forests of the corresponding preset dimensions by combining the signal feature to be detected and combinations of the signal feature to be detected, obtaining abnormal data of each preset dimension, each abnormal data carrying a corresponding detector identifier, the detector identifier referring to a preset identifier of the isolated forest that determines the signal feature to be detected or the combination of the signal feature to be detected as abnormal data; the signal feature to be detected corresponds to a one-dimensional isolated forest, the combination of the signal feature to be detected is composed of any two or more types of signal features to be detected, and the combination of the signal feature to be detected corresponds to a multi-dimensional isolated forest; performing an intersection operation on each multi-dimensional abnormal data with all one-dimensional abnormal data to obtain the one-dimensional abnormal data corresponding to the multi-dimensional abnormal data; and updating the detector identifier of the multi-dimensional abnormal data to the detector identifier of the corresponding one-dimensional abnormal data.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An abnormal data detection method, characterized in that, Applications in vehicle-to-everything (V2X) communication include: Acquire the features of the signal to be detected; the features of the signal to be detected refer to the feature vector obtained by normalizing the vehicle signal to be detected; the features of the signal to be detected include mileage and fuel consumption; Based on the features of the signal to be detected, an isolated forest detector is constructed. The isolated forest detector includes multiple isolated forests, and each isolated forest corresponds to a preset dimension. The detected signal features and combinations thereof are traversed through the isolated forests of corresponding preset dimensions to obtain abnormal data for each preset dimension. Each abnormal data carries a corresponding detector identifier, which is a preset identifier that identifies the detected signal features or combinations thereof as an isolated forest of abnormal data. The detected signal features correspond to a one-dimensional isolated forest, and the combinations thereof are composed of any two or more types of detected signal features. The combinations thereof correspond to a multi-dimensional isolated forest. Perform an intersection operation between each multidimensional abnormal data and all one-dimensional abnormal data to obtain the one-dimensional abnormal data corresponding to the multidimensional abnormal data, and update the detector identifier of the multidimensional abnormal data to the detector identifier of the corresponding one-dimensional abnormal data.

2. The abnormal data detection method according to claim 1, characterized in that, The steps for obtaining the features of the signal to be detected include: Obtain message data reported by multiple vehicles; Based on a preset message parsing protocol, the message data is parsed to obtain the vehicle signal; Based on the vehicle type corresponding to the vehicle signal, the multiple vehicle signals are classified. For any vehicle type, the vehicle signals of the same vehicle are combined from multiple sources according to a preset merging period to obtain at least one merged signal dataset for each vehicle. The merged signal dataset includes all vehicle signals reported by multiple sources of a vehicle within one merging period. The vehicle signals of the preset target information source in the merged signal dataset are identified as the signals to be detected; The signal to be detected is normalized to obtain the features of the signal to be detected.

3. The abnormal data detection method according to claim 1, characterized in that, The steps for constructing an isolated forest detector based on the characteristics of the signal to be detected include: Based on all the features of the signal to be detected, feature sampling of a preset dimension is performed to obtain multiple sampled feature sets for each preset dimension. Based on the sampled feature set, isolated subtrees corresponding one-to-one with the sampled feature set are constructed. Multiple isolated subtrees of each preset dimension form an isolated forest of the corresponding preset dimension. The number of isolated forests of each preset dimension is determined based on the corresponding preset dimension and the number of types of the signal features to be detected. All isolated forests form the isolated forest detector.

4. The abnormal data detection method according to claim 3, characterized in that, The step of performing feature sampling of a preset dimension based on all the features of the signal to be detected, and obtaining multiple sampled feature sets for each preset dimension, includes: When the preset dimension includes one-dimensional, two-dimensional and target dimensions, one-dimensional feature random sampling is performed based on any type of the signal feature to be detected to obtain multiple one-dimensional sampled feature sets, wherein the one-dimensional sampled feature set includes a first preset number of one-dimensional signal features to be detected. Based on any two types of the signal features to be detected, two-dimensional feature random sampling is performed to obtain multiple two-dimensional sampled feature sets. The two-dimensional sampled feature sets include a second preset number of two-dimensional combinations of the signal features to be detected. The two-dimensional signal features to be detected are composed of any two types of the signal features to be detected. Based on all the features of the signal to be detected, random sampling of the target dimension features is performed to obtain multiple target dimension sampled feature sets. The target dimension sampled feature sets include a third preset number of combinations of the signal features to be detected in the target dimension, and the combinations of the signal features to be detected in the target dimension are composed of all types of the signal features to be detected.

5. The abnormal data detection method according to claim 3, characterized in that, The steps for constructing isolated subtrees corresponding one-to-one with the sampling feature set include: A sampling point is randomly selected from the sampling feature set as the cutting node; Based on the cutting node, sampling points with values ​​less than the cutting node are determined as the left child node of the cutting node, and sampling points with values ​​greater than or equal to the cutting node are determined as the right child node of the cutting node. Based on the left child node and the right child node, the process is gradually cut until the cutting is completed or the height of the child node has reached the preset limit height, thus completing the construction of the corresponding isolated subtree.

6. The abnormal data detection method according to claim 1, characterized in that, Also includes: Based on a preset clustering model, all the abnormal data are clustered to obtain the abnormal data with cluster identifiers. The data anomaly type is determined based on the vehicle signals corresponding to the abnormal data with the cluster identifier.

7. An abnormal data detection system, characterized in that, Applications in vehicle-to-everything (V2X) communication include: The signal feature acquisition module is used to acquire the features of the signal to be detected; the signal features to be detected refer to the feature vector obtained by normalizing the vehicle signal to be detected; the signal features to be detected include mileage and fuel consumption; An isolated forest detector construction module is used to construct an isolated forest detector based on the features of the signal to be detected. The isolated forest detector includes multiple isolated forests, and each isolated forest corresponds to a preset dimension. An abnormal data detection module is used to traverse the isolated forest corresponding to a preset dimension of the signal feature to be detected and the combination of signal features to be detected, respectively, to obtain abnormal data in each preset dimension. Each abnormal data carries a corresponding detector identifier, which is a preset identifier that determines the isolated forest of abnormal data of the signal feature to be detected or the combination of signal features to be detected. The signal feature to be detected corresponds to a one-dimensional isolated forest, and the combination of signal features to be detected is composed of any two or more types of signal features to be detected. The combination of signal features to be detected corresponds to a multi-dimensional isolated forest. The anomaly source determination module is used to perform an intersection operation between each multidimensional anomaly data and all one-dimensional anomaly data to obtain the one-dimensional anomaly data corresponding to the multidimensional anomaly data, and update the detector identifier of the multidimensional anomaly data to the detector identifier of the corresponding one-dimensional anomaly data.

8. An abnormal data detection platform, characterized in that, include: The display module, and the abnormal data detection system as described in claim 7; The display module is used to display one-dimensional abnormal data, multi-dimensional abnormal data, and the source of the abnormality in the multi-dimensional abnormal data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the abnormal data detection method as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the abnormal data detection method as described in any one of claims 1 to 6.

Images