A method and system for detecting anomalies in sensor data of autonomous vehicles

Abstract

This invention discloses a method and system for detecting anomalies in sensor data of autonomous vehicles. It reconstructs vehicle sensor data using a pre-trained model to obtain reconstructed data. Error calculations are performed on the reconstructed data and the vehicle sensor data used to acquire it. If the error between the reconstructed data and the vehicle sensor data used to acquire it exceeds a set threshold, the reconstructed data is considered anomaly; otherwise, it is considered normal data. This invention effectively detects and interprets anomalies, ensuring the safety of autonomous vehicle operation. It enhances the model's detection performance for various anomaly types in sensor data. Furthermore, by calculating the contribution of each input dimension to the anomaly detection results, the model's reliability is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of fault diagnosis of autonomous vehicle systems, and specifically relates to a method and system for detecting abnormal sensor data in autonomous vehicles. Background Technology

[0002] In recent years, autonomous vehicles have attracted widespread attention due to their advantages in improving traffic safety and efficiency. Autonomous vehicles rely heavily on data collected from various sensors, including data about the vehicle itself and its surrounding environment, to achieve autonomous driving. The accuracy of this sensor data is crucial for the safe operation of autonomous vehicles. Anomaly detection in the sensor data of autonomous vehicles is one way to ensure their safety. However, existing methods for anomaly detection in autonomous vehicle sensor data have poor detection capabilities, lack the ability to model time series data, and cannot effectively extract temporal information from the sensor data, making it difficult to detect various anomalies. Furthermore, they lack interpretability for the anomaly detection results. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for detecting anomalies in sensor data of autonomous vehicles, so as to overcome the problems of low accuracy and lack of interpretation of detection results in the existing technology for detecting anomalies in sensor data of autonomous vehicles.

[0004] A method for detecting anomalies in sensor data of an autonomous vehicle includes the following steps:

[0005] S1, real-time acquisition of vehicle sensor data, and use a reconstruction pre-trained model to perform abnormal reconstruction of the vehicle sensor data to obtain reconstructed data;

[0006] S2, perform error calculation on the acquired reconstructed data and the vehicle sensor data that acquired the reconstructed data. If the error between the reconstructed data and the vehicle sensor data that acquired the reconstructed data is greater than a set threshold, the vehicle sensor data that acquired the reconstructed data is abnormal; otherwise, it is normal data.

[0007] Preferably, the reconstructed pre-trained model is trained using accurate multi-dimensional temporal positioning sensor data obtained during normal vehicle operation.

[0008] Preferably, the multidimensional temporal positioning sensor data is obtained through preprocessing, specifically including the following steps:

[0009] Useful fields are extracted from the sensor data, including protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward velocity, northward velocity, and celestial velocity. For samples with incomplete or incorrect fields, linear interpolation of the corresponding fields from two adjacent samples before and after the sample is used to complete the incorrect or incomplete fields, forming useful multidimensional time-series positioning sensor data.

[0010] Preferably, the reconstructed pre-trained model specifically adopts a reconstructed model based on variational Transformer unsupervised features.

[0011] Preferably, the reconstructed pre-trained model includes a neural network with encoder and decoder structures, and both the encoder and decoder of the reconstructed pre-trained model are composed of L layers of variational self-attention layers.

[0012] Preferably, after passing through an encoder with L layers of variational self-attention layers, the input data is encoded into hidden layer variables, and the decoder decodes and outputs the hidden layer variables. The decoder consists of L layers of variational self-attention layers and outputs reconstructed data.

[0013] Preferably, by calculating the contribution of each input dimension to the anomaly detection result, the anomaly channel contributes to the anomaly result R. nd The degree of influence is defined as:

[0014]

[0015] The molecule is x. nd and x' nd 'The exponential function of the constructed residuals; the denominator represents the sum of the exponential functions of the residuals between each feature of the sample and the reconstructed feature in the feature direction; R nd The value represents the degree to which the data dimensional features of a sample influence the determination of the sample as an anomaly; R nd The larger the value, the greater the influence of that sample dimension on whether the sample is judged as an anomaly; according to R... nd The magnitude of the value can identify the abnormal data channels that cause sample anomalies.

[0016] An autonomous vehicle sensor data anomaly detection system includes a data acquisition module and a data anomaly judgment module.

[0017] The data acquisition module performs anomaly reconstruction on the collected vehicle sensor data to obtain reconstructed data;

[0018] The data anomaly detection module performs error calculations on the acquired reconstructed data and the vehicle sensor data that acquired the reconstructed data. If the error between the reconstructed data and the vehicle sensor data that acquired the reconstructed data is greater than a set threshold, the vehicle sensor data that acquired the reconstructed data is considered abnormal; otherwise, it is considered normal data.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for detecting abnormal sensor data in an autonomous vehicle.

[0020] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for detecting abnormal sensor data in an autonomous vehicle.

[0021] Compared with the prior art, the present invention has the following beneficial technical effects:

[0022] This invention provides a method for detecting anomalies in sensor data of autonomous vehicles. The method reconstructs vehicle sensor data by reconstructing a pre-trained model to obtain reconstructed data. Error calculation is performed on the obtained reconstructed data and the vehicle sensor data that obtained the reconstructed data. If the error between the reconstructed data and the vehicle sensor data that obtained the reconstructed data is greater than a set threshold, the vehicle sensor data that obtained the reconstructed data is considered abnormal; otherwise, it is considered normal data. This invention can effectively detect and interpret anomalies, ensuring the safety of autonomous vehicle operation.

[0023] Preferably, this invention considers the time-series patterns of sensor data from autonomous vehicles, uses multi-dimensional time-series localization of sensor data, enhances the learning ability of data distribution parameters, and improves the model's detection performance for various anomaly types in sensor data. Furthermore, by calculating the contribution of each input dimension to the anomaly detection results, the invention interprets the anomaly detection results, thereby enhancing the model's reliability. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method for detecting abnormal sensor data in autonomous vehicles in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the system structure in an embodiment of the present invention.

[0026] Figure 3 This is a structural diagram of the variational Transformer model in an embodiment of the present invention.

[0027] Figure 4 This is a structural diagram of the variational attention mechanism in an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] like Figure 1 As shown, this invention provides a method for detecting anomalies in sensor data of autonomous vehicles, used to detect anomalies in data acquired by vehicle sensors, specifically including the following steps:

[0031] S1, real-time acquisition of vehicle sensor data, and use a reconstruction pre-trained model to perform abnormal reconstruction of the vehicle sensor data to obtain reconstructed data;

[0032] S2, perform error calculation on the acquired reconstructed data and the vehicle sensor data that acquired the reconstructed data. If the error between the reconstructed data and the vehicle sensor data that acquired the reconstructed data is greater than a set threshold, the vehicle sensor data that acquired the reconstructed data is abnormal; otherwise, it is normal data.

[0033] In this embodiment, positioning system sensor data is used as an example to illustrate a specific implementation method. Combined navigation (GNSS + inertial navigation) data is selected to specifically explain the implementation of the autonomous vehicle anomaly diagnosis method.

[0034] For integrated navigation information, the data acquisition device includes a high-precision MEMS (Micro-Electro-Mechanical Systems) integrated navigation system installed inside the vehicle, a GNSS antenna for receiving satellite signals, an industrial control computer for recording the GNSS log of the integrated navigation system, and power supply equipment for the above devices. The calibrated integrated navigation system and industrial control computer are placed and fixed at the rear of the vehicle and connected to the GNSS antenna via a feeder cable. The GNSS antenna is screwed onto two strong magnetic chucks and fixedly positioned in the forward and reverse directions of the autonomous vehicle, ideally at the highest point of the vehicle to ensure good GNSS signal reception. Simultaneously, the line connecting the phase centers of the two GNSS antennas must be aligned with or parallel to the central axis of the test vehicle.

[0035] The reconstructed pre-trained model is trained using accurate multi-dimensional temporal positioning sensor data acquired during normal vehicle operation, specifically including the following steps:

[0036] 1. Data Extraction and Preprocessing: The acquired data from the positioning system sensors is processed by extracting useful fields and performing data cleaning and noise reduction to prepare for subsequent model training. For integrated navigation information, the extracted useful fields include protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward velocity, northward velocity, and azimuth velocity. The extracted data may contain incomplete or missing records and cannot be directly input into the model for training. In this invention, for samples with incomplete or incorrect fields, linear interpolation of the corresponding fields from two adjacent samples is used to complete the incorrect or incomplete fields, forming useful multidimensional time-series positioning sensor data.

[0037] For the dataset X = {x1, x2, ..., x...} N}, Model input data x i The input data is processed through a sliding window mechanism to obtain h0, as described in equation (1).

[0038] h0 = [x i ,x i+1 ,…,x i+w (1)

[0039] Where w is the window size. Attention calculation is a parallel processing method, which leads to the loss of relative positional information in the sequence. Therefore, positional information is embedded into h0 to ensure that the model learns relative positional information. h0′=h0+p, where: p is the positional encoding of the same dimension as h0, which is calculated by equation (2):

[0040] p (k,2i) =sin(k / 10000) 2i / dmodel )

[0041]

[0042] Where: p (k,2i) It is the 2i-th component of the encoding vector at position k, d model It is the dimension of the model.

[0043] 2. Construction and training of a reconstruction model based on unsupervised features of variational Transformer:

[0044] Based on the multi-dimensional temporal localization sensor data obtained during normal operation of autonomous vehicles, a variational Transformer-based unsupervised feature reconstruction model is constructed to learn its distribution characteristics. The trained pre-trained reconstruction model can reconstruct the data under normal operating conditions well, thus producing a small reconstruction error. Specifically, the following steps are included:

[0045] 2.1 Construction of an unsupervised feature reconstruction model based on variational Transformer:

[0046] Reconstructing a pre-trained model is a neural network that includes encoder and decoder structures, such as the variational Transformer structure. Figure 3 As shown. The encoder and decoder of the reconstructed pre-trained model are both composed of L layers of variational self-attention layers. The variational self-attention layer is the core module in the variational Transformer model. It is a query (query, q), key (key, k), and value (value, v) model. The q, k, and v matrices are obtained by linear mapping from the input data. The specific operation is described by equations (3)-(5):

[0047] q = x·W q (3)

[0048] k = x·W k (4)

[0049] v = x·W v (5)

[0050] Where x is the input data, W q W k and W v These are the parameter matrices learned by the model during training. m and n are the lengths of k (or v) and q, d q d k and d v It is the dimension of q, k, v, and d q =d k The attention weights are obtained by a dot product of q and k, and the attention weight A is defined as follows:

[0051]

[0052] in: Attention weights are obtained through dot product operations, and in each operation, the attention weight is a deterministic value.

[0053] Since the aforementioned reconstructed model can only perform calculations for deterministic variables and lacks the ability to learn data distributions, variational inference techniques are introduced. The attention distribution is obtained using variational inference. in This represents the variational parameters. The goal is to find the parameters. We want to approximate the true posterior distribution p(A|x,y), where p(A|x,y) represents the probability distribution of A when the model input is x and the output is y. Therefore, the objective is to minimize p(A|x,y) and... The KL divergence between them. Based on the calculation method of KL divergence and Bayesian theory, the calculation method of KL divergence is derived, as shown in equation (8):

[0054]

[0055] log p(x,y) and L(x,y) are called evidence and the lower bound of evidence, respectively. Log p(x,y) is difficult to compute, but since the KL divergence is greater than 0, the above problem can be transformed into maximizing L(x,y).

[0056] log p(x,y)≥L(x,y)

[0057]

[0058] Where p(A) is the prior distribution. L(x,y) serves as the loss function to optimize the model parameters, forcing the attention distribution to change during model training. It approximates the true posterior distribution. Since the model training uses normal data or datasets containing only a very small portion of outliers, and thanks to the universality of the Gaussian distribution's modeling capabilities and the simplicity of sampling, a multivariate Gaussian distribution is used as the prior distribution for the embedding. The variational self-attention mechanism network structure is as follows: Figure 4 As shown. q l k l and v l It is the output h of layer l-1 of the encoder (decoder). l-1 linear mapping, Variational posterior distribution It can be approximated as:

[0059]

[0060] Where, μ0 and These are the mean and variance of a prior multivariate Gaussian distribution. Variational attention learning A l The distribution parameter mean μ l and variance As shown in equations (10)-(11).

[0061] μ l =f1(BatchNorm(A l (10)

[0062] σ l =f2(BatchNorm(A l (11)

[0063] Here, f1 and f2 represent linear mappings. To accelerate network convergence, a Batch Normalization operation is added. Since direct sampling cannot optimize model parameters through backpropagation, reparameterization is used to optimize the distribution parameters μ. l , Sampling. Define I as the identity matrix, and first sample ε from N(0,I). l Then let

[0064] z l =μ l +σ l ⊙ε l (12)

[0065] Where ⊙ represents element-wise multiplication, after reparameterization, Z l That is, from the distribution N(μ) l ,σ l The samples obtained from sampling in layer I. Finally, the output h of layer I. l via z l and v l The matrix multiplication yields the result.

[0066] h l =z l ·v l (13)

[0067] Since the given prior distribution is a multivariate Gaussian distribution, The distribution follows N(μ) l ,σ l ), the first layer and prior distribution p(A) l The KL divergence of ) is calculated as follows:

[0068]

[0069] Where, μ li σ li μ l and σ l The i-th element. Therefore, KL divergence KLq of p(A) φ (A)‖p(A) can be calculated as:

[0070]

[0071] The first term in the loss function This can usually be obtained by calculating the error between the input and output data. It is also known as the reconstruction error term.

[0072]

[0073] Ultimately, the loss function of the entire network can be defined as:

[0074]

[0075] The first term in the loss function ensures that the network can reconstruct the data well, while the second term is a constraint term added to the attention weights to force the distribution of the attention weights to approximate the given prior multivariate Gaussian distribution.

[0076] After passing through an encoder with L layers of variational self-attention layers, the input data is encoded into hidden layer variables, and the decoder decodes and outputs these hidden layer variables. Similarly, the decoder consists of L layers of variational self-attention layers. Finally, the reconstructed data x′ is output, and the error (reconstruction error) between the input data (the vehicle sensor data used to acquire this reconstructed data) and the reconstructed data is used to determine whether the input data is abnormal.

[0077] Because the model is trained on normal data, it learns the distribution characteristics of normal data and can reconstruct the input normal data well. However, abnormal data reconstructed by the variational Transformer model will result in reconstructed data with a large reconstruction error. The reconstruction error is defined as:

[0078] ε=||x′-x|| 2 (18)

[0079] Here, ||·|| represents the norm, with the L2 norm commonly used. The variational Transformer model performs anomaly detection based on reconstruction error. If the reconstruction error of the input data after model reconstruction exceeds a set threshold, the model classifies the input data as an anomaly.

[0080] The reconstructed model trained using the above method is used for online anomaly detection. For each piece of multidimensional data input to the reconstructed model, the model performs encoding and decoding operations to output the reconstructed data.

[0081] 3. Anomaly Detection Result Interpretation: The data identified as anomaly by the above methods is interpreted to determine the sensors responsible for the anomalies. This is achieved by calculating the contribution of each input dimension to the anomaly detection results. Let the test set consist of… Let be the number of samples in the test set, where N is the number of samples in the test set and D is the dimension of the test set. The d-values ​​of samples n in the test set are... th Dimension x nd The data was considered anomalous and, after being reconstructed by the variational Transformer, the output became x'. nd Abnormal channels affect abnormal results R. nd The degree of influence is defined as:

[0082]

[0083] The molecule is x. nd and x' nd 'The exponential function of the constructed residuals; the denominator represents the sum of the exponential functions of the residuals between each feature of the sample and the reconstructed feature in the feature direction. R' nd The value represents the degree to which the data dimensional features of a sample influence the determination of the sample as an anomaly. R nd The larger the value, the greater the influence of that sample dimension on whether the sample is judged as an anomaly. (According to R...) nd The magnitude of the value can identify the abnormal data channels that cause sample anomalies.

[0084] This invention presents an anomaly detection framework for an unsupervised autonomous vehicle localization system based on variational Transformer. Sensor data from the autonomous vehicle is input to this framework in real time. After preprocessing by a preprocessing module, the data is input into a variational Transformer-based reconstruction network to extract data features and output reconstructed data. An anomaly judgment module determines the data state by calculating the error between the reconstructed data and the original input data. An anomaly detection result interpretation module determines the data dimension causing the anomaly by calculating the importance of the data channels for each anomaly sample. This framework is designed from a data-driven perspective, primarily targeting the multidimensional sensor data of autonomous vehicles, and can effectively detect anomalies in sensor data and the operational state of the autonomous vehicle, providing reasonable interpretations of the detection results.

Claims

1. An automatic driving car sensor data anomaly detection method, characterized in that, Includes the following steps: S1, real-time acquisition of vehicle sensor data, and use a reconstruction pre-trained model to perform abnormal reconstruction of the vehicle sensor data to obtain reconstructed data; S2, perform error calculation on the acquired reconstructed data and the vehicle sensor data that acquired the reconstructed data. If the error between the reconstructed data and the vehicle sensor data that acquired the reconstructed data is greater than a set threshold, the vehicle sensor data that acquired the reconstructed data is abnormal; otherwise, it is normal data. Multidimensional temporal positioning sensor data is obtained through preprocessing, specifically including the following steps: Extract useful fields from sensor data, including protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward velocity, northward velocity, and celestial velocity; for samples with incomplete or incorrect fields, use linear interpolation of the corresponding fields from two adjacent samples to complete the incorrect or incomplete fields, forming useful multidimensional time-series positioning sensor data; The reconstruction of the pre-trained model specifically adopts a reconstruction model based on variational Transformer unsupervised features; The reconstructed pre-trained model is a neural network containing encoder and decoder structures. Both the encoder and decoder of the reconstructed pre-trained model are composed of L layers of variational self-attention layers. After the encoder of the L-layer variational self-attention layer, the input data is encoded into the hidden layer variable, and the decoder decodes the hidden layer variable and outputs, the decoder is composed of L The layer variational self-attention layer is composed of, and the output is the reconstructed data.

2. The method for detecting anomalies in sensor data of an autonomous vehicle according to claim 1, characterized in that, The reconstructed pre-trained model is trained using accurate multi-dimensional temporal positioning sensor data obtained during normal vehicle operation.

3. An anomaly detection system for sensor data of an autonomous vehicle used in the method of claim 1, characterized in that, Data acquisition and reconstruction module and data anomaly detection module, The data acquisition and reconstruction module performs anomaly reconstruction on the acquired vehicle sensor data to obtain reconstructed data; The data anomaly detection module performs error calculations on the acquired reconstructed data and the vehicle sensor data that acquired the reconstructed data. If the error between the reconstructed data and the vehicle sensor data that acquired the reconstructed data is greater than a set threshold, the vehicle sensor data that acquired the reconstructed data is considered abnormal; otherwise, it is considered normal data.

4. The autonomous vehicle sensor data anomaly detection system according to claim 3, characterized in that, The reconstructed pre-trained model is trained using accurate multi-dimensional temporal positioning sensor data acquired during normal vehicle operation; the multi-dimensional temporal positioning sensor data is obtained through preprocessing, specifically including the following steps: Useful fields are extracted from the sensor data, including protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward velocity, northward velocity, and celestial velocity. For samples with incomplete or incorrect fields, linear interpolation of the corresponding fields from two adjacent samples before and after the sample is used to complete the incorrect or incomplete fields, forming useful multidimensional time-series positioning sensor data.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the autonomous vehicle sensor data anomaly detection method as described in any one of claims 1 to 2.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the autonomous vehicle sensor data anomaly detection method as described in any one of claims 1 to 2.

Images