Method and system for detecting data anomaly of sensor of self-driving automobile

By using reconstructed pre-trained models and variational Transformer models to detect sensor data in autonomous vehicles, the problems of poor detection capabilities and insufficient interpretation capabilities in the prior art are solved, and efficient abnormal detection and detection of sensor data are achieved. The results explain, ensuring the safety of autonomous vehicles.

CN118991806AActive Publication Date: 2024-11-22CHANGAN UNIV

Patent Information

Application Number
CN202411086767.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-22
Estimated Expiration
2044-08-08

AI Technical Summary

Technical Problem

The existing autonomous driving vehicle sensor data abnormality detection methods have poor detection capabilities, lack the ability to model time series, cannot effectively discover the time information in sensor data, and lack the ability to interpret the detection results.

Method used

The reconstructed pre-trained model is used to reconstruct the vehicle sensor data abnormality, judge the data abnormality through error calculation, and enhance the learning of the data distribution parameters by positioning the sensor data by multi-dimensional timing, and use the unsupervised feature reconstruction model based on the variational Transformer Perform anomaly detection and calculate the contribution of each input dimension to sample anomaly detection results to explain the anomaly detection results.

Benefits of technology

Effectively detect and interpret abnormalities in the sensor data of autonomous driving cars, improve detection accuracy and model credibility, and ensure the safe operation of autonomous driving cars.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118991806A_ABST
    Figure CN118991806A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic driving vehicle sensor data anomaly detection method and system. The method comprises the following steps: performing anomaly reconstruction on vehicle sensor data by reconstructing a pre-training model to obtain reconstructed data; and performing error calculation on the obtained reconstruction data and the vehicle sensor data of the obtained reconstruction data, if the error between the reconstruction data and the vehicle sensor data of the obtained reconstruction data is greater than a set threshold value, determining that the vehicle sensor data of the obtained reconstruction data is abnormal, otherwise, determining that the vehicle sensor data of the obtained reconstruction data is normal data. According to the invention, abnormity can be effectively detected and explained, and the running safety of the self-driving automobile is ensured. And the detection performance of the model on various abnormal types of sensor data is enhanced. And the anomaly detection result is explained by calculating the contribution of each input dimension to the sample anomaly detection result, so that the credibility of the model is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving vehicle system fault diagnosis, and specifically relates to an autonomous driving vehicle sensor data anomaly detection method and system. Background Art

[0002] In recent years, self-driving cars have attracted widespread attention due to their advantages in improving traffic safety and traffic efficiency. Self-driving cars rely heavily on data collected by various sensors on the vehicle itself and the surrounding environment to achieve autonomous driving. The accuracy of sensor data is crucial to the safe operation of self-driving cars. Anomaly detection of sensor data of self-driving cars is a way to ensure the safety of self-driving cars. Existing methods for anomaly detection of sensor data of self-driving cars have poor detection capabilities and lack the ability to model time series. They cannot effectively explore the time information in sensor data, making it difficult to detect various anomalies in sensor data. At the same time, they lack the ability to explain anomaly detection results. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for detecting anomaly in sensor data of an autonomous vehicle, so as to overcome the problems of low accuracy in detecting anomaly in sensor data of an autonomous vehicle and lack of explanation of the detection results in the prior art.

[0004] A method for detecting anomaly in sensor data of an autonomous driving vehicle comprises the following steps:

[0005] S1, real-time acquisition of vehicle sensor data, using a reconstruction pre-training model to perform abnormal reconstruction on the vehicle sensor data to obtain reconstructed data;

[0006] S2, performing error calculation on the acquired reconstructed data and the vehicle sensor data for acquiring the reconstructed data. If the error between the reconstructed data and the vehicle sensor data for acquiring the reconstructed data is greater than a set threshold, the vehicle sensor data for acquiring the reconstructed data is abnormal, otherwise it is normal data.

[0007] Preferably, the reconstructed pre-trained model is trained using accurate multi-dimensional time-series positioning sensor data obtained during normal operation of the vehicle.

[0008] Preferably, the multi-dimensional time series positioning sensor data is obtained by preprocessing, which specifically includes the following steps:

[0009] Useful fields of sensor data are extracted, including protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward speed, northward speed, and celestial speed. For samples with incomplete or erroneous fields, linear interpolation of the corresponding fields of the two adjacent samples before and after the sample is used to complete the erroneous or incomplete fields, thus forming useful multi-dimensional time series positioning sensor data.

[0010] Preferably, the reconstruction pre-training model specifically adopts a reconstruction model based on variational Transformer unsupervised features.

[0011] Preferably, the reconstructed pre-trained model includes a neural network of an encoder and a decoder structure, and the encoder and the decoder of the reconstructed pre-trained model are both composed of L layers of variational self-attention layers.

[0012] Preferably, after passing through an encoder of L layers of variational self-attention layers, the input data is encoded into hidden variables, and a decoder decodes and outputs the hidden variables. The decoder is composed of L layers of variational self-attention layers and outputs reconstructed data.

[0013] Preferably, by calculating the contribution of each input dimension to the sample anomaly detection result, the abnormal channel contributes to the abnormal result R nd The impact is defined as:

[0014]

[0015] The numerator is x nd and x' nd 'The exponential function of the constructed residual; the denominator represents the sum of the exponential functions of the residuals between each feature of the sample and the reconstructed feature in the feature direction; R nd The value indicates the degree of influence of the data dimension characteristics of the sample on determining the sample as abnormal; R nd The larger the value, the greater the impact of the sample dimension on the sample being judged as abnormal; according to R nd The size of the value can identify the abnormal data channel that causes the sample abnormality.

[0016] An autonomous driving vehicle sensor data anomaly detection system, a data acquisition module and a data anomaly judgment module,

[0017] The data acquisition module reconstructs the collected vehicle sensor data abnormally to obtain reconstructed data;

[0018] The data anomaly judgment module performs error calculation on the acquired reconstructed data and the vehicle sensor data for acquiring the reconstructed data. If the error between the reconstructed data and the vehicle sensor data for acquiring the reconstructed data is greater than a set threshold, the vehicle sensor data for acquiring the reconstructed data is abnormal; otherwise, it is normal data.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for detecting abnormality in sensor data of an autonomous driving vehicle when executing the computer program.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for detecting abnormality in sensor data of an autonomous driving vehicle.

[0021] Compared with the prior art, the present invention has the following beneficial technical effects:

[0022] The present invention provides a method for detecting anomalies in sensor data of an autonomous driving vehicle. The method comprises the following steps: abnormally reconstructing vehicle sensor data by reconstructing a pre-trained model to obtain reconstructed data; performing error calculation on the obtained reconstructed data and vehicle sensor data for obtaining the reconstructed data. If the error between the reconstructed data and the vehicle sensor data for obtaining the reconstructed data is greater than a set threshold, the vehicle sensor data for obtaining the reconstructed data is abnormal; otherwise, the vehicle sensor data for obtaining the reconstructed data is normal. The present invention can effectively detect and explain anomalies, thereby ensuring the safety of the operation of the autonomous driving vehicle.

[0023] Preferably, the present invention considers the time series pattern of the sensor data of the autonomous vehicle, uses multi-dimensional time series to locate the sensor data, enhances the learning ability of the data distribution parameters, and enhances the model's detection performance of multiple types of anomalies in the sensor data. And by calculating the contribution of each input dimension to the sample anomaly detection result, the anomaly detection result is interpreted, thereby enhancing the credibility of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a flow chart of a method for detecting abnormal data from sensors of an autonomous driving vehicle in an embodiment of the present invention.

[0025] Figure 2 Schematic diagram of the system structure in an embodiment of the present invention.

[0026] Figure 3 It is a structural diagram of a variational Transformer model in an embodiment of the present invention.

[0027] Figure 4 It is a structural diagram of the variational attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] like Figure 1 As shown, the present invention provides a method for detecting abnormal data of a sensor of an autonomous driving vehicle, which is used to detect abnormal data obtained by a vehicle sensor, and specifically comprises the following steps:

[0031] S1, real-time acquisition of vehicle sensor data, using a reconstruction pre-training model to perform abnormal reconstruction on the vehicle sensor data to obtain reconstructed data;

[0032] S2, performing error calculation on the acquired reconstructed data and the vehicle sensor data for acquiring the reconstructed data. If the error between the reconstructed data and the vehicle sensor data for acquiring the reconstructed data is greater than a set threshold, the vehicle sensor data for acquiring the reconstructed data is abnormal, otherwise it is normal data.

[0033] In this embodiment, the positioning system sensor data is taken as an example to illustrate the specific embodiment, and combined navigation (GNSS+inertial navigation) data is selected to specifically explain the implementation of the abnormality diagnosis method of the autonomous driving vehicle.

[0034] For the combined navigation information, the data acquisition device used includes a high-precision MEMS (micro-electromechanical system) combined navigation system installed in the car and a set of GNSS antennas for receiving satellite signals, an industrial computer for recording the GNSS log of the combined navigation system, and a power supply device supporting the above-mentioned device. The calibrated combined navigation system and the industrial computer are fixed at the rear of the car and connected to the GNSS antenna through a feeder. The GNSS antennas are screwed onto two strong magnetic suction cups and fixed in the forward and backward directions of the self-driving car. They are placed as high as possible on the self-driving car to ensure that good GNSS signals can be received. At the same time, the connection line formed by the phase centers of the two GNSS antennas must be consistent with or parallel to the central axis of the test carrier.

[0035] The reconstruction pre-training model is trained using accurate multi-dimensional time-series positioning sensor data obtained during normal operation of the vehicle, which specifically includes the following steps:

[0036] 1. Data extraction and preprocessing: For the data obtained from the positioning system sensor, useful fields are extracted and data cleaning and data noise reduction are performed to prepare for the subsequent model training. For the combined navigation information, the extracted useful fields include the protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward speed, northward speed, and celestial speed; the data after the useful fields are extracted may have incomplete or missing records and cannot be directly input into the model for training. In the present invention, for samples with incomplete or erroneous fields, the linear interpolation of the corresponding fields of the two adjacent samples before and after the sample is used to complete the erroneous fields or incomplete fields, thereby forming useful multi-dimensional time series positioning sensor data.

[0037] For a data set X = {x1, x2, ..., x N}, model input data x i , the input data is passed through the sliding window mechanism to obtain h0, as described in formula (1),

[0038] h0=[x i ,x i+1 ,…,x i+w ] (1)

[0039] Where w is the window size. The calculation of attention is a parallel processing method, which will lead to the loss of relative position information in the sequence. Therefore, the position information is embedded and added to h0 to ensure that the model learns the relative position information. h0′=h0+p, where: p is the position encoding of the same dimension as h0, calculated by formula (2):

[0040] p (k,2i) = sin(k / 10000 2i / dmodel )

[0041]

[0042] Where: p (k,2i) is the 2ith component of the encoding vector at position k, d model is the dimension of the model.

[0043] 2. Construction and training of reconstruction model based on variational Transformer unsupervised features:

[0044] For the multi-dimensional time series positioning sensor data obtained during normal operation of the autonomous driving vehicle, an unsupervised feature reconstruction model based on variational Transformer is constructed to learn its distribution characteristics. The trained reconstruction pre-training model can well reconstruct the data under normal operation, thereby generating a smaller reconstruction error. Specifically, the following steps are included:

[0045] 2.1, Unsupervised feature reconstruction model construction based on variational Transformer:

[0046] The reconstructed pre-trained model is a neural network containing an encoder and decoder structure. The variational Transformer structure is as follows Figure 3 As shown. The encoder and decoder of the reconstructed pre-trained model are both composed of L layers of variational self-attention layers. The variational self-attention layer is the core module in the variational Transformer model. It is a query (query, q), key (key, k), value (value, v) model. The q, k, and v matrices are obtained by linear mapping the input data. The specific operation is described as formula (3)-(5):

[0047] q=x·W q (3)

[0048] k=x·W k (4)

[0049] v=x·W v (5)

[0050] Where x is the input data, W q , W k and W v They are the parameter matrices learned by the model during training, m and n are the lengths of k (or v) and q, d q ,d k and d v is the dimension of q, k, v, and d q =d k The attention weight is obtained by dot product operation of q and k, and the attention weight A is defined as:

[0051]

[0052] in: The attention weight is obtained through dot product operation. In each operation, the attention weight is a deterministic value.

[0053] Since the above reconstruction model can only calculate deterministic variables and does not have the ability to learn data distribution, variational inference technology is introduced. Using variational inference, the attention distribution is obtained in represents the variational parameter. The goal is to find the parameter To approximate the true posterior distribution p(A|x,y), p(A|x,y) represents the distribution probability of A when the model input is x and the output is y. Therefore, the goal is transformed into minimizing p(A|x,y) and According to the calculation method of KL divergence and Bayesian theory, the calculation method of KL divergence is derived, as shown in formula (8):

[0054]

[0055] log p(x,y) and L(x,y) are called evidence and evidence lower bound. log p(x,y) is difficult to calculate, and since KL divergence is greater than 0, the above problem can be transformed into maximizing L(x,y).

[0056] log p(x,y)≥L(x,y)

[0057]

[0058] Where p(A) is the prior distribution. L(x,y) is used as the loss function to optimize the model parameters, forcing attention to the distribution during model training. Close to the true posterior distribution. Since the model training uses normal data or a data set containing only a very small part of abnormal data, thanks to the universal modeling ability of the Gaussian distribution and the ease of sampling, the multivariate Gaussian distribution is used as the prior distribution of the embedding. The variational self-attention mechanism network structure is as follows Figure 4 As shown. l , k l and v l is the encoder (decoder) layer l-1 output h l-1 The linear mapping of Variational Posterior Distribution It can be approximately calculated as:

[0059]

[0060] Among them, μ0 and are the mean and variance of the prior multivariate Gaussian distribution. Variational Attention Learning A l The distribution parameter mean μ l and variance As shown in formulas (10)-(11).

[0061] μ l =f1(BatchNorm(A l )) (10)

[0062] σ l =f2(BatchNorm(A l )) (11)

[0063] Among them, f1 and f2 represent linear mappings. In order to accelerate network convergence, Batch Normalization operation is added. Since direct sampling operation cannot optimize model parameters through back propagation, reparameterization is used to optimize the distribution parameters μ l , Sampling. Define I as the unit matrix, first sample from N(0,I) to get ε l , then let

[0064] z l =μ l +σ l ⊙ε l (12)

[0065] Where ⊙ represents the element-by-element multiplication operation. After reparameterization, Z l That is, from the distribution N(μ l ,σ l ) is the sample obtained from the sampling. Finally, the output h of the I layer l By z l and v l By multiplying the matrix of

[0066] h l =z l ·v l (13)

[0067] Since the given prior distribution is a multivariate Gaussian distribution, The distribution of N(μ l ,σ l ), layer I and the prior distribution p(A l ) is calculated as:

[0068]

[0069] Among them, μ li , σ li Represents μ l and σ l The i-th element of . Therefore, and the KL divergence KLq of p(A) φ (A)‖p(A) can be calculated as:

[0070]

[0071] The first term in the loss function It can usually be obtained by calculating the error between the input data and the output data. Therefore, Also called the reconstruction error term.

[0072]

[0073] Finally, the loss function of the entire network can be defined as:

[0074]

[0075] The first term in the loss function ensures that the network can reconstruct the data well, and the second term is a constraint added to the attention weight to force the distribution of the attention weight to approach the given prior multivariate Gaussian distribution.

[0076] After passing through the encoder of L layers of variational self-attention layers, the input data is encoded into hidden variables, and the decoder decodes and outputs the hidden variables. Similarly, the decoder is composed of L layers of variational self-attention layers. Finally, the reconstructed data x' is output, and the error (reconstruction error) between the input data (the vehicle sensor data for obtaining the reconstructed data) and the reconstructed data is used to determine whether the input data is abnormal data.

[0077] Since the model is trained with normal data and has learned the distribution characteristics of normal data, it can reconstruct the input normal data well. However, when abnormal data is reconstructed by the variational Transformer model, reconstructed data with a large reconstruction error will be obtained. The reconstruction error is defined as:

[0078] ε=||x′-x|| 2 , (18)

[0079] Among them, ||·|| represents the norm, and the L2 norm is often used. The variational Transformer model performs anomaly detection based on reconstruction error. If the reconstruction error of the input data after reconstruction by the model is greater than the set threshold, the model will judge the input data as abnormal.

[0080] The reconstruction model trained by the above method performs online abnormality judgment. For each multidimensional data input to the reconstruction model, the reconstruction model performs encoding and decoding operations on it, and then outputs the reconstructed data.

[0081] 3. Explanation of abnormal detection results: Explain the data determined as abnormal in the above method and determine the sensor where the abnormality occurs. Calculate the contribution of each input dimension to the sample abnormality detection result. Suppose the test set is composed of where N is the number of samples in the test set and D is the dimension of the test set. th Dimension x nd It is considered as abnormal data and is reconstructed by the variational Transformer and output as x' nd Abnormal channel to abnormal result R nd The impact is defined as:

[0082]

[0083] The numerator is x nd and x' nd 'The exponential function of the constructed residual; the denominator represents the sum of the exponential functions of the residuals between each feature of the sample and the reconstructed feature in the feature direction. R nd The value indicates the degree to which the data dimension characteristics of the sample affect the determination of the sample as an anomaly. nd The larger the value, the greater the impact of the sample dimension on whether the sample is judged to be abnormal. nd The size of the value can identify the abnormal data channel that causes the sample abnormality.

[0084] The present invention is based on the variational Transformer unsupervised autonomous driving car positioning system anomaly detection framework. The autonomous driving car sensor data is input into the framework in real time. After the data is processed by the preprocessing module, it is input into the reconstruction network based on the variational Transformer to extract data features and output the reconstructed data. The anomaly judgment module determines the data state by calculating the error between the reconstructed data and the original input data. The anomaly detection result interpretation module determines the data dimension that causes the sample anomaly by calculating the importance of the data channel of each abnormal sample. The framework is designed from a data-driven perspective, mainly targeting the multi-dimensional sensor data of autonomous driving cars. It can effectively detect sensor data anomalies and abnormalities in the operating state of autonomous driving cars and make reasonable explanations for the detection results.

Claims

1. A method for detecting anomaly in sensor data of autonomous vehicles, It is characterized in that The following steps are involved: S1, real-time acquisition of vehicle sensor data, using a reconstruction pre-training model to perform abnormal reconstruction on the vehicle sensor data to obtain reconstructed data; S2, performing error calculation on the acquired reconstructed data and the vehicle sensor data for acquiring the reconstructed data. If the error between the reconstructed data and the vehicle sensor data for acquiring the reconstructed data is greater than a set threshold, the vehicle sensor data for acquiring the reconstructed data is abnormal, otherwise it is normal data.

2. The method for detecting abnormality in sensor data of an autonomous driving vehicle according to claim 1, It is characterized in that The reconstructed pre-training model is trained using accurate multi-dimensional time-series positioning sensor data obtained during normal operation of the vehicle.

3. The method for detecting abnormality in sensor data of an autonomous driving vehicle according to claim 2, It is characterized in that The multi-dimensional time series positioning sensor data is obtained through preprocessing, which specifically includes the following steps: Useful fields of sensor data are extracted, including protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward speed, northward speed, and celestial speed. For samples with incomplete or erroneous fields, linear interpolation of the corresponding fields of the two adjacent samples before and after the sample is used to complete the erroneous or incomplete fields, thus forming useful multi-dimensional time series positioning sensor data.

4. The method for detecting abnormality in sensor data of an autonomous driving vehicle according to claim 1, It is characterized in that The reconstruction pre-training model specifically adopts a reconstruction model based on the variational Transformer unsupervised features.

5. The method for detecting abnormality in sensor data of an autonomous driving vehicle according to claim 4, It is characterized in that The reconstructed pre-trained model includes a neural network of an encoder and a decoder structure, and the encoder and the decoder of the reconstructed pre-trained model are both composed of L layers of variational self-attention layers.

6. The method for detecting abnormality in sensor data of an autonomous driving vehicle according to claim 5, It is characterized in that After passing through the encoder of L layers of variational self-attention layers, the input data is encoded into hidden variables, and the decoder decodes and outputs the hidden variables. The decoder is composed of L layers of variational self-attention layers and outputs reconstructed data.

7. An autonomous vehicle sensor data anomaly detection system, It is characterized in that Data collection module and data anomaly judgment module, The data acquisition module reconstructs the collected vehicle sensor data abnormally to obtain reconstructed data; The data anomaly judgment module performs error calculation on the acquired reconstructed data and the vehicle sensor data for acquiring the reconstructed data. If the error between the reconstructed data and the vehicle sensor data for acquiring the reconstructed data is greater than a set threshold, the vehicle sensor data for acquiring the reconstructed data is abnormal; otherwise, it is normal data.

8. The autonomous driving vehicle sensor data anomaly detection system according to claim 7, It is characterized in that The reconstructed pre-trained model is trained using accurate multi-dimensional time series positioning sensor data obtained during normal operation of the vehicle; the multi-dimensional time series positioning sensor data is obtained by preprocessing, specifically including the following steps: Useful fields of sensor data are extracted, including protocol header, sampling time, longitude, latitude, altitude, heading angle, eastward speed, northward speed, and celestial speed. For samples with incomplete or erroneous fields, linear interpolation of the corresponding fields of the two adjacent samples before and after the sample is used to complete the erroneous or incomplete fields, thus forming useful multi-dimensional time series positioning sensor data.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method for detecting anomaly in sensor data of an autonomous driving vehicle as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method for detecting anomaly in sensor data of an autonomous driving vehicle as described in any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Multidimensional data abnormity detecting method and device

    CN108287782A

  • Automatic driving automobile anomaly detection method and related system

    CN117056845A

Cited By

  • Vehicle-road cloud integrated complex time sequence real-time anomaly detection method and system

    CN120354889A