Abnormity detection and restoration method for non-Gaussian multi-modal sensor data

Abstract

The invention discloses an anomaly detection and repair method for non-Gaussian multi-mode sensor data, and belongs to the technical field of unmanned system multi-source autonomous navigation. The method solves the problems that an existing method is poor in performance when facing non-Gaussian data, anomaly detection is difficult to carry out on high-dimensional unstructured data, the robustness of a static anomaly detection model is insufficient, and redundancy compensation is lacked when abnormal data of a sensor is repaired. The method specifically comprises the following steps: step 1, acquiring IMU, GNSS, point cloud and image data of an unmanned system, and then carrying out preprocessing and time synchronization on the acquired data; step 2, respectively extracting features of IMU, GNSS, point cloud and image data at each acquisition moment after time synchronization; step 3, fusing the data features at each acquisition moment; and 4, performing anomaly detection on the data at each acquisition moment according to a fusion result, and further performing data restoration if the data is detected to be abnormal. The method can be applied to data anomaly detection and repair.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-source autonomous navigation of unmanned systems, and in particular relates to an anomaly detection and repair method for non-Gaussian multimodal sensor data. Background Art

[0002] In the positioning system, sensor data anomaly detection and repair is a key issue to improve system reliability and accuracy. The environment of unmanned driving and robot navigation is often complex and full of uncertainty, such as the obstruction of GNSS signals by buildings in the city, the impact of night and bad weather on visual sensors, and the interference of ground bumps on inertial measurement units (IMUs). In addition, due to the aging of sensor hardware or occasional equipment failures, sensor data may also have anomalies such as drift and jump. With the dynamic changes of the environment and the non-ideal performance of sensors, these factors will lead to a decrease in data accuracy, an increase in noise, and even significant abnormal data, posing a potential threat to the accuracy of positioning and the security of the system.

[0003] At present, multi-sensor fusion positioning systems mostly rely on classic state estimation methods such as Kalman Filter, Particle Filter, and Factor Graph Optimization. These methods usually assume that sensor data satisfies Gaussian distribution and noise obeys a specific model. Although these assumptions can effectively improve positioning accuracy in some cases, when data changes suddenly or abnormally, these algorithms have limited tolerance for abnormal data, which may lead to increased cumulative errors or even system crashes. Especially in autonomous vehicles and mobile robot applications, timely detection and repair of abnormal sensor data is particularly important to ensure safe driving and navigation of vehicles.

[0004] In practical applications, the data types of different sensors and their anomaly detection methods vary greatly. For example, IMU and GNSS generate time series data, and anomalies can be detected by analyzing the temporal continuity and change trend of the data, such as detecting sudden acceleration or angular velocity changes in IMU, or detecting drift or mutations in GNSS signals. Cameras and Lidar generate non-time series data such as images and point clouds, and their anomaly detection requires the help of image processing and deep learning technologies, such as using convolutional neural networks (CNNs) to identify occlusions or artifacts in images, or using clustering algorithms to detect abnormal points in point clouds.

[0005] However, the existing data processing methods have the following problems:

[0006] (1) Inadequate processing of non-Gaussian data

[0007] Traditional anomaly detection methods, such as Kalman filtering and its extended versions (such as EKF), usually assume that data noise and errors follow a Gaussian distribution. This Gaussian assumption simplifies the calculation in theory and helps the convergence of the model, but in real environments, this assumption often does not hold. Especially for GNSS data, signals are easily disturbed by multipath effects, occlusions, and random reflections in complex environments, resulting in non-Gaussian characteristics such as skewed distribution and heavy-tailed distribution. In addition, sensors may be affected by nonlinear system characteristics and sudden environmental changes, resulting in large jumps or drifts in the data, and these anomalies usually do not conform to the Gaussian distribution. Existing Gaussian assumption models perform poorly in the face of these anomalies and are unable to effectively deal with sudden non-Gaussian noise, which leads to increased positioning errors and even system failures.

[0008] (2) Anomaly detection in high-dimensional unstructured data is difficult

[0009] Image data generated by cameras and point cloud data generated by lidar are typical high-dimensional unstructured data, which present unique challenges in anomaly detection. Traditional anomaly detection methods mainly rely on statistical analysis or low-dimensional feature extraction, which is difficult to effectively apply to high-dimensional, unstructured data. In image data, occlusion, illumination changes, and dynamic objects can easily cause fluctuations in image quality, and existing methods are difficult to accurately identify abnormal areas in such cases. Similarly, point cloud data usually exhibits sparsity and irregularity. When the point cloud contains a large amount of noise or interference points, traditional anomaly detection algorithms are difficult to effectively distinguish between environmental changes and true sensor anomalies. In addition, the high-dimensional characteristics of image and point cloud data make anomaly detection computationally complex and difficult to ensure real-time performance, which limits their application in unmanned vehicle and robot navigation. To address these challenges, there is an urgent need to develop efficient high-dimensional data anomaly detection algorithms to improve the system's adaptability to complex environments.

[0010] Multimodal sensors (such as GNSS, IMU, LiDAR, and cameras) play important roles in unmanned driving and robot navigation, providing environmental information of different dimensions. However, the data types of these sensors are significantly different, including time series data of IMU and GNSS, and non-time series data of LiDAR and cameras (such as images and point clouds). Most of the existing anomaly detection methods are optimized based on a single sensor type, such as processing IMU data through Kalman filtering, or analyzing camera data through image processing methods. Therefore, the existing anomaly detection methods have certain limitations in terms of uniformity and applicability, and it is difficult to take into account the abnormal data processing requirements of different types of sensors. Moreover, this single processing method lacks the ability to realize information sharing and fusion between different sensors, resulting in a lack of redundant compensation in the system when a sensor data is abnormal. Especially in autonomous driving applications with high real-time requirements, due to the lack of a cross-modal fusion framework, the data between sensors is difficult to be processed uniformly, which limits the effect of multi-sensor fusion anomaly detection and reduces the robustness of the overall system. Therefore, it is necessary to further strengthen the development of cross-modal data fusion technology, and use deep learning algorithms to realize the unified processing of multi-sensor data in complex environments to cope with the challenges of variability and uncertainty in the positioning process. This will not only help improve the safety of unmanned driving technology, but will also promote the development of robot autonomous navigation systems and enable them to be used in a wider range of fields.

[0011] At the same time, during the operation of unmanned vehicles and robots, the error distribution of sensor data will continue to fluctuate with changes in the external environment. Different environmental conditions, such as temperature, humidity, light intensity, terrain changes, etc., will cause significant changes in the error characteristics of sensor data. For example, IMU data drift increases when driving on bumpy roads, GNSS will produce multipath errors in urban canyons, and lidar is prone to increased noise in rainy and snowy weather. Static anomaly detection models often show insufficient robustness because they cannot adapt to such dynamic changes in real time. With the continuous changes in the external environment, fixed error models gradually find it difficult to accurately describe the actual distribution characteristics of sensor data, resulting in a decrease in the effect of anomaly detection. In order to improve the robustness of the positioning system, the anomaly detection system needs to have adaptive capabilities, be able to dynamically adjust the detection strategy based on real-time environmental feedback, and adapt to the dynamic changes in error distribution to ensure the accuracy and stability of the system. Summary of the invention

[0012] The purpose of the present invention is to solve the problems that the existing data anomaly detection methods perform poorly when facing non-Gaussian data, it is difficult to detect anomalies in high-dimensional unstructured data, the static anomaly detection model is not robust enough, and there is a lack of redundancy compensation when repairing sensor anomaly data. An anomaly detection and repair method for non-Gaussian multimodal sensor data is proposed.

[0013] The technical solution adopted by the present invention to solve the above technical problems is:

[0014] A method for detecting and repairing anomalies in non-Gaussian multimodal sensor data, the method specifically comprising the following steps:

[0015] Step 1: Collect IMU data, GNSS data, point cloud data and image data of the unmanned system, pre-process the collected IMU data, GNSS data, point cloud data and image data respectively, and time synchronize the pre-processed IMU data, GNSS data, point cloud data and image data;

[0016] Step 2: Extract the features of IMU data, GNSS data, point cloud data and image data at each acquisition moment after time synchronization;

[0017] Step 3: For each acquisition moment, the features of the IMU data, GNSS data, point cloud data, and image data are fused to obtain the feature fusion results for each acquisition moment.

[0018] Step 4: Perform anomaly detection on the data at each collection time according to the feature fusion results. If anomalies are detected in the data, data repair is required. If no anomalies are detected in the data, data repair is not required.

[0019] Furthermore, the collected IMU data, GNSS data, point cloud data and image data are preprocessed respectively, specifically:

[0020] The collected IMU data, GNSS data, point cloud data and image data are denoised respectively.

[0021] Furthermore, the feature extraction method of the time-synchronized point cloud data is:

[0022] Step 1: Record the point cloud data corresponding to any acquisition time as Among them, P represents the point cloud, p i represents the three-dimensional coordinates of the i-th point, represents a real number, and N represents the number of points;

[0023] Find the value in P that matches p i The k points with the smallest Euclidean distance are used as p i The k nearest neighbor points form p i The local neighborhood

[0024] Based on the local neighborhood of each point Constructing the neighborhood index matrix Among them, I i,j Represents point p i The index of the jth neighbor of ;

[0025] Step 2: For point p i , respectively calculate the local neighborhood Each point in the i The coordinates of Δp ij =p j -p i , Δp ij Represents a local neighborhood The jth point in the i The coordinates of

[0026] P i and Δp ij Perform splicing to obtain the splicing result Then use the multi-layer perceptron to stitch the results Perform nonlinear transformation to obtain point p i The nonlinear transformation result of

[0027] Step 3: Nonlinear transformation results Processing, get point p i The local feature representation h i ;

[0028]

[0029] Step 4: Perform global average pooling on the local feature representations of all points in P to obtain the global feature h global ; Then according to the global feature h global Get the final feature representation of each point:

[0030] h′ i =[h i ,h global ]

[0031] Among them, h′ i It's point p i The final feature representation of

[0032] The final feature representations of all points in the point cloud data corresponding to the current acquisition time are spliced ​​to obtain the point cloud data features at the current acquisition time;

[0033] Step 5: Use the methods of steps 1 to 4 to extract features from the point cloud data collected at each moment.

[0034] Furthermore, the splicing result is processed by using a multi-layer perceptron Perform nonlinear transformation and obtain nonlinear transformation result Specifically:

[0035]

[0036] Among them, σ is the activation function, L is the number of layers of the multilayer perceptron, Indicates the splicing result The output after the lth layer in the multi-layer perceptron is W (l) represents the weight matrix of the lth layer in the multilayer perceptron, b (l) represents the bias vector of the lth layer in the multilayer perceptron, Indicates the splicing result The output after the l-1th layer in the multi-layer perceptron.

[0037] Furthermore, the feature extraction method of the IMU data after time synchronization is:

[0038] Step (1), time series data composed of time-synchronized IMU data Among them, x t represents the observed value at time t, D is the data dimension at each moment, and T is the length of the time series;

[0039] Step (2): For the time series data X ts Perform normalization:

[0040]

[0041] Among them, x t,d Represents x t The data of the dth dimension in μ d Represents X ts The mean of the data in the dth dimension, σ d Represents X ts The standard deviation of the data in the dth dimension, Represents the data of the dth dimension after normalization at the tth moment;

[0042] Then the data at time t after normalization is

[0043] Step (3), use the quantization function q(·) to Convert to discrete Token sequence s t :

[0044]

[0045] Where Q represents the number of discrete intervals, b1, b2, …, b Q-1 To represent the boundary of the discrete interval, s t =[st,1 ,s t,2 ,…,s t,D ];

[0046] Step (4), s t Transformed into an embedding vector e through the embedding layer of the Transformer model t :

[0047] e t =Embedding(s t )

[0048] Among them, Embedding(·) represents the embedding layer;

[0049] Give e t Add position code p t Get the position embedding vector

[0050]

[0051] Then embed the position into the vector Input into the encoder of the Transformer model to obtain the feature representation h of the time series data at time t t :

[0052]

[0053] Among them, TransformerEncoder(·) represents the encoder of the Transformer model.

[0054] Furthermore, the specific process of step three is:

[0055] For any collection time, the features of the IMU data, GNSS data, point cloud data, and image data at the current collection time are recorded as H1, H2, H3, and H4, respectively, and the attention score of each feature is calculated respectively:

[0056]

[0057] Among them, e′ p It is feature H p Attention score, p = 1, 2, 3, 4; W represents the parameter matrix, v and b represent the parameter vectors, the superscript T represents the transpose, and tanh(·) is the hyperbolic tangent function;

[0058] The softmax function is used to normalize the attention score of each feature to obtain the attention weight α of each feature. p :

[0059]

[0060] Use the attention weight to weight the features and get the fused feature H fusion :

[0061]

[0062] Similarly, the fused features at each acquisition moment are obtained respectively.

[0063] Furthermore, the data at each collection moment are respectively subjected to anomaly detection according to the feature fusion result, specifically:

[0064] The fusion feature H obtained at any data collection time fusion For example

[0065] The fusion feature H fusion As the input of the first variational autoencoder, H fusion The output of the encoder after the first variational self-encoder is recorded as f enc (H fusion );

[0066] According to f enc (H fusion ) Calculate the mean μ and variance σ of the latent space 2 :

[0067] μ=f enc (H fusion )

[0068] logσ 2 =f enc (H fusion )

[0069] The latent variables are sampled using the reparameterization technique:

[0070]

[0071] Among them, z is the latent variable, ⊙ represents the Hadamard product, and ∈ represents the variable that obeys the standard normal distribution;

[0072] The decoder of the first variational autoencoder is used to map the sampled latent variable z back to the original feature space to obtain the input feature H fusion The reconstruction result:

[0073]

[0074] Among them, f dec (·) is the decoder of the first variational autoencoder;

[0075] The reconstruction error for:

[0076]

[0077] Among them, ||·|| represents the 2-norm;

[0078] Comparing reconstruction errors With the size of the threshold δ, if the reconstruction error If it is greater than the threshold δ, there is abnormal data in the IMU data, GNSS data, point cloud data and image data at the current acquisition time; otherwise, there is no abnormal data in the IMU data, GNSS data, point cloud data and image data at the current acquisition time.

[0079] Furthermore, the specific process of data repair is as follows:

[0080] Take the data at any collection time as an example

[0081] The IMU data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the IMU data;

[0082] Using the GNSS data collected at this moment as the input of the decoder of the second variational autoencoder to obtain a reconstruction result of the GNSS data;

[0083] The image data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain a reconstruction result of the image data;

[0084] The point cloud data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the point cloud data.

[0085] Furthermore, the threshold δ is obtained by:

[0086]

[0087] Among them, AdvancedTSModel(·) represents the time series model, θ ts represents the parameters of the time series model, b represents the window size of the historical time step, represents the reconstruction error at the tth acquisition time, represents the reconstruction error at the tbth acquisition time, represents the predicted reconstruction error at the t+1th acquisition time;

[0088] Then the threshold δ at time t+1 is t+1 for:

[0089]

[0090] in, represents the standard deviation of the reconstruction error from the tb-th acquisition time to the t-th acquisition time, and κ is the adjustment parameter.

[0091] Furthermore, the loss function used in the training process of the first variational autoencoder is for:

[0092]

[0093] Among them, β is a hyperparameter;

[0094]

[0095] in, is the reconstructed feature during training, and h is the original input feature during training;

[0096]

[0097] in, Denotes the variance σ′ 2 The data of the mth dimension in μ′ m represents the data of the mth dimension in the mean μ′, M represents the dimension of the latent space, μ′ and σ′ 2 According to f enc (h) The mean and variance of the calculated latent space, f enc (h) represents the output of h after passing through the encoder of the first variational autoencoder;

[0098] The loss function used in the training process of the second variational autoencoder is for:

[0099]

[0100] Among them, h abnormal is the original non-anomalous sensor data input to the second variational autoencoder, The sensor data reconstructed by the decoder of the second variational autoencoder.

[0101] The beneficial effects of the present invention are:

[0102] The present invention proposes a sensor data processing method based on variational autoencoder VAE, which can accurately model the real distribution of non-Gaussian distributed sensor data without relying on the Gaussian distribution assumption, and can process non-Gaussian data that is easily affected by complex environments, thereby significantly improving the accuracy and reliability of anomaly detection and adapting to more complex noise distributions. For anomaly detection and repair of high-dimensional unstructured data, the present invention adopts a local feature extraction network based on deep learning, which can capture subtle changes and abnormal patterns of high-dimensional point clouds and image data, and can perform feature repair after detecting anomalies, ensuring the stable and continuous operation of the system in complex environments. The present invention also proposes a dynamic error distribution control mechanism, which adjusts the detection threshold in real time based on time series modeling, enhances the adaptability and robustness of the system to environmental changes, and reduces the possibility of false detection and missed detection. Moreover, the present invention also designs a unified multimodal data feature fusion framework, which realizes the dynamic fusion of GNSS data, IMU data, lidar data and image data features through an attention mechanism, and can improve the anomaly detection capability based on fusion features. The second variational autoencoder uses the complementary information of each sensor to provide compensation when repairing sensor abnormal data, thereby improving the robustness of the system.

[0103] Moreover, the method of the present invention can not only adapt to common abnormal situations, but also has zero-sample prediction capabilities, can flexibly identify new types of abnormalities in unknown environments, and achieve good generalization performance. This feature makes the system more stable and reliable in complex and unpredictable real-world scenarios. Compared with existing methods, the method of the present invention has significant improvements in multi-sensor fusion, real-time performance, robustness, and generalization capabilities, and is particularly suitable for real-time and changeable scenarios such as autonomous driving and robot navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 It is a flow chart of an anomaly detection and repair method for non-Gaussian multimodal sensor data of the present invention;

[0105] Figure 2 It is a flowchart of data feature extraction. DETAILED DESCRIPTION

[0106] Specific implementation method 1: Combination Figure 1 The present embodiment describes an anomaly detection and repair method for non-Gaussian multimodal sensor data, and the method specifically includes the following steps:

[0107] Step 1: Collect IMU data, GNSS data, point cloud data and image data of the unmanned system, pre-process the collected IMU data, GNSS data, point cloud data and image data respectively, and time synchronize the pre-processed IMU data, GNSS data, point cloud data and image data;

[0108] Step 2: Extract the features of IMU data, GNSS data, point cloud data and image data at each acquisition moment after time synchronization;

[0109] Step 3: For each acquisition moment, the features of the IMU data, GNSS data, point cloud data, and image data are fused to obtain the feature fusion results for each acquisition moment.

[0110] Step 4: Perform anomaly detection on the data at each collection time according to the feature fusion results. If anomalies are detected in the data, data repair is required. If no anomalies are detected in the data, data repair is not required.

[0111] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the collected IMU data, GNSS data, point cloud data and image data are pre-processed respectively, specifically:

[0112] The collected IMU data, GNSS data, point cloud data and image data are denoised respectively.

[0113] The other steps and parameters are the same as those in the first embodiment.

[0114] Different denoising methods can be used according to the data characteristics of different sensors. For example, for IMU data, a low-pass filter can be used to filter out high-frequency noise; for GNSS data, a Kalman filter can be used for smoothing. For IMU data, GNSS data, point cloud data, and image data, statistical methods (triple standard deviation principle) and machine learning algorithms (isolation forest) can be used to detect and remove outliers. Using the data with the lowest acquisition frequency as the alignment benchmark, synchronize the timestamps of data from different sensors, remove the data corresponding to the unaligned timestamps, and obtain an aligned multi-sensor data set, which can ensure the temporal and spatial consistency of the data from each sensor.

[0115] Specific implementation method three: Combination Figure 2 The present embodiment is different from the first or second embodiment in that the feature extraction method of the time-synchronized point cloud data is:

[0116] Step 1: Record the point cloud data corresponding to any acquisition time as Among them, P represents the point cloud, p i represents the three-dimensional coordinates of the i-th point, represents a real number, and N represents the number of points;

[0117] In order to capture local geometric information, find i The k points with the smallest Euclidean distance are used as pi The k nearest neighbor points form p i The local neighborhood

[0118] Based on the local neighborhood of each point Constructing the neighborhood index matrix Among them, I i,j Represents point p i The index of the jth neighbor of ;

[0119] Step 2: For point p i , respectively calculate the local neighborhood Each point in the i The coordinates of Δp ij =p j -p i , Δp ij Represents a local neighborhood The jth point in the i The coordinates of

[0120] P i and Δp ij Perform splicing to obtain the splicing result Then use the multi-layer perceptron (MLP) to stitch the results Perform nonlinear transformation to obtain point p i The nonlinear transformation result of

[0121] Step 3: Nonlinear transformation results Processing, get point p i The local feature representation h i ;

[0122]

[0123] Step 4: Perform global average pooling on the local feature representations of all points in P to obtain the global feature h global ; Then according to the global feature h global Get the final feature representation of each point:

[0124] h′ i =[h i ,h global ]

[0125] Among them, h′ i It's point p i The final feature representation, h′ i Not only does it include point p i It not only contains the local information of the point cloud, but also the global context information of the entire point cloud data;

[0126] The final feature representations of all points in the point cloud data corresponding to the current acquisition time are spliced ​​to obtain the point cloud data features at the current acquisition time;

[0127] Step 5: Use the methods of steps 1 to 4 to extract features from the point cloud data collected at each moment.

[0128] The other steps and parameters are the same as those in the first or second embodiment.

[0129] The method of this embodiment is also applicable to feature extraction of image data, in which case each pixel point in the image is equivalent to a point with two-dimensional coordinates.

[0130] Specific implementation method 4: This implementation method is different from the specific implementation methods 1 to 3 in that the splicing result is processed by using a multi-layer perceptron. Perform nonlinear transformation and obtain nonlinear transformation result Specifically:

[0131]

[0132] Among them, σ is the activation function (such as ReLU), L is the number of layers of the multi-layer perceptron, Indicates the splicing result The output after the lth layer in the multi-layer perceptron is W (l) represents the weight matrix of the lth layer in the multilayer perceptron (the matrix contains the connection weights from the previous layer to the current layer, and each element represents the influence of a certain node on the nodes in the next layer), b (l) Represents the bias vector of the lth layer in the multilayer perceptron (the bias term is an additional parameter used to adjust the output value of the activation function), Indicates the splicing result The output after the l-1th layer in the multi-layer perceptron.

[0133] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0134] Specific implementation method five: Combination Figure 2 This embodiment is different from the first to fourth embodiments in that the feature extraction method of the IMU data after time synchronization is:

[0135] Step (1), time series data composed of time-synchronized IMU data Among them, x t represents the observed value at time t, D is the data dimension at each moment, and T is the length of the time series;

[0136] Step (2): In order to reduce noise and data dimension, the time series data X ts Perform normalization:

[0137]

[0138] Among them, x t,d Represents x t The data of the dth dimension in μ d Represents X ts The mean of the data in the dth dimension, σ d Represents X ts The standard deviation of the data in the dth dimension, Represents the data of the dth dimension after normalization at the tth moment;

[0139] Then the data at time t after normalization is

[0140] Step (3), use the quantization function q(·) to Convert to discrete Token sequence s t :

[0141]

[0142] Where Q represents the number of discrete intervals, b1, b2, …, b Q-1 To represent the boundary of the discrete interval (which can be set according to actual conditions), s t =[s t,1 ,s t,2 ,…,s t,D ];

[0143] Step (4), s t Transformed into an embedding vector e through the embedding layer of the Transformer model t :

[0144] e t =Embedding(s t )

[0145] Among them, Embedding(·) represents the embedding layer;

[0146] In order to introduce time position information, we give e t Add position code p t Get the position embedding vector

[0147]

[0148] Then embed the position into the vector Input into the encoder of the Transformer model to obtain the feature representation h of the time series data at time t t :

[0149]

[0150] Among them, TransformerEncoder(·) represents the encoder of the Transformer model.

[0151] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0152] The present invention selects the Transformer model for feature extraction. The multi-head self-attention mechanism of the Transformer model can effectively capture long-range dependencies and key patterns in time series. The method of this embodiment can obtain the features of the IMU data at each moment, wherein the dimension of the IMU data at each moment is 6 dimensions. Similarly, the method of this embodiment can obtain the features of the GNSS data at each moment, wherein the dimension of the GNSS data at each moment is 3 dimensions.

[0153] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the specific process of step 3 is as follows:

[0154] For any collection time, the features of the IMU data, GNSS data, point cloud data, and image data at the current collection time are recorded as H1, H2, H3, and H4, respectively, and the attention score of each feature is calculated respectively:

[0155]

[0156] Among them, e′ p It is feature H p , p = 1, 2, 3, 4; W represents the learnable parameter matrix, v and b represent the learnable parameter vectors, the superscript T represents the transpose, and tanh(·) is the hyperbolic tangent function;

[0157] The softmax function is used to normalize the attention score of each feature to obtain the attention weight α of each feature. p :

[0158]

[0159] Use the attention weight to weight the features and get the fused feature H fusion :

[0160]

[0161] Similarly, the fused features at each acquisition moment are obtained respectively.

[0162] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0163] This implementation can flexibly adjust the weights of each sensor feature according to its importance in the current task. When the sensor data quality is high or contains more key information, the corresponding feature weight will be assigned a higher value, otherwise the weight will be reduced to reduce the negative impact of abnormal sensors on the overall system. This fully utilizes the information complementarity of multiple sensors, ensures that the fused features contain rich information from multiple sensors, and suppress the interference of noise and redundant data, and realizes the targeted fusion of different sensor features. In addition, the attention mechanism also enhances the flexibility and interpretability of feature fusion, so that the model can better adapt to the changes in multimodal data in different environments and scenarios.

[0164] The fused feature representation is more comprehensive and discriminative, significantly improving the accuracy and robustness of anomaly detection. In addition, the adaptability of the system is enhanced, so that when the quality of some sensor data deteriorates or fails, the system can automatically rely on the data of other normal sensors to maintain the stability of the overall performance. This multimodal feature fusion method based on the attention mechanism not only optimizes the quality of feature representation, but also provides more reliable and rich input features for subsequent anomaly detection and repair, significantly improving the performance and reliability of the entire system.

[0165] Specific implementation method 7: This implementation method is different from the specific implementation methods 1 to 6 in that the data at each acquisition time is respectively detected for abnormality based on the feature fusion result, specifically:

[0166] The fusion feature H obtained at any data collection time fusion For example

[0167] The fusion feature H fusion As the input of the first variational autoencoder (VAE), H fusion The output of the encoder after the first variational self-encoder is recorded as f enc (H fusion );

[0168] According to f enc (H fusion ) Calculate the mean μ and variance σ of the latent space 2 :

[0169] μ=f enc (H fusion )

[0170] logσ2 =f enc (H fusion )

[0171] The latent variables are sampled using the reparameterization technique:

[0172]

[0173] Among them, z is the latent variable, ⊙ represents the Hadamard product, and ∈ represents the variable that obeys the standard normal distribution;

[0174] The decoder of the first variational autoencoder is used to map the sampled latent variable z back to the original feature space to obtain the input feature H fusion The reconstruction result:

[0175]

[0176] Among them, f dec (·) is the decoder of the first variational autoencoder;

[0177] The reconstruction error for:

[0178]

[0179] Among them, ||·|| represents the 2-norm;

[0180] Comparing reconstruction errors With the size of the threshold δ, if the reconstruction error If it is greater than the threshold δ, there is abnormal data in the IMU data, GNSS data, point cloud data and image data at the current acquisition time; otherwise, there is no abnormal data in the IMU data, GNSS data, point cloud data and image data at the current acquisition time.

[0181] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0182] The method of this embodiment can be used to determine whether the data at each acquisition moment is abnormal. Through this non-parametric modeling method, VAE can adapt to complex non-Gaussian data distributions, capture potential structures and patterns in sensor features, and the reconstruction error can effectively reflect the difference between the input data and the normal data distribution learned by the model, thereby achieving high-accuracy anomaly detection. Moreover, no additional classifier is required, which simplifies the model structure, reduces the computational complexity, and makes the detection process more efficient and simple. In addition, the reconstruction error, as a direct anomaly indicator, provides an intuitive assessment of the degree of anomaly, which helps to quickly locate and respond to potential sensor failures.

[0183] Specific implementation eight: This implementation differs from specific implementations one to seven in that the specific process of data repair is as follows:

[0184] Take the data at any collection time as an example

[0185] The IMU data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the IMU data;

[0186] Using the GNSS data collected at this moment as the input of the decoder of the second variational autoencoder to obtain a reconstruction result of the GNSS data;

[0187] The image data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain a reconstruction result of the image data;

[0188] The point cloud data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the point cloud data.

[0189] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0190] According to the fusion features, it can only be determined whether there is abnormal data in the data at the current acquisition time, but it is impossible to determine which sensor's data is abnormal. Therefore, in this embodiment, the data of each sensor is passed through the decoder of the second VAE. Since the second VAE is trained based on the normal data distribution, when the abnormal data is input, the potential representation will be guided to the normal distribution range during training, and the reconstruction result generated by the decoder will be more inclined to the normal data feature. Therefore, the output of the second VAE has a natural abnormal repair effect. Therefore, the present invention utilizes the generation ability of the second VAE to enable it to reconstruct and repair the abnormal data and generate features consistent with the normal data distribution. Even if some sensors fail or the data quality decreases, the system can still rely on the data of other normal sensors to maintain normal operation, which significantly improves the robustness and reliability of the system. In addition, since the second VAE is distributed and trained based on the normal data of all sensors, the information complementarity of multiple sensors can be fully utilized when repairing the abnormal data. The repaired data not only makes up for the lack of abnormal sensors, but also enhances the stability and accuracy of the overall system, ensuring that the system can continue to provide high-quality positioning and perception services in a complex and dynamically changing environment. The anomaly detection and repair module achieves efficient detection and timely repair of sensor anomalies through accurate identification of reconstruction errors and effective use of multi-sensor features, thereby improving the reliability and safety of the entire system.

[0191] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the method for obtaining the threshold δ is:

[0192]

[0193] Among them, AdvancedTSModel(·) represents a time series model (such as a Transformer model, LSTM model, or other efficient time series prediction models), θ ts represents the parameters of the time series model, b represents the window size of the historical time step, represents the reconstruction error at the tth acquisition time, represents the reconstruction error at the tbth acquisition time, represents the predicted reconstruction error at the t+1th acquisition time;

[0194] Then the threshold δ at time t+1 is t+1 for:

[0195]

[0196] in, represents the standard deviation of the reconstruction error from the tbth acquisition time to the tth acquisition time, and κ is an adjustment parameter used to control the sensitivity of the threshold.

[0197] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0198] In a multi-sensor anomaly detection and repair system, the error distribution of sensor data often changes dynamically with time and environment. Static anomaly detection models are difficult to adapt to such changes, resulting in a decrease in detection performance. The present invention uses advanced time series modeling technology to dynamically model and predict the distribution of reconstruction errors. The model can capture the time series characteristics and trends of reconstruction errors and predict the reconstruction error of the next time step, thereby adjusting the model parameters and anomaly detection thresholds in real time to ensure that anomaly detection maintains high robustness, accuracy and adaptability under different environmental and time conditions.

[0199] In terms of adaptability, dynamic error distribution control enables the model to flexibly adjust the detection threshold according to the real-time changes in the error distribution, thereby significantly improving the robustness of the detection and reducing the occurrence of false positives and false negatives. In terms of early warning capability, by predicting the changing trend of the reconstruction error in advance, the system can adjust the model parameters in a timely manner to avoid the degradation of detection performance caused by the change of the error distribution, and ensure the stable operation of the system in a complex and dynamic environment. By combining time series prediction with dynamic adjustment of thresholds, the system's sensitivity and response speed to abnormal patterns are enhanced, making the overall abnormality detection and repair process more efficient and reliable. Therefore, the dynamic error distribution control module plays a key role in improving the robustness and adaptability of the multi-sensor system, and provides a solid guarantee for the reliability and safety of the system in practical applications.

[0200] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 1 to 9 in that the loss function used in the training process of the first variational autoencoder is for:

[0201]

[0202] Among them, β is a hyperparameter;

[0203] Used to measure the reconstruction features during training The difference between the original input feature h is:

[0204]

[0205] in, is the reconstructed feature during training, and h is the original input feature during training;

[0206] The KL divergence loss This makes the distribution of latent variables close to the standard normal distribution:

[0207]

[0208] in, Denotes the variance σ′ 2 The data of the mth dimension in μ′ m represents the data of the mth dimension in the mean μ′, M represents the dimension of the latent space, μ′ and σ′ 2 According to f enc (h) The mean and variance of the calculated latent space, f enc (h) represents the output of h after passing through the encoder of the first variational autoencoder;

[0209] The loss function used in the training process of the second variational autoencoder is for:

[0210]

[0211] Among them, h abnormal is the original non-anomalous sensor data input to the second variational autoencoder, The sensor data reconstructed by the decoder of the second variational autoencoder.

[0212] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0213] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for anomaly detection and repair of non-Gaussian multimodal sensor data, characterized in that: The method specifically comprises the following steps: Step 1: Collect IMU data, GNSS data, point cloud data and image data of the unmanned system, pre-process the collected IMU data, GNSS data, point cloud data and image data respectively, and time synchronize the pre-processed IMU data, GNSS data, point cloud data and image data; Step 2: Extract the features of IMU data, GNSS data, point cloud data and image data at each acquisition moment after time synchronization; Step 3: For each acquisition moment, the features of the IMU data, GNSS data, point cloud data, and image data are fused to obtain the feature fusion results for each acquisition moment. Step 4: Perform anomaly detection on the data at each collection time according to the feature fusion results. If anomalies are detected in the data, data repair is required. If no anomalies are detected in the data, data repair is not required.

2. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 1, characterized in that: The preprocessing of the collected IMU data, GNSS data, point cloud data and image data is specifically as follows: The collected IMU data, GNSS data, point cloud data and image data are denoised respectively.

3. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 2, characterized in that: The feature extraction method of the time-synchronized point cloud data is: Step 1: Record the point cloud data corresponding to any acquisition time as Among them, P represents the point cloud, p i represents the three-dimensional coordinates of the i-th point, represents a real number, and N represents the number of points; Find the value in P that matches p i The k points with the smallest Euclidean distance are used as p i The k nearest neighbor points form p i The local neighborhood Based on the local neighborhood of each point Constructing the neighborhood index matrix Among them, I i,j Represents point p i The index of the jth neighbor of ; Step 2: For point p i , respectively calculate the local neighborhood Each point in the i The coordinates of Δp ij =p j -p i , Δp ij Represents a local neighborhood The jth point in the i The coordinates of P i and Δp ij Perform splicing to obtain the splicing result Then use the multi-layer perceptron to stitch the results Perform nonlinear transformation to obtain point p i The nonlinear transformation result of Step 3: Nonlinear transformation results Processing, get point p i The local feature representation h i ; Step 4: Perform global average pooling on the local feature representations of all points in P to obtain the global feature h global ; Then according to the global feature h global Get the final feature representation of each point: h′ i [h i ,h global ] Among them, h′ i It's point p i The final feature representation of The final feature representations of all points in the point cloud data corresponding to the current acquisition time are spliced ​​to obtain the point cloud data features at the current acquisition time; Step 5: Use the methods of steps 1 to 4 to extract features from the point cloud data collected at each moment.

4. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 3, characterized in that: The splicing results are obtained by using a multi-layer perceptron Perform nonlinear transformation and obtain nonlinear transformation result Specifically: Among them, σ is the activation function, L is the number of layers of the multilayer perceptron, Indicates the splicing result The output after the lth layer in the multi-layer perceptron is W (l) represents the weight matrix of the lth layer in the multilayer perceptron, b (l) represents the bias vector of the lth layer in the multilayer perceptron, Indicates the splicing result The output after the l-1th layer in the multi-layer perceptron.

5. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 1, characterized in that: The feature extraction method of the IMU data after time synchronization is: Step (1), time series data composed of time-synchronized IMU data Among them, x t represents the observed value at time t, D is the data dimension at each moment, and T is the length of the time series; Step (2): For the time series data X ts Perform normalization: Among them, x t,d Represents x t The data of the dth dimension in μ d Represents X ts The mean of the data in the dth dimension, σ d Represents X ts The standard deviation of the data in the dth dimension, Represents the data of the dth dimension after normalization at the tth moment; Then the data at time t after normalization is Step (3), use the quantization function q(·) to Convert to discrete Token sequence s t : Where Q represents the number of discrete intervals, b1, b2, …, b Q-1 To represent the boundary of the discrete interval, s t =[s t,1 ,s t,2 ,…,s t,D ]; Step (4): t Transformed into an embedding vector e through the embedding layer of the Transformer model t : e t =Embedding(s t ) Among them, Embedding(·) represents the embedding layer; Give e t Add position code p t Get the position embedding vector Then embed the position into the vector Input into the encoder of the Transformer model to obtain the feature representation h of the time series data at time t t : Among them, TransformerEncoder(·) represents the encoder of the Transformer model.

6. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 4 or 5, characterized in that: The specific process of step three is: For any collection time, the features of the IMU data, GNSS data, point cloud data, and image data at the current collection time are recorded as H1, H2, H3, and H4, respectively, and the attention score of each feature is calculated respectively: Among them, e′ p It is feature H p Attention score, p = 1, 2, 3, 4; W represents the parameter matrix, v and b represent the parameter vectors, the superscript T represents the transpose, and tanh(·) is the hyperbolic tangent function; The softmax function is used to normalize the attention score of each feature to obtain the attention weight α of each feature. p : Use the attention weight to weight the features and get the fused feature H fusion : Similarly, the fused features at each acquisition moment are obtained respectively.

7. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 6, characterized in that: The anomaly detection is performed on the data at each collection moment according to the feature fusion result, specifically: The fusion feature H obtained at any data collection time fusion For example The fusion feature H fusion As the input of the first variational autoencoder, H fusion The output of the encoder after the first variational self-encoder is recorded as f enc (H fusion ); According to f enc (H fusion ) Calculate the mean μ and variance σ of the latent space 2 : μ=f enc (H fusion ) logσ 2 =f enc (H fusion ) The latent variables are sampled using the reparameterization technique: Among them, z is the latent variable, ⊙ represents the Hadamard product, and ∈ represents the variable that obeys the standard normal distribution; The decoder of the first variational autoencoder is used to map the sampled latent variable z back to the original feature space to obtain the input feature H fusion The reconstruction result: Among them, f dec (·) is the decoder of the first variational autoencoder; The reconstruction error for: Among them, ||·|| represents the 2-norm; Comparing reconstruction errors With the size of the threshold δ, if the reconstruction error If it is greater than the threshold δ, there is abnormal data in the IMU data, GNSS data, point cloud data and image data at the current acquisition time; otherwise, there is no abnormal data in the IMU data, GNSS data, point cloud data and image data at the current acquisition time.

8. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 7, characterized in that: The specific process of data repair is as follows: Take the data at any collection time as an example The IMU data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the IMU data; Using the GNSS data collected at this moment as the input of the decoder of the second variational autoencoder to obtain a reconstruction result of the GNSS data; The image data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain a reconstruction result of the image data; The point cloud data collected at this moment is used as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the point cloud data.

9. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 8, characterized in that: The method for obtaining the threshold δ is: Among them, AdvancedTSModel(·) represents the time series model, θ ts represents the parameters of the time series model, b represents the window size of the historical time step, represents the reconstruction error at the tth acquisition time, represents the reconstruction error at the tbth acquisition time, represents the predicted reconstruction error at the t+1th acquisition time; Then the threshold δ at time t+1 is t+1 for: in, represents the standard deviation of the reconstruction error from the tb-th acquisition time to the t-th acquisition time, and κ is the adjustment parameter.

10. The method for anomaly detection and repair of non-Gaussian multimodal sensor data according to claim 9, characterized in that: The loss function used in the training process of the first variational autoencoder is for: Among them, β is a hyperparameter; in, is the reconstructed feature during training, and h is the original input feature during training; in, Denotes the variance σ′ 2 The data of the mth dimension in μ′ m represents the data of the mth dimension in the mean μ′, M represents the dimension of the latent space, μ′ and σ′ 2 According to f enc (h) The mean and variance of the calculated latent space, f enc (h) represents the output of h after passing through the encoder of the first variational autoencoder; The loss function used in the training process of the second variational autoencoder is for: Among them, h abnormal is the original non-anomalous sensor data input to the second variational autoencoder, The sensor data reconstructed by the decoder of the second variational autoencoder.

Images