Anomaly detection and repair method for non-gaussian multi-modal sensor data

Abstract

The application discloses an abnormality detection and repair method for non-Gaussian multi-modal sensor data, and belongs to the technical field of multi-source autonomous navigation of unmanned systems.The application solves the problems that the existing method performs poorly when facing non-Gaussian data, is difficult to perform abnormality detection on high-dimensional unstructured data, has insufficient robustness of a static abnormality detection model, and lacks redundancy compensation when repairing abnormal sensor data.The application specifically comprises the following steps:Step 1, collecting IMU, GNSS, point cloud and image data of an unmanned system, and then performing preprocessing and time synchronization on the collected data;Step 2, extracting features of the IMU, GNSS, point cloud and image data at each collection time after time synchronization;Step 3, fusing the data features at each collection time;Step 4, performing abnormality detection on the data at each collection time according to the fusion result, and further performing data repair if it is detected that the data is abnormal.The method can be applied to data abnormality detection and repair.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multi-source autonomous navigation of unmanned systems, and particularly relates to an anomaly detection and repair method for non-Gaussian multi-modal sensor data. BACKGROUND

[0002] In positioning systems, anomaly detection and repair of sensor data is a key issue to improve system reliability and accuracy. The environment of unmanned driving and robot navigation is often complex and full of uncertainties, such as the shielding of GNSS signals by buildings in the city, the influence of night and bad weather on visual sensors, the interference of ground bumps on inertial measurement units (IMU), etc. In addition, due to the aging of sensor hardware or accidental equipment failure, sensor data may also appear to drift, jump and other anomalies. With the dynamic changes of the environment and the non-ideal performance of the sensors, these factors will cause the data accuracy to decrease, the noise to increase, and even significant abnormal data to appear, which poses a potential threat to the accuracy of positioning and the safety of the system.

[0003] The current multi-sensor fusion positioning system relies mainly on classical state estimation methods such as Kalman Filter, Particle Filter, Factor Graph Optimization, etc. These methods usually assume that the sensor data satisfies the Gaussian distribution and the noise follows a specific model. Although these assumptions can effectively improve the positioning accuracy in some cases, when the data appears to mutate or be abnormal, the tolerance of these algorithms to abnormal data is limited, which may lead to an increase in cumulative error or even system crash. In particular, in the application of autonomous vehicles and mobile robots, timely detection and repair of abnormal sensor data are particularly important to ensure the safe driving and navigation of vehicles.

[0004] In practical applications, the data types of different sensors and their anomaly detection methods differ greatly. For example, IMU and GNSS produce time series data, and by analyzing the time continuity and trend of the data, anomalies can be detected, such as detecting sudden acceleration or angular velocity changes in IMU, or detecting drift or mutations in GNSS signals. Cameras and Lidar produce non-time series data such as images and point clouds, and their anomaly detection needs to rely on image processing and deep learning techniques, such as identifying occlusions or artifacts in images through convolutional neural networks (CNN), or detecting abnormal points in point clouds using clustering algorithms.

[0005] However, the existing data processing methods have the following problems:

[0006] (1) Insufficient non-Gaussian data processing

[0007] Traditional anomaly detection methods, such as Kalman filter and its extended versions (e.g., EKF), usually assume that data noise and errors follow a Gaussian distribution. This Gaussian assumption simplifies the calculation in theory and helps the convergence of the model, but in real-world environments, this assumption often cannot be established. In particular, for GNSS data, the signal is easily disturbed by multipath effects, obstructions, and random reflections in complex environments, resulting in non-Gaussian characteristics such as skewed distribution, heavy-tailed distribution, etc. In addition, sensors may be affected by nonlinear system characteristics and sudden environmental changes, resulting in large jumps or drifts in data, and these anomalies often do not conform to the Gaussian distribution. Existing Gaussian assumption models perform poorly when faced with these anomalies, making it difficult to effectively deal with sudden non-Gaussian noise, resulting in increased positioning errors, and even causing system failure.

[0008] (2) Difficulty in detecting anomalies in high-dimensional unstructured data

[0009] Image data generated by cameras and point cloud data generated by lidar are typical high-dimensional unstructured data, and there are unique challenges in anomaly detection. Traditional anomaly detection methods mainly rely on statistical analysis or low-dimensional feature extraction, which are difficult to effectively apply to high-dimensional, unstructured data. In image data, obstructions, lighting changes, and dynamic objects can easily cause fluctuations in image quality, and existing methods are difficult to accurately identify abnormal regions in such cases. Similarly, point cloud data often exhibits sparsity and irregularity, and when the point cloud contains a large number of noise or interference points, traditional anomaly detection algorithms are difficult to effectively distinguish between environmental changes and true sensor anomalies. In addition, the high-dimensional nature of image and point cloud data makes the anomaly detection computation complexity high, and real-time performance is difficult to guarantee, resulting in limited application in unmanned vehicle and robot navigation. To address these challenges, there is an urgent need to develop efficient high-dimensional data anomaly detection algorithms to improve the adaptability of the system to complex environments.

[0010] As multi-modal sensors (e.g. GNSS, IMU, lidar, camera) play important roles in autonomous driving and robot navigation, they provide different dimensions of environmental information. However, the data types of these sensors differ significantly, with both time-series data from IMU and GNSS, and non-time-series data (e.g. images and point clouds) from lidar and camera. Existing anomaly detection methods are mostly optimized for a single sensor type, such as processing IMU data through Kalman filtering, or analyzing camera data through image processing methods. Therefore, existing anomaly detection methods have certain limitations in uniformity and applicability, and it is difficult to meet the needs of anomaly data processing of different types of sensors. Moreover, this single processing method lacks the ability to share and fuse information between different sensors, resulting in a lack of redundancy compensation when a certain sensor data is abnormal. Especially in autonomous driving applications with high real-time requirements, due to the lack of cross-modal fusion framework, the data between sensors is difficult to process uniformly, limiting the effectiveness of multi-sensor fusion anomaly detection and reducing the robustness of the overall system. Therefore, further development of cross-modal data fusion technology is needed to achieve unified processing of multi-sensor data in complex environments using deep learning algorithms to address the variability and uncertainty challenges in the positioning process. This not only helps to improve the safety of autonomous driving technology, but also promotes the development of robot autonomous navigation systems, enabling their application in a wider range of fields.

[0011] At the same time, during the operation of unmanned vehicles and robots, the error distribution of sensor data will fluctuate constantly with changes in the external environment. Different environmental conditions, such as temperature, humidity, light intensity, and terrain changes, can cause significant changes in the error characteristics of sensor data. For example, IMU data drifts more severely on bumpy roads, GNSS produces multipath errors in urban canyons, and lidar is more prone to increased noise in rainy and snowy weather. Static anomaly detection models often lack robustness due to their inability to adapt to such dynamic changes in real time. As the external environment continues to change, fixed error models gradually fail to accurately describe the actual distribution characteristics of sensor data, leading to a decline in the effectiveness of anomaly detection. To improve the robustness of the positioning system, the anomaly detection system needs to have adaptive capabilities to dynamically adjust the detection strategy based on real-time environmental feedback, adapt to the dynamic changes in error distribution, and ensure the accuracy and stability of the system. SUMMARY

[0012] The present application aims to solve the problems of existing data anomaly detection methods, which perform poorly when facing non-Gaussian data, are difficult to detect anomalies in high-dimensional unstructured data, lack robustness in static anomaly detection models, and lack redundancy compensation when repairing sensor anomaly data. A method for anomaly detection and repair of non-Gaussian multi-modal sensor data is proposed.

[0013] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0014] A method for anomaly detection and repair of non-Gaussian multimodal sensor data, the method specifically includes the following steps:

[0015] Step 1: Collect IMU data, GNSS data, point cloud data, and image data from the unmanned system. Then, preprocess the collected IMU data, GNSS data, point cloud data, and image data respectively, and synchronize the preprocessed IMU data, GNSS data, point cloud data, and image data in time.

[0016] Step 2: Extract features from IMU data, GNSS data, point cloud data, and image data at each acquisition time after time synchronization;

[0017] Step 3: For each acquisition time, feature fusion is performed on the features of IMU data, GNSS data, point cloud data, and image data to obtain the feature fusion results for each acquisition time.

[0018] Step 4: Perform anomaly detection on the data at each acquisition time based on the feature fusion results. If anomalies are detected, data repair is required. If no anomalies are detected, data repair is not required.

[0019] Furthermore, the preprocessing of the collected IMU data, GNSS data, point cloud data, and image data is as follows:

[0020] The collected IMU data, GNSS data, point cloud data, and image data were denoised separately.

[0021] Furthermore, the feature extraction method for the time-synchronized point cloud data is as follows:

[0022] Step 1: Record the point cloud data corresponding to any acquisition time as Where P represents a point cloud, p i Represents the three-dimensional coordinates of the i-th point. Let N represent a real number, and let N represent the number of points.

[0023] Find the match between P and p. i The k points with the smallest Euclidean distance are used as p. i The k nearest neighbors are p. i local neighborhood

[0024] Based on the local neighborhood of each point Constructing a neighborhood index matrix Among them, I i,j Point p i The index of the j-th neighbor;

[0025] Step 2, for point p i Calculate the local neighborhood respectively Each point within is relative to point p i The coordinates, i.e., Δp ij =p j -p i , Δp ij Represents local neighborhood The j-th point relative to point p i The coordinates;

[0026] For p i and Δp ij The pieces are then stitched together to obtain the stitched result. Then, a multilayer perceptron is used to analyze the stitching results. Perform a nonlinear transformation to obtain point p. i Nonlinear transformation results

[0027] Step 3: Process the nonlinear transformation results After processing, point p is obtained. i Local feature representation h i ;

[0028]

[0029] Step 4: Perform global average pooling on the local feature representations of all points in P to obtain the global feature h. global Then, based on the global feature h global The final feature representation of each point is obtained:

[0030] h′ i =[h i ,h global ]

[0031] Where, h′ i It is point p i The final feature representation;

[0032] The final feature representations of all points in the point cloud data corresponding to the current acquisition time are concatenated to obtain the point cloud data features at the current acquisition time.

[0033] Step 5: Using the methods from Step 1 to Step 4, extract features from the point cloud data collected at each time point.

[0034] Furthermore, the stitching result is processed using a multilayer perceptron. Perform a nonlinear transformation to obtain the nonlinear transformation result. Specifically,

[0035]

[0036] where σ is an activation function, L is the number of layers of the multi-layer perception, denotes the splicing result the output after passing through the lth layer of the multi-layer perception, W (l) denotes the weight matrix of the lth layer of the multi-layer perception, b (l) denotes the bias vector of the lth layer of the multi-layer perception, denotes the splicing result the output after passing through the l-1th layer of the multi-layer perception.

[0037] Further, the feature extraction method of the time-synchronized IMU data is:

[0038] Step (1), time series data composed of time-synchronized IMU data where x t denotes the observation value at the tth moment, D is the data dimension at each moment, and T is the length of the time series;

[0039] Step (2), normalizing the time series data X ts :

[0040]

[0041] where x t,d denotes the data of the dth dimension in x t , μ d denotes the mean of the data of the dth dimension in X ts , σ d denotes the standard deviation of the data of the dth dimension in X ts , denotes the data of the dth dimension after normalization at the tth moment;

[0042] Then the data after normalization at the tth moment is

[0043] Step (3), converting X into a discrete Token sequence s t using a quantization function q(·):

[0044]

[0045] where Q denotes the number of discrete intervals, b1, b2, …, b Q-1 denote the boundaries of the discrete intervals, and s t =[st,1 s t,2 ,…,s t,D ];

[0046] Step (4), converting s t into an embedding vector e t by an embedding layer of the Transformer model:

[0047] e t =Embedding(s t )

[0048] where Embedding(·) represents the embedding layer;

[0049] Adding a position encoding p t to e t to obtain a position embedding vector

[0050]

[0051] The position embedding vector is then input into an encoder of the Transformer model to obtain a feature representation h t of the time series data at the t-th time point:

[0052]

[0053] where TransformerEncoder(·) represents the encoder of the Transformer model.

[0054] Further, the specific process of step three is as follows:

[0055] For any collection time, the features of the IMU data, GNSS data, point cloud data and image data at the current collection time are denoted as H1, H2, H3 and H4 respectively, and the attention scores of each feature are calculated respectively:

[0056]

[0057] where e′ p is the attention score of the feature H p , p = 1, 2, 3, 4; W represents a parameter matrix, v and b represent parameter vectors, the superscript T represents transposition, and tanh(·) is a hyperbolic tangent function;

[0058] The attention scores of each feature are normalized by using a softmax function to obtain the attention weight a p of each feature:

[0059]

[0060] The features are weighted and summed using attention weights to obtain the fused features H. fusion :

[0061]

[0062] Similarly, the fused features at each acquisition time are obtained.

[0063] Furthermore, the step of performing anomaly detection on the data at each acquisition time based on the feature fusion results specifically involves:

[0064] The fusion feature H obtained at any data acquisition time fusion For example

[0065] Fusion feature H fusion As the input to the first variational autoencoder, H fusion The output of the encoder after the first variational autoencoder is denoted as f. enc (H fusion );

[0066] Then according to f enc (H fusion Calculate the mean μ and variance σ of the latent space. 2 :

[0067] μ = f enc (H fusion )

[0068] logσ 2 =f enc (H fusion )

[0069] Latent variables are obtained through sampling using reparameterization techniques:

[0070]

[0071] Where z is a latent variable, ⊙ represents the Hadamard product, and ∈ represents a variable that follows a standard normal distribution;

[0072] The decoder of the first variational autoencoder then maps the sampled latent variable z back to the original feature space to obtain the input feature H. fusion Reconstruction results:

[0073]

[0074] Among them, f dec (·) is the decoder of the first variational autoencoder;

[0075] Then the reconstruction error for:

[0076]

[0077] where ||·|| denotes the 2-norm;

[0078] Comparing the reconstruction error with the size of the threshold value δ, if the reconstruction error is greater than the threshold value δ, there is abnormal data in the IMU data, GNSS data, point cloud data and image data of the current acquisition time, otherwise there is no abnormal data in the IMU data, GNSS data, point cloud data and image data of the current acquisition time.

[0079] Further, the specific process of the data repair is:

[0080] Taking the data of any acquisition time as an example

[0081] The IMU data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the IMU data is obtained;

[0082] The GNSS data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the GNSS data is obtained;

[0083] The image data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the image data is obtained;

[0084] The point cloud data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the point cloud data is obtained.

[0085] Further, the threshold value δ is obtained by:

[0086]

[0087] where AdvancedTSModel(·) represents a time series model, θ ts represents the parameters of the time series model, b represents the window size of the historical time step, represents the reconstruction error of the tthacquisition time, represents the reconstruction error of the t-bthacquisition time, represents the predicted reconstruction error of the t+1thacquisition time;

[0088] The threshold value δ of the t+1thtime is t+1 :

[0089]

[0090] where, is the standard deviation of reconstruction error from the t-bth acquisition time to the tth acquisition time, and κ is a regulation parameter.

[0091] Further, the loss function adopted by the first variational autoencoder in the training process is:

[0092]

[0093] where β is a hyperparameter;

[0094]

[0095] where, is the reconstructed feature during training, and h is the input original feature during training;

[0096]

[0097] where, represents the variance σ' 2 of the mth dimension of the data, μ' m represents the mean of the mth dimension of the data, M represents the dimension of the latent space, and μ' and σ' 2 are the mean and variance of the latent space calculated according to f enc (h), respectively. enc (h) represents the output of h after passing through the encoder of the first variational autoencoder.

[0098] The loss function adopted by the second variational autoencoder in the training process is:

[0099]

[0100] where h abnormal is the original non-anomalous sensor data input into the second variational autoencoder, is the reconstructed sensor data after the decoder of the second variational autoencoder.

[0101] The beneficial effects of the present application are:

[0102] The application provides a sensor data processing method based on a variational autoencoder VAE, which can accurately model the real distribution of non-Gaussian distributed sensor data without relying on Gaussian distribution assumption, can process non-Gaussian data which is easily affected by complex environment, and thus significantly improves the accuracy and reliability of abnormality detection and adapts to more complex noise distribution. For abnormality detection and repair of high-dimensional unstructured data, the application adopts a local feature extraction network based on deep learning, which can capture subtle changes and abnormal patterns of high-dimensional point cloud and image data, and can perform feature repair after detecting abnormality, thereby ensuring stable and continuous operation of the system in a complex environment. The application also provides a dynamic error distribution control mechanism, which adjusts the detection threshold in real time based on time series modeling, enhances the adaptability and robustness of the system to environmental changes, and reduces the possibility of false detection and missed detection. Moreover, the application also designs a unified multi-modal data feature fusion framework, which realizes dynamic fusion of GNSS data, IMU data, lidar data and image data features through an attention mechanism, and based on the fused features, the abnormality detection capability can be improved, and the second variational autoencoder provides compensation when repairing abnormal sensor data based on the complementary information of each sensor, thereby improving the robustness of the system.

[0103] Moreover, the method of the application not only can adapt to common abnormal situations, but also has zero sample prediction capability, can flexibly identify new types of abnormalities in unknown environments, and realizes good generalization performance. This feature makes the system more stable and reliable in complex and unpredictable real scenes. The method of the application has obvious improvement in multi-sensor fusion, real-time performance, robustness and generalization capability compared with existing methods, and is particularly suitable for real-time and variable scenes such as automatic driving and robot navigation. BRIEF DESCRIPTION OF DRAWINGS

[0104] Figure 1 is a flowchart of a non-Gaussian multi-modal sensor data abnormality detection and repair method of the application;

[0105] Figure 2 is a flowchart of data feature extraction. DETAILED DESCRIPTION

[0106] Specific implementation one: combined with Figure 1 The application provides a non-Gaussian multi-modal sensor data abnormality detection and repair method, which specifically comprises the following steps:

[0107] Step one, collect IMU data, GNSS data, point cloud data and image data of an unmanned system, then respectively pre-process the collected IMU data, GNSS data, point cloud data and image data, and time synchronize the pre-processed IMU data, GNSS data, point cloud data and image data;

[0108] Step two, extract the features of the IMU data, GNSS data, point cloud data and image data of each collection time after time synchronization respectively;

[0109] Step three, for each collection time, the features of the IMU data, GNSS data, point cloud data and image data are fused to obtain the feature fusion result of each collection time;

[0110] Step four, according to the feature fusion result, the data of each collection time is detected for abnormality, if the data is detected to be abnormal, data repair is needed, if the data is detected to be normal, data repair is not needed.

[0111] Specific implementation method two: the difference between this implementation method and specific implementation method one is that the collected IMU data, GNSS data, point cloud data and image data are respectively preprocessed, specifically:

[0112] The collected IMU data, GNSS data, point cloud data and image data are respectively denoised.

[0113] The other steps and parameters are the same as those in specific implementation method one.

[0114] According to the data characteristics of different sensors, different denoising methods can be used. For example, for IMU data, a low-pass filter can be used to filter out high-frequency noise; for GNSS data, a Kalman filter can be used for smoothing. For IMU data, GNSS data, point cloud data and image data, statistical methods (three times standard deviation principle) and machine learning algorithms (isolated forest) can be used to detect and eliminate outliers. With the lowest data collection frequency as the alignment reference, the timestamps of different sensors are synchronized, the data corresponding to the timestamps that are not aligned are eliminated, and the aligned multi-sensor data set is obtained, which can ensure the spatio-temporal consistency of the data of each sensor.

[0115] Specific implementation method three: combined with Figure 2 This implementation method is explained. The difference between this implementation method and specific implementation method one or two is that the feature extraction method of the time-synchronized point cloud data is:

[0116] Step 1, the point cloud data corresponding to any one collection time is denoted as P Where P represents the point cloud, p i represents the three-dimensional coordinates of the i-th point, represents a real number, and N represents the number of points;

[0117] In order to capture local geometric information, find the k points with the smallest Euclidean distance from p i in P to be pi The k nearest neighbors are p. i local neighborhood

[0118] Based on the local neighborhood of each point Constructing a neighborhood index matrix Among them, I i,j Point p i The index of the j-th neighbor;

[0119] Step 2, for point p i Calculate the local neighborhood respectively Each point within is relative to point p i The coordinates, i.e., Δp ij =p j -p i Δp ij Represents local neighborhood The j-th point relative to point p i The coordinates;

[0120] For p i and Δp ij The pieces are then stitched together to obtain the stitched result. Then, a multilayer perceptron (MLP) is used to analyze the stitching results. Perform a nonlinear transformation to obtain point p. i Nonlinear transformation results

[0121] Step 3: Process the nonlinear transformation results After processing, point p is obtained. i Local feature representation h i ;

[0122]

[0123] Step 4: Perform global average pooling on the local feature representations of all points in P to obtain the global feature h. global Then, based on the global feature h global The final feature representation of each point is obtained:

[0124] h′ i =[h i ,h global ]

[0125] Where, h′ i It is point p i The final characteristic representation, h′ i It not only includes point p i The local information also includes the global context information of the entire point cloud data;

[0126] The final feature representation of all points in the point cloud data corresponding to the current collection time is spliced to obtain the point cloud data feature of the current collection time;

[0127] Step 5, using the method of steps 1 to 4, the point cloud data collected at each time is respectively extracted.

[0128] The other steps and parameters are the same as those in embodiment one or two.

[0129] The method of the embodiment is also applicable to feature extraction of image data, and each pixel point in the image corresponds to a point with two-dimensional coordinates.

[0130] Embodiment four: different from one of embodiments one to three, the nonlinear transformation of the spliced result is performed by using a multi-layer perception machine to obtain a nonlinear transformation result Specifically:

[0131]

[0132] Wherein, σ is an activation function (such as ReLU), L is the number of layers of the multi-layer perception machine, represents the output after the spliced result passes through the lth layer of the multi-layer perception machine, W (l) represents the weight matrix of the lth layer of the multi-layer perception machine (the matrix includes the connection weight from the previous layer to the current layer, and each element represents the influence degree of a node on the next layer node), b (l) represents the bias vector of the lth layer of the multi-layer perception machine (the bias term is an additional parameter used to adjust the output value of the activation function), represents the output after the spliced result passes through the (l-1)th layer of the multi-layer perception machine.

[0133] The other steps and parameters are the same as one of embodiments one to three.

[0134] Embodiment five: in combination Figure 2 with the above, the embodiment is described. Different from one of embodiments one to four, the feature extraction method of the time-synchronized IMU data is:

[0135] Step (1), the time series data composed of the time-synchronized IMU data Wherein, x t represents the observation value at the tth time, D is the data dimension of each time, and T is the length of the time series;

[0136] Step (2), in order to reduce noise and data dimension, the time series data X ts needs to be normalized:

[0137]

[0138] where x t,d represents the data of the dth dimension in x t , μ d represents the mean of the data of the dth dimension in X ts , σ d represents the standard deviation of the data of the dth dimension in X ts , and x t represents the data of the dth dimension after normalization at the tth moment.

[0139] Then the data at the tth moment after normalization is

[0140] Step (3), convert X t into discrete Token sequence s using quantization function q(·):

[0141]

[0142] where Q represents the number of discrete intervals, b1, b2, …, b Q-1 represent the boundaries of the discrete intervals (which can be set according to actual conditions), and s t = [s t,1 , s t,2 , …, s t,D ].

[0143] Step (4), convert s t into embedding vector e t through the embedding layer of the Transformer model:

[0144] e t = Embedding(s t )

[0145] where Embedding(·) represents the embedding layer.

[0146] In order to introduce time position information, add position encoding p t to e t to obtain position embedding vector

[0147]

[0148] Then convert the position embedding vector The input is input into an encoder of a Transformer model to obtain a feature representation h of the time series data at the t-th moment t :

[0149]

[0150] wherein, TransformerEncoder(·) represents an encoder of the Transformer model.

[0151] The other steps and parameters are the same as one of the first to fourth embodiments.

[0152] The Transformer model is selected for feature extraction in the present application, and the multi-head self-attention mechanism of the Transformer model can effectively capture long-range dependencies and key patterns in time series. The method of the present embodiment can obtain the feature of the IMU data at each moment, wherein the dimension of the IMU data at each moment is 6 dimensions. Similarly, the method of the present embodiment can obtain the feature of the GNSS data at each moment, wherein the dimension of the GNSS data at each moment is 3 dimensions.

[0153] The present embodiment is different from one of the first to fifth embodiments in that the specific process of step three is:

[0154] For any collection moment, the features of the IMU data, GNSS data, point cloud data and image data of the current collection moment are denoted as H1, H2, H3 and H4 respectively, and the attention scores of each feature are calculated respectively:

[0155]

[0156] wherein, e′ p is the attention score of the feature H p , p = 1, 2, 3, 4; W represents a learnable parameter matrix, v and b represent learnable parameter vectors, the superscript T represents transposition, and tanh(·) is a hyperbolic tangent function.

[0157] The attention scores of each feature are normalized by using a softmax function to obtain the attention weight a p :

[0158]

[0159] The features are weighted and summed by using the attention weight to obtain the fused feature H fusion :

[0160]

[0161] Similarly, the fused features at each acquisition time are obtained.

[0162] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0163] This implementation can flexibly adjust the weights of each sensor feature based on its importance in the current task. When sensor data is of high quality or contains more critical information, its corresponding feature weight will be assigned a higher value; conversely, its weight will be reduced to minimize the negative impact of abnormal sensors on the overall system. This fully utilizes the complementary information of multiple sensors, ensuring that the fused features not only contain rich information from multiple sensors but also suppress noise and redundant data interference, achieving targeted fusion of different sensor features. Furthermore, the attention mechanism enhances the flexibility and interpretability of feature fusion, enabling the model to better adapt to multimodal data changes in different environments and scenarios.

[0164] The fused feature representation is more comprehensive and discriminative, significantly improving the accuracy and robustness of anomaly detection. Furthermore, it enhances the system's adaptability, enabling it to automatically rely on data from other functioning sensors to maintain overall performance stability when data quality deteriorates or malfunctions in certain sensors. This attention-based multimodal feature fusion method not only optimizes the quality of feature representation but also provides more reliable and abundant input features for subsequent anomaly detection and repair, significantly improving the overall system performance and reliability.

[0165] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the anomaly detection is performed on the data at each acquisition time based on the feature fusion result, specifically as follows:

[0166] The fusion feature H obtained at any data acquisition time fusion For example

[0167] Fusion feature H fusion As the input to the first variational autoencoder (VAE), H fusion The output of the encoder after the first variational autoencoder is denoted as f. enc (H fusion );

[0168] Then according to f enc (H fusion Calculate the mean μ and variance σ of the latent space. 2 :

[0169] μ = f enc (H fusion )

[0170] logσ2 = f enc (H fusion )

[0171] The latent variable is sampled by a reparameterization trick:

[0172]

[0173] where z is the latent variable, ⊙ denotes the Hadamard product, and ∈ denotes a variable subject to a standard normal distribution;

[0174] The latent variable z sampled is mapped back to the original feature space by using the decoder of the first variational autoencoder to obtain the reconstruction result of the input feature H fusion :

[0175]

[0176] where f dec (·) is the decoder of the first variational autoencoder;

[0177] The reconstruction error is:

[0178]

[0179] where ||·|| denotes the 2-norm;

[0180] The reconstruction error is compared with a threshold value δ, and if the reconstruction error is greater than the threshold value δ, there is abnormal data in the IMU data, GNSS data, point cloud data and image data at the current collection time, otherwise there is no abnormal data in the IMU data, GNSS data, point cloud data and image data at the current collection time.

[0181] The other steps and parameters are the same as one of the first to sixth embodiments.

[0182] The method of the embodiment can respectively judge whether the data at each collection time is abnormal. Through this non-parametric modeling method, the VAE can adapt to complex non-Gaussian data distribution, capture the potential structure and pattern in the sensor features, and the reconstruction error can effectively reflect the difference between the input data and the normal data distribution learned by the model, thereby realizing high-accuracy anomaly detection. Moreover, no additional classifier is needed, which simplifies the model structure, reduces the computational complexity, and makes the detection process more efficient and simple. In addition, the reconstruction error as a direct anomaly indicator provides intuitive anomaly degree evaluation, which helps to quickly locate and respond to potential sensor failures.

[0183] Embodiment eight: different from one of embodiments one to seven, the specific process of the data repair is:

[0184] Taking the data at any one collection time as an example

[0185] The IMU data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the IMU data is obtained;

[0186] The GNSS data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the GNSS data is obtained;

[0187] The image data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the image data is obtained;

[0188] The point cloud data collected at this time is taken as the input of the decoder of the second variational autoencoder, and the reconstruction result of the point cloud data is obtained.

[0189] The other steps and parameters are the same as one of embodiments one to seven.

[0190] According to the fusion features, it can only be determined whether there is abnormal data in the data collected at the current collection time, but it cannot be determined which sensor data is abnormal, therefore, in this embodiment, the data of each sensor is subjected to the decoder of the second VAE, since the second VAE is trained based on normal data distribution, when abnormal data is input, the latent representation will be guided to the normal distribution range during training, and the reconstruction result generated by the decoder will be more inclined to normal data features. Therefore, the output of the second VAE has a natural abnormality repair effect. Therefore, the application utilizes the generation ability of the second VAE to enable it to reconstruct and repair abnormal data, and generate features consistent with normal data distribution. Even in the case of failure of some sensors or degradation of data quality, the system can still rely on the data of other normal sensors to maintain normal operation, significantly improving the robustness and reliability of the system. In addition, since the second VAE is trained based on the normal data distribution of all sensors, the information complementarity of multiple sensors can be fully utilized when repairing abnormal data, and the repaired data not only makes up for the loss of abnormal sensors, but also enhances the stability and accuracy of the overall system, ensuring that the system can continuously provide high-quality positioning and perception services in complex and dynamic environments. The abnormality detection and repair module realizes efficient detection and timely repair of sensor abnormalities through accurate identification of reconstruction errors and effective use of multi-sensor features, improving the reliability and safety of the entire system.

[0191] Embodiment nine: different from one of embodiments one to eight, the threshold value δ is obtained by:

[0192]

[0193] where AdvancedTSModel(·) represents a time series model (such as a Transformer model, an LSTM model, or other efficient time series prediction models), θ ts denotes the parameters of the time series model, b denotes the window size of the historical time steps, denotes the reconstruction error at the t-th acquisition time, denotes the reconstruction error at the t-b-th acquisition time, denotes the predicted reconstruction error at the t+1-th acquisition time;

[0194] then the threshold value δ t+1 at the t+1-th time is:

[0195]

[0196] where, denotes the standard deviation of the reconstruction errors from the t-b-th acquisition time to the t-th acquisition time, and κ is an adjustment parameter used to control the sensitivity of the threshold value.

[0197] The other steps and parameters are the same as one of the first to eighth embodiments.

[0198] In the multi-sensor anomaly detection and repair system, the error distribution of sensor data often dynamically adjusts with time and environmental changes. Static anomaly detection models are difficult to adapt to such changes, leading to a decline in detection performance. The present application uses advanced time series modeling techniques to dynamically model and predict the distribution of reconstruction errors. The model can capture the temporal characteristics and trends of reconstruction errors and predict the reconstruction error at the next time step, thereby adjusting the model parameters and anomaly detection threshold in real time, ensuring that anomaly detection remains highly robust, accurate, and adaptable under different environmental and temporal conditions.

[0199] In terms of adaptability, dynamic error distribution control enables the model to flexibly adjust the detection threshold according to real-time changes in error distribution, significantly improving detection robustness and reducing false positives and false negatives. In terms of early warning capability, by predicting the trend of reconstruction error changes in advance, the system can adjust model parameters in time to avoid detection performance degradation due to changes in error distribution, ensuring stable operation of the system in complex and dynamic environments. By combining time series prediction and dynamic threshold adjustment, the system's sensitivity to abnormal patterns and response speed are enhanced, making the overall anomaly detection and repair process more efficient and reliable. Therefore, the dynamic error distribution control module plays a key role in improving the robustness and adaptability of multi-sensor systems, providing a solid guarantee for the reliability and safety of the system in practical applications.

[0200] Specific implementation ten: different from one of the specific implementations one to nine, the loss function used in the first variational autoencoder training process is:

[0201]

[0202] Where β is a hyperparameter;

[0203] The difference between the reconstructed feature and the input original feature h during training is measured:

[0204]

[0205] Where, is the reconstructed feature during training, and h is the input original feature during training;

[0206] And the KL divergence loss encourages the distribution of the latent variable to approach the standard normal distribution:

[0207]

[0208] Where, represents the variance σ' 2 of the mth dimension of the data, μ' m represents the mean of the mth dimension of the data μ', M represents the dimension of the latent space, μ' and σ' 2 are the mean and variance of the latent space calculated according to f enc (h), respectively. enc (h) represents the output of h after the encoder of the first variational autoencoder.

[0209] The loss function used in the second variational autoencoder training process is:

[0210]

[0211] Where h abnormal is the original non-anomalous sensor data input into the second variational autoencoder, is the sensor data reconstructed by the decoder of the second variational autoencoder.

[0212] The other steps and parameters are the same as one of the specific implementations one to nine.

[0213] The above calculation examples of the present application are only used to illustrate the calculation model and calculation process of the present application, and are not used to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art, and all the embodiments cannot be exhausted here. Any obvious changes or variations derived from the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A method for anomaly detection and repair for non-Gaussian multi-modal sensor data, the method comprising: The method specifically comprises the following steps: Step 1, collecting IMU data, GNSS data, point cloud data and image data of the unmanned system, and respectively pre-processing the collected IMU data, GNSS data, point cloud data and image data, and time synchronizing the pre-processed IMU data, GNSS data, point cloud data and image data; Step 2, respectively extracting features of the IMU data, GNSS data, point cloud data and image data at each collection time after time synchronization; The feature extraction method of the time-synchronized point cloud data is: Step 1, the point cloud data corresponding to any one collection time is recorded as wherein P represents the point cloud, p i represents the i-th point, the point p i is presented in the form of three-dimensional coordinates, represents a real number, and N represents the number of points; Find the k points in P that are closest to p i with the smallest Euclidean distance i The k nearest neighbors of p form the local neighborhood of p i ​ based on local neighborhoods of the respective points constructing a neighborhood index matrix where I i,j denotes the index of the j-th neighbor of point p i ​ Step 2, for point p i , calculate the coordinates of each point within the local neighborhood , respectively, relative to point p i , i.e. Δp ij denotes the coordinates of the jth point within the local neighborhood , relative to point p i . p i and Δp ij are spliced to obtain a splicing result The splicing result is subjected to nonlinear transformation by using a multilayer perception machine to obtain a nonlinear transformation result of the point p i ​​ Step 3, processing the non-linear transform result to obtain a local feature representation h i of the point p i ; Step 4, global average pooling on the local feature representation of all points in P to get global feature h global ; then according to the global feature h global get the final feature representation of each point: h' i = [h i , h global ] where h ′ i is the final feature representation of point p i ; Splicing the final feature representation of all points in the point cloud data corresponding to the current collection time to obtain the point cloud data feature of the current collection time; Step 5, using the method of steps 1 to 4, respectively extracting features of the point cloud data collected at each time; Step 3, for each collection time, features of the IMU data, GNSS data, point cloud data and image data are fused to obtain a feature fusion result of each collection time; Step 4, respectively detecting anomalies of the data of each collection time according to the feature fusion result, if it is detected that the data has an anomaly, data repair is required, if it is detected that the data has no anomaly, data repair is not required.

2. The method of claim 1, wherein, The pre-processing of the collected IMU data, GNSS data, point cloud data and image data is specifically: Respectively denoising the collected IMU data, GNSS data, point cloud data and image data.

3. The method of claim 2, wherein, The method comprises the following steps: acquiring a plurality of images of a target object; performing image stitching on the plurality of images to obtain a stitching result; and performing nonlinear transformation on the stitching result to obtain a nonlinear transformation result. performing nonlinear transformation on the stitching result to obtain a nonlinear transformation result Specifically, the method comprises the following steps: where σ is an activation function, L is the number of layers of the multi-layer perceptron, denotes the concatenation result denotes the output after the l-th layer of the multi-layer perceptron, W (l) denotes the weight matrix of the l-th layer of the multi-layer perceptron, b (l) denotes the bias vector of the l-th layer of the multi-layer perceptron, denotes the concatenation result denotes the output after the l-th layer of the multi-layer perceptron, W 4. The method of claim 1, wherein, The feature extraction method of the time-synchronized IMU data is: Step (1), time series data composed of the IMU data after time synchronization wherein x t denotes the observation value at the t-th time point, d is the data dimension at each time point, and T is the length of the time series Step (2), normalizing the time series data X ts Normalization: where x t,d represents the data in the dth dimension of X t d represents the mean of the data in the dth dimension of X ts d represents the standard deviation of the data in the dth dimension of X ts represents the data in the dth dimension of X​​​ The data at the t-th moment after normalization is Step (3), converting the quantized function q(·) to a discrete Token sequence s t :​ Where Q represents the number of discrete intervals, b1, b2, ..., b Q-1 s represents the boundary of a discrete interval. t =[s t,1 ,s t,2 ,…,s t,D ]; Step (4), s t converted into an embedding vector e by an embedding layer of the Transformer model t : e t = Embedding(s t ) Wherein, Embedding(·) represents an embedding layer; To e t Add position encoding p t Get position embedding vector : A set of position embedding vectors is input into an encoder of a Transformer model to obtain a feature representation h t : Wherein, TransformerEncoder(·) represents an encoder of a Transformer model.

5. The method of claim 3 or 4, wherein, The specific process of step 3 is: For any collection time, the features of the IMU data, GNSS data, point cloud data and image data of the current collection time are denoted as H1, H2, H3 and H4 respectively, and the attention score of each feature is calculated respectively: e′ p = v T tanh(WH p + b) where e'p p is the attention score of feature H p p = 1, 2, 3, 4; W denotes a parameter matrix, v and b denote parameter vectors, the upper index T represents transposition, and tanh(·) is a hyperbolic tangent function. The attention weight of each feature is obtained by normalizing the attention score of each feature by using a softmax function p : The features are weighted and summed using attention weights to obtain fused features H fusion : Similarly, the fused features of each collection time are obtained.

6. The method of claim 5, wherein, The specific process of step 4 is: The fusion feature H obtained at any one data collection time fusion For example, The fused features H fusion As input to the first variational autoencoder, H fusion The output of the encoder of the first variational autoencoder is denoted by f enc (H fusion ). Again according to f enc (H fusion ) compute the mean μ and variance σ of the latent space 2 : μ = f enc (H fusion ) log σ 2 = f enc (H fusion ) The latent variable z is obtained by sampling through the reparameterization trick: Wherein, z is a latent variable, and represents a Hadamard product; ∈ represents a variable subject to a standard normal distribution; And using the decoder of the first variational autoencoder to map the sampled latent variable z back to the original feature space, obtain the reconstruction result of the input feature H fusion : wherein f dec (·) is the decoder of the first variational autoencoder; then the reconstruction error is: Wherein, ||·|| represents a 2-norm; reconstruction error With the size of the threshold δ, if the reconstruction error is greater than the threshold δ, there is abnormal data in the IMU data, GNSS data, point cloud data and image data of the current acquisition time, otherwise there is no abnormal data in the IMU data, GNSS data, point cloud data and image data of the current acquisition time.

7. The method of anomaly detection and repair for non-Gaussian multi-modal sensor data according to claim 6, wherein, The specific process of data repair is: Taking the data of any collection time as an example, The IMU data collected at this time is taken as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the IMU data; The GNSS data collected at this time is taken as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the GNSS data; The image data collected at this time is taken as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the image data; The point cloud data collected at this time is taken as the input of the decoder of the second variational autoencoder to obtain the reconstruction result of the point cloud data.

8. The method of claim 7, wherein, The threshold δ is obtained by: where AdvancedTSModel(·) denotes a time series model, θ ts denotes parameters of the time series model, b denotes a window size of historical time steps, denotes a reconstruction error at the t-th acquisition time, denotes a reconstruction error at the t-b-th acquisition time, denotes a predicted reconstruction error at the t+1-th acquisition time; the threshold value δ at the t+1 time is t+1 is: wherein, denotes the standard deviation of the reconstruction errors from the t-bth acquisition time to the tth acquisition time, k is an adjustment parameter.

9. The method of claim 8, wherein, The loss function employed by the first variational autoencoder training process is: Wherein, β is a hyperparameter; wherein, is the reconstructed feature at training time, h is the input raw feature at training time; wherein, denotes the data in the mth dimension of the variance σ'2, μ' m denotes the data in the mth dimension of the mean μ', M denotes the dimension of the latent space, μ' and σ' 2 are the mean and variance of the latent space computed according to f enc (h) are the mean and variance of the latent space computed according to f enc (h) denotes the output of h after passing through the encoder of the first variational autoencoder; The loss function employed by the second variational autoencoder training process is: wherein h abnormal is the original non-anomalous sensor data input to the second variational autoencoder, is the reconstructed sensor data from the decoder of the second variational autoencoder.

Images