Data loop-oriented multi-modal sporadic anomaly environment generalization perception method and system

Abstract

The application discloses a kind of multi-modal sporadic abnormal environment generalization perception methods and systems for data closed loop, it is related to automatic driving and intelligent transportation system technical field.The method comprises: using a variety of sensors real-time collection and pre-processing environmental data, obtains the high-dimensional feature representation of each mode coding through the feature extraction module of perception model;Utilize the sporadic abnormality in high-dimensional feature representation to be identified by exception detection model, when detecting abnormality, determine abnormal state by reconstruction difference and trigger adaptive adjustment mechanism;Through multi-modal data fusion technology, the fusion feature is obtained by fusing each mode feature;Through environmental generalization learning algorithm, introduce environment-related constraint loss, prompt the perception model to learn environment-independent general features to optimize perception algorithm.The application aims to realize the accurate detection and efficient processing of sporadic abnormality in complex environment through data closed loop feedback and environmental generalization learning, improve the accuracy and robustness of perception task.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving and intelligent transportation systems, and in particular to a multimodal generalized perception method and system for sporadic abnormal environments oriented towards data closed loop. Background Technology

[0002] In autonomous driving and intelligent transportation systems, environmental perception is a core technology for ensuring the safe and reliable operation of the system. Existing technologies typically utilize cameras, LiDAR, and millimeter-wave radar to provide environmental information, and perform target localization and classification through data fusion.

[0003] However, single sensors have limitations. Cameras are susceptible to lighting conditions at night or in extreme weather, lidar has insufficient reflectivity in certain environments, and millimeter-wave radar may exhibit multipath effects in complex urban environments. Existing deep learning-based multimodal fusion methods often require large amounts of costly labeled data for supervised learning and are inadequate in dealing with sporadic anomalies such as sensor failures or sudden environmental changes, lacking the ability to adjust and process in real time.

[0004] Therefore, how to improve the robustness of the sensing system in complex environments and achieve efficient handling of occasional anomalies has become an urgent technical challenge. Summary of the Invention

[0005] The main objective of this invention is to provide a generalized sensing method and system for multimodal occasional anomalies in data closed-loop environments, aiming to improve the robustness of the sensing system in complex environments and achieve efficient handling of occasional anomalies.

[0006] To achieve the above objectives, this invention proposes a generalized perception method for multimodal sporadic anomaly environments oriented towards data closure, comprising the following steps:

[0007] Step S1: Collect environmental data in real time using multiple sensors, perform time consistency alignment and preprocessing on the environmental data, and encode and process the environmental data through the feature extraction module of the perception model to obtain high-dimensional feature representations after encoding each modality.

[0008] Step S2: Use the anomaly detection model to identify occasional anomalies in the encoded high-dimensional feature representation. When an anomaly is detected, enter the data closed-loop feedback stage. Adjust the anomaly sensor parameters, fusion weights, or model parameters through an adaptive adjustment mechanism to handle the occasional anomalies. The anomaly detection model reconstructs the high-dimensional feature representation of each modality and determines the anomaly state based on the reconstruction differences.

[0009] Step S3: Perform weighted fusion of the high-dimensional feature representations of each modality according to the fusion weights corresponding to each modality to obtain the fused features. ,in,

[0010]

[0011] For multimodal fusion features, For modal indexing, For modal sets, For modality High-dimensional feature representation, For modality The fusion weights, and ;

[0012] Step S4: Perform environment generalization learning based on data from different environments, and feed the learning results back to the perception model to optimize the perception algorithm and output target detection, localization or classification perception results for decision control of autonomous vehicles; wherein, the environment generalization learning algorithm enables the perception model to learn environment-independent general features by introducing environment-related constraint loss.

[0013] Preferably, ensuring the temporal consistency of the environmental data in step S1 specifically involves:

[0014] Synchronize the clocks of each sensor using a precise time protocol and add timestamps to the data collected by each sensor.

[0015] Simultaneously, hardware trigger signals are used to ensure that multiple sensors collect data at the same time.

[0016] Preferably, the preprocessing of the environmental data in step S1 includes:

[0017] For camera data, Gaussian filtering for noise reduction, white balance color correction, and camera calibration for distortion correction are performed sequentially.

[0018] For LiDAR data, a voxel grid filter is used to remove irrelevant point clouds;

[0019] For millimeter-wave radar data, Fourier transform and wavelet transform are used to denoise the echo signal and extract frequency domain features.

[0020] Preferably, the mathematical model used in the preprocessing includes:

[0021] Gaussian filtering process for camera images:

[0022]

[0023] in, pixel coordinates The filtered image pixel values, Coordinates in the original image Pixel value at that location, For Gaussian kernel function, and These are the x and y coordinates of the target pixel, respectively. and These are the horizontal and vertical offset indices within the filter window, respectively. The radius of the filtering window. The standard deviation of the Gaussian kernel;

[0024] Voxel filtering process for lidar data:

[0025]

[0026] in, These are the coordinates of the center point of the voxel after filtering. For voxels The coordinate vector of a point, Indices of voxel interior points The number of points within a voxel. This indicates that the coordinates of points within a voxel are averaged.

[0027] Preferably, the process of obtaining the encoded high-dimensional feature representation through the feature extraction module in step S1 specifically involves:

[0028] Extracting image features from camera data using convolutional neural networks ;

[0029] Extracting geometric features from LiDAR data using point cloud extraction networks The feature extraction process is represented as follows:

[0030]

[0031] in, For the geometric features of the point cloud of lidar, For the first point in the point cloud set One point, For point index, The number of points in the point cloud set. and These are the first-layer perceptron and the second-layer perceptron, respectively. For max pooling operators, This represents a feature set composed of the features of each point;

[0032] Extracting temporal features from millimeter-wave radar data using recurrent neural networks Spatial features of millimeter-wave radar data are extracted using convolutional neural networks. and the timing features Spatial features By splicing or weighted fusion, the comprehensive characteristics of millimeter-wave radar are obtained. .

[0033] Preferably, obtaining the encoded high-dimensional feature representation in step S1 further includes:

[0034] Image features Geometric characteristics of lidar Comprehensive characteristics of millimeter-wave radar Perform fully connected layer mapping separately, with the mapping formula as follows:

[0035]

[0036] in, For modal indexing, , , and These represent camera mode, lidar mode, and millimeter-wave radar mode, respectively. For modality The mapped high-dimensional feature vector, For modality The input feature vector, For modality The corresponding weight matrix, For modality The corresponding bias vector;

[0037] And the mapped features Perform batch normalization:

[0038]

[0039] in, For modality The normalized high-dimensional feature representation is used as the basis for subsequent anomaly detection and fusion processing. ; For feature vectors The mean, For feature vectors standard deviation This is a modal index.

[0040] Preferably, the determination of the abnormal state using reconstruction differences in step S2 specifically involves:

[0041] The high-dimensional feature representations of each modality are input into the encoder to obtain the low-dimensional latent space representation. :

[0042]

[0043] in, This is a low-dimensional latent space representation. For encoder mapping functions, , and These are high-dimensional feature representations for camera mode, lidar mode, and millimeter-wave radar mode, respectively. This represents a feature set consisting of multiple modal features; subsequently, the latent space is represented... The features are re-decoded by the decoder. And calculate the reconstruction error :

[0044]

[0045] in, For reconstruction error, For modality Corresponding reconstruction features;

[0046] when When it is judged as abnormal, among which, For reconstruction error, This is a preset threshold.

[0047] Preferably, the adaptive adjustment mechanism in step S2 specifically includes:

[0048] Sensor data quality adjustment: The impact of abnormal data can be mitigated by adjusting the scanning frequency, sensitivity, or filtering parameters of the sensing algorithm of abnormal sensors.

[0049] Fusion weight adjustment: Based on the reconstruction error of each mode, reduce the fusion weight of abnormal modes and increase the fusion weight of non-abnormal modes;

[0050] Model parameter update: Based on the incremental learning mechanism, the parameters of the perception model are updated using abnormal data that has been verified by labeled or pseudo-labeled data.

[0051] Preferably, the environment-related constraint loss in step S4 is a domain adversarial loss. The calculation formula is as follows:

[0052]

[0053] in, For domain confrontation losses, For expectation operator, For multimodal fusion features, For domain tags, For domain classifiers, The probability of the environment source is output by the domain classifier; the goal of the perceptual model is to minimize the task loss while maximizing the domain adversarial loss, so that the domain classifier cannot distinguish the fused features. The environmental source.

[0054] This application also discloses a multimodal, sporadic anomaly environment generalization perception system for data closure, including:

[0055] The data acquisition module is used to collect environmental data in real time using multiple sensors, and to perform time consistency alignment and preprocessing on the environmental data;

[0056] The feature extraction module is used to encode and process the environmental data to obtain high-dimensional feature representations after encoding each modality;

[0057] An anomaly detection module is used to identify occasional anomalies in the encoded high-dimensional feature representation. When an anomaly is detected, it enters the data closed-loop feedback stage and adjusts the anomaly sensor parameters, fusion weights, or model parameters through an adaptive adjustment mechanism to handle the occasional anomalies. The anomaly detection module reconstructs the high-dimensional feature representation of each modality and determines the anomaly state based on the reconstruction differences.

[0058] The multimodal fusion module is used to perform weighted fusion of the high-dimensional feature representations of each modality according to the fusion weights corresponding to each modality, so as to obtain the fused features;

[0059] The generalization learning module is used to receive data from different environments and train or update the perception model based on the environment generalization learning algorithm to optimize the perception algorithm.

[0060] The above technical solution has the following advantages:

[0061] This invention ensures the accuracy and synchronization of input features by utilizing multiple sensors to collect environmental data in real time and performing time-consistency alignment and preprocessing. High-dimensional feature representations are obtained using the feature extraction module of the perception model, and each modal feature is reconstructed using an anomaly detection model. This enables real-time identification of occasional anomalies such as sensor malfunctions or environmental changes, triggering an adaptive adjustment mechanism for dynamic parameter updates. Multimodal data fusion technology improves the accuracy of perception results, while the environmental generalization learning algorithm introduces environment-related constraint losses to enable the model to learn environment-independent general features, enhancing the system's adaptability to unseen environments. This method constructs an efficient data closed-loop system, reducing dependence on specific environmental distributions while ensuring perception continuity, thus improving the robustness and safety of the autonomous driving system in complex scenarios. Attached Figure Description

[0062] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings, wherein:

[0063] Figure 1This is a flowchart illustrating a multimodal, sporadic, abnormal environment generalization perception method for data closure provided in an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the sensor data preprocessing and feature extraction process provided in an embodiment of the present invention.

[0065] Figure 3 This is a schematic diagram of the anomaly detection and closed-loop feedback mechanism provided in an embodiment of the present invention.

[0066] Figure 4 This is a structural block diagram of a multimodal, sporadic, abnormal environment generalization perception system for data closed-loop systems provided in an embodiment of the present invention.

[0067] 10. Camera; 20. LiDAR; 30. Millimeter-wave radar; 101. Data acquisition module; 102. Feature extraction module; 103. Anomaly detection module; 104. Multimodal fusion module; 105. Generalization learning module. Detailed Implementation

[0068] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0069] The following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should understand that various modifications and equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope thereof.

[0070] In this invention, an occasional anomaly refers to an abnormal state caused by sensor obstruction, dirt, malfunction, rain, fog, strong light, low illumination, multipath effect, or sudden changes in road environment, which causes the distribution shift or reconstruction error of the high-dimensional feature representation of at least one mode to exceed a preset threshold.

[0071] To address the technical challenges of unstable perception accuracy and insufficient handling of occasional anomalies in autonomous driving systems operating under complex conditions, this embodiment provides a multimodal generalized perception method for occasional anomalies oriented towards data closure. This embodiment achieves real-time monitoring and adaptive adjustment of sensor anomalies by constructing an efficient data closure system, while simultaneously enhancing the robustness of the perception model in unseen scenarios through environmental generalization learning.

[0072] Example 1

[0073] like Figures 1 to 4As shown, this embodiment provides a multimodal, sporadic anomaly-based generalized perception method for data closure. Since autonomous vehicles encounter complex situations during actual driving, such as rainy / foggy weather, dirty sensors, and sudden changes in lighting, these sporadic anomalies can cause a sharp decline in perception performance. To address this issue, the method in this embodiment first executes step S1, utilizing multiple sensors to collect environmental data in real time. Specifically, this embodiment uses a camera 10, a LiDAR 20, and a millimeter-wave radar 30 as perception sources. To ensure high consistency of data in the time dimension, the system uses a Precise Time Protocol (PTP) to synchronize the clocks of each sensor and adds a timestamp to the data collected by each sensor. Simultaneously, the system uses hardware trigger signals to ensure that multiple sensors perform acquisition actions at the same time point, thereby ensuring the time consistency of all sensors.

[0074] After acquiring the raw environmental data, this embodiment preprocesses the data to eliminate noise and distortion. For the image data acquired by camera 10, the system sequentially performs Gaussian filtering for noise reduction, white balance color correction, and camera 10 calibration for distortion correction. The Gaussian filtering process for the camera 10 image follows the following mathematical model:

[0075]

[0076] in, pixel coordinates The filtered image pixel values, Coordinates in the original image Pixel value at that location, For Gaussian kernel function, and These are the x and y coordinates of the target pixel, respectively. and These are the horizontal and vertical offset indices within the filter window, respectively. For the summation operator, the symbol " " indicates multiplication, The radius of the filtering window. The standard deviation of the Gaussian kernel is represented by the subscript "filtered," indicating the filtered result. This process effectively suppresses Gaussian noise in the image while preserving edge information.

[0077] For the point cloud data acquired by LiDAR 20, the system uses a voxel grid filter to remove irrelevant point clouds, such as scattered noise points caused by rain and fog. The voxel filtering process for LiDAR 20 data is represented as follows:

[0078]

[0079] in, These are the coordinates of the center point of the voxel after filtering. For voxels The coordinate vector of a point, Indices of voxel interior points The number of points within a voxel. This is an operator that sums the coordinates of points within a voxel. This indicates averaging the coordinates of points within the voxel, with the `filtered` subscript indicating the filtered result. This step reduces the amount of data while preserving the geometric features of the environment.

[0080] For millimeter-wave radar data, the system uses Fourier transform and wavelet transform to denoise the echo signal and extract its frequency domain features. The denoised signal is then analyzed in the frequency domain to obtain the frequency domain features of the echo signal.

[0081] Subsequently, the system encodes the preprocessed environmental data through the feature extraction module of the perception model. Specifically, it uses a convolutional neural network (CNN) to extract image features from the camera data. Geometric features of LiDAR 20 data were extracted using a point cloud extraction network. The feature extraction process is represented as follows:

[0082]

[0083] in, For the geometric features of the point cloud of lidar, For the first point in the point cloud set One point, For point index, The number of points in the point cloud set. and These are the first-layer perceptron and the second-layer perceptron, respectively. For max pooling operators, This represents the feature set composed of the features of each point. For millimeter-wave radar data, the system uses a recurrent neural network (RNN) to extract temporal features. Spatial features are extracted using convolutional neural networks (CNNs). and time series characteristics Spatial features By splicing or weighted fusion, the comprehensive characteristics of millimeter-wave radar are obtained. .

[0084] To achieve a unified representation of features across different modalities, this embodiment focuses on image features. 20 Geometric Features of LiDAR and millimeter-wave radar 30 integrated characteristics Perform fully connected layer mapping separately, with the mapping formula as follows:

[0085]

[0086] in, For modal indexing, , , and These represent camera mode, lidar mode, and millimeter-wave radar mode, respectively. For modality The mapped high-dimensional feature vector, For modality The input feature vector, For modality The corresponding weight matrix, For modality The corresponding bias vector, symbol " "" indicates the multiplication of a matrix and a vector, and the mapping of the features Perform batch normalization:

[0087]

[0088] in, For modality The normalized high-dimensional feature representation is used as the basis for subsequent anomaly detection and fusion processing. , For modality The mapped high-dimensional feature vector, For feature vectors The mean, For feature vectors standard deviation For modal indexing, the fractional representation is used to represent pairs of modal indices. Standardization is then performed. Following this step, each modality is encoded as a high-dimensional feature representation.

[0089] Next, step S2 is executed, using the anomaly detection model to identify occasional anomalies in the encoded high-dimensional feature representation. When an anomaly is detected, the data closed-loop feedback stage is entered, where an adaptive adjustment mechanism is used to adjust the anomaly sensor parameters, fusion weights, or model parameters to handle occasional anomalies. In this embodiment, the anomaly detection model reconstructs the high-dimensional feature representation of each modality and determines the anomaly state based on the reconstruction differences. Specifically, the high-dimensional feature representation of each modality is input into the encoder to obtain the low-dimensional latent space representation. :

[0090]

[0091] in, This is a low-dimensional latent space representation. For encoder mapping functions, , and These are high-dimensional feature representations for camera mode, lidar mode, and millimeter-wave radar mode, respectively. This represents a feature set consisting of multiple modal features. Subsequently, the latent space is represented... The features are re-decoded by the decoder. Next, the reconstruction error is calculated. :

[0092]

[0093] in, For reconstruction error, For modal indexing, For modal sets, , and These represent camera mode, lidar mode, and millimeter-wave radar mode, respectively. For modality High-dimensional feature representation, For modality The corresponding reconstruction features, For the summation operator, This represents the square of the L2 norm.

[0094] when When it is judged as abnormal, among which, For reconstruction error, A preset threshold is used. For example, when the camera lens is obstructed, its feature reconstruction error will increase significantly. After detecting an anomaly, the system initiates an adaptive adjustment mechanism. The adjustment mechanism includes sensor data quality adjustment, that is, adjusting the scanning frequency, sensitivity, or filtering parameters of the sensing algorithm of the anomalous sensor to mitigate the impact of anomalous data. At the same time, the system updates model parameters, that is, based on an incremental learning mechanism, using the anomalous data verified by labeled or pseudo-labeled data to incrementally update the parameters of the sensing model, enabling the model to recognize and process such anomalies.

[0095] Execute step S3, and perform weighted fusion of the high-dimensional feature representations of each modality according to the fusion weights corresponding to each modality to obtain the fused features. ,in , For multimodal fusion features, For modal indexing, For modal sets, For modality High-dimensional feature representation, For modality The fusion weight, For the summation operator, and Feature fusion can combine the advantages of various sensors. For example, when camera 10 is interfered with by strong light, the depth and velocity information of lidar 20 and millimeter-wave radar 30 can be used to maintain perception accuracy.

[0096] Finally, step S4 is executed, where environment generalization learning is performed based on data from different environments, and the learning results are fed back to the perception model to optimize the perception algorithm. The environment generalization learning algorithm introduces environment-related constraint loss to enable the perception model to learn environment-independent general features. Specifically, the environment-related constraint loss is a domain adversarial loss. The calculation formula is as follows:

[0097]

[0098] in, For domain confrontation losses, For expectation operator, For multimodal fusion features, For domain tags, For domain classifiers, The probability of the environment source is output by the domain classifier. The logarithmic operator is used. The goal of the perceptual model is to minimize the task loss while maximizing the domain adversarial loss, so that the domain classifier cannot distinguish the fused features. The environmental sources are complex, such as the inability to distinguish whether the data comes from a sunny or rainy day. In this way, the perceptual model learns universal feature representations, significantly improving the system's perceptual generalization ability in unfamiliar or changing environments.

[0099] The method in this embodiment enables the perception system to autonomously respond to occasional anomalies and continuously evolve from new environments through data closed-loop feedback and environmental generalization learning, which greatly enhances the driving safety of autonomous vehicles.

[0100] Example 2

[0101] This embodiment provides a multimodal, sporadic anomaly environment generalization perception system for data closure, used to implement the perception method described in Embodiment 1 above. (Refer to...) Figure 4The system includes a data acquisition module 101, a feature extraction module 102, an anomaly detection module 103, a multimodal fusion module 104, and a generalization learning module 105. The data acquisition module 101 is mainly used to collect environmental data in real time using multiple sensors and to perform time consistency alignment and preprocessing on the environmental data. The feature extraction module 102 is connected to the data acquisition module 101 and is used to encode and process the environmental data to obtain high-dimensional feature representations after encoding each modality. The anomaly detection module 103 is used to identify occasional anomalies in the encoded high-dimensional feature representations. When an anomaly is detected, the system enters a data closed-loop feedback stage, adjusting the abnormal sensor parameters, fusion weights, or model parameters through an adaptive adjustment mechanism to handle the occasional anomaly. Specifically, the anomaly detection module 103 reconstructs the high-dimensional feature representations of each modality and determines the abnormal state based on the reconstruction differences. The multimodal fusion module 104 is used to perform weighted fusion of the high-dimensional feature representations of each modality according to the fusion weights corresponding to each modality to obtain fused features. The generalization learning module 105 receives data from different environments, introduces environment-related constraint losses based on the environment generalization learning algorithm to train or update the perception model, and feeds the learning results back to the perception model to optimize the perception algorithm. Through its modular design, this system enables autonomous vehicles to dynamically optimize parameters through data closure when facing occasional anomalies such as sensor malfunctions or severe weather, ensuring the continuity and accuracy of perception.

[0102] Example 3

[0103] This embodiment further explains the details of data acquisition and preprocessing based on the above embodiment. To ensure the spatiotemporal alignment accuracy of the sensing system under high-speed driving conditions, the data acquisition module 101 uses the Precision Time Protocol (PTP) for time synchronization during step S1, adding a timestamp to the data acquired by each sensor. The system has an internal clock synchronization unit with a synchronization accuracy within 1ms. Simultaneously, the system uses a hardware trigger signal to ensure that multiple sensors acquire data at the same time point, avoiding environmental snapshot misalignment caused by differences in sensor internal exposure or scanning mechanisms. In the preprocessing stage, for the images acquired by the camera 10, the system sequentially performs Gaussian filtering for noise reduction, white balance color correction, and camera 10 calibration distortion correction. For Gaussian noise, the system applies a Gaussian filtering model for smoothing, and its calculation process follows the formula...

[0104]

[0105] in, pixel coordinates The filtered image pixel values, Coordinates in the original image Pixel value at that location, For Gaussian kernel function, and These are the x and y coordinates of the target pixel, respectively. and These are the horizontal and vertical offset indices within the filter window, respectively. For the summation operator, the symbol " " indicates multiplication, The radius of the filtering window. The standard deviation is the Gaussian kernel value, and the subscript "filtered" indicates the filtered result. For LiDAR 20 data, the system uses a voxel grid filter to remove outliers and irrelevant point clouds caused by raindrop scattering. The voxel filtering process is represented as follows:

[0106]

[0107] in, These are the coordinates of the center point of the voxel after filtering. For voxels The coordinate vector of a point, Indices of voxel interior points The number of points within a voxel. This is an operator that sums the coordinates of points within a voxel. This indicates averaging the coordinates of points within the voxel, with the subscript "filtered" indicating the filtered result. For millimeter-wave radar data, the system uses Fourier transform and wavelet transform to denoise the echo signal in order to extract the frequency domain features of the target.

[0108] Example 4

[0109] This embodiment details the encoding process of the feature extraction module 102. After acquiring the preprocessed data, the feature extraction module 102 uses a convolutional neural network (CNN) to extract image features from the camera 10 data. For the 3D point cloud of LiDAR 20, the system utilizes a point cloud extraction network to extract geometric features. The feature extraction process is represented as follows:

[0110]

[0111] in, For the geometric features of the point cloud of lidar, For the first point in the point cloud set One point, For point index, The number of points in the point cloud set. and These are the first-layer perceptron and the second-layer perceptron, respectively. For max pooling operators, This represents a feature set composed of features at various points. Data processing for the millimeter-wave radar 30 includes extracting temporal features using a recurrent neural network (RNN). Spatial features are extracted using convolutional neural networks (CNNs). and time series characteristics Spatial features By splicing or weighted fusion, the comprehensive characteristics of millimeter-wave radar are obtained. To map features of different dimensions to a unified high-dimensional space, the system performs image feature mapping. 20 Geometric Features of LiDAR Millimeter-wave radar 30 comprehensive characteristics Perform mapping on the fully connected layers separately. The mapping formula is as follows: .in, For modal indexing, , , and These represent camera mode, lidar mode, and millimeter-wave radar mode, respectively. For modality The mapped high-dimensional feature vector, For modality The input feature vector, For modality The corresponding weight matrix, For modality The corresponding bias vector, symbol " "" indicates matrix-vector multiplication. Subsequently, the system processes the mapped features. Perform batch normalization:

[0112]

[0113] in, For modality The normalized high-dimensional feature representation is used as the basis for subsequent anomaly detection and fusion processing. , For modality The mapped high-dimensional feature vector, For feature vectors The mean, For feature vectors standard deviation For modal indexing, the fractional representation is used to represent pairs of modal indices. Standardization is performed. Normalized features can effectively eliminate data bias and improve the convergence speed of training.

[0114] Example 5

[0115] This embodiment further illustrates the closed-loop mechanism of anomaly detection and adaptive adjustment. The anomaly detection module 103 executes step S2 based on the autoencoder structure, determining the anomaly state by reconstructing differences. Specifically, the system inputs the high-dimensional feature representations of each modality into the encoder to obtain the low-dimensional latent space representation. The calculation formula is:

[0116]

[0117] in, This is a low-dimensional latent space representation. For encoder mapping functions, , and These are high-dimensional feature representations for camera mode, lidar mode, and millimeter-wave radar mode, respectively. This represents a feature set consisting of multiple modal features. The decoder then represents the latent space. Re-decode into reconstructed features The system calculates the reconstruction error. To determine whether there are occasional anomalies, the calculation formula is as follows:

[0118]

[0119] in, For reconstruction error, For modal indexing, For modal sets, , and These represent camera mode, lidar mode, and millimeter-wave radar mode, respectively. For modality High-dimensional feature representation, For modality The corresponding reconstruction features, For the summation operator, Let represent the square of the L2 norm. When When it is judged as abnormal, among which, For reconstruction error, A pre-set fluctuation threshold is set for the system. Once an anomaly flag is triggered, the system immediately initiates an adaptive adjustment mechanism. This mechanism includes sensor data quality adjustment; for example, when an anomaly caused by blurring of the camera 10 lens is detected, the system automatically reduces the weight of camera 10 data in the fusion stage and simultaneously adjusts the filtering parameters of its perception algorithm. Simultaneously, the system updates model parameters based on an incremental learning mechanism, using anomaly data verified by labeled or pseudo-labeled data to incrementally update the perception model's parameters. This enables the model to recognize such newly emerging noise patterns, thus maintaining robustness in perception when encountering similar situations subsequently.

[0120] Example 6

[0121] This embodiment illustrates the specific application of the environment generalization learning algorithm. When an autonomous vehicle switches from urban roads to suburban unpaved roads, or from a sunny day to a rainy environment, the generalization learning module 105 executes step S4 to optimize perception capabilities. The system introduces environment-related constraint losses, namely domain adversarial losses. Its calculation formula is

[0122]

[0123] in, For domain confrontation losses, For expectation operator, For multimodal fusion features, Domain labels representing the source of the environment, for example, 0 represents daytime and 1 represents nighttime. For domain classifiers, The probability of the environment source is output by the domain classifier. The logarithmic operator is used. The optimization objective of the perceptual model during training is to minimize the perceptual task loss, such as the cross-entropy loss for object classification, while maximizing the domain adversarial loss. This adversarial training prompts the feature extraction module 102 to mine environment-independent general features, making it impossible for the domain classifier to distinguish the environmental origin of the fused features. In this way, the system can output stable and accurate perceptual features under different environmental conditions. It still has good adaptability to environmental conditions not covered in the training data.

[0124] Finally, this invention provides a generalized perception method and system for multimodal, sporadic anomaly environments oriented towards data closure. Through the detailed description of the above embodiments, this invention can effectively address challenges such as sensor anomalies and sudden environmental changes in autonomous driving scenarios. In practical applications, those skilled in the art can reasonably adjust and combine the above steps or modules according to specific needs, such as limitations on computing resources or the characteristics of specific sensors.

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. It should be noted that no reference numerals in the claims should be considered as limitations on the claims. In this invention, the terms "comprising, including, or any other variation thereof" are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

Claims

1. A generalized perception method for multimodal sporadic anomaly environments oriented towards data closure, characterized in that, Includes the following steps: Step S1: Collect environmental data in real time using multiple sensors, perform time consistency alignment and preprocessing on the environmental data, and encode and process the environmental data through the feature extraction module of the perception model to obtain high-dimensional feature representations after encoding each modality. Step S2: Use the anomaly detection model to identify occasional anomalies in the encoded high-dimensional feature representation. When an anomaly is detected, enter the data closed-loop feedback stage. Adjust the anomaly sensor parameters, fusion weights, or model parameters through an adaptive adjustment mechanism to handle the occasional anomalies. The anomaly detection model reconstructs the high-dimensional feature representation of each modality and determines the anomaly state based on the reconstruction differences. Step S3: Perform weighted fusion of the high-dimensional feature representations of each modality according to the fusion weights corresponding to each modality to obtain the fused features. ,in, For multimodal fusion features, For modal indexing, For modal sets, For modality High-dimensional feature representation, For modality The fusion weights, and ; Step S4: Perform environment generalization learning based on data from different environments, and feed the learning results back to the perception model to optimize the perception algorithm and output target detection, localization or classification perception results for decision control of autonomous vehicles; wherein, the environment generalization learning algorithm enables the perception model to learn environment-independent general features by introducing environment-related constraint loss.

2. The multimodal sporadic anomaly environment generalization perception method for data closure as described in claim 1, characterized in that, In step S1, ensuring the temporal consistency of the environmental data specifically involves: Synchronize the clocks of each sensor using a precise time protocol and add timestamps to the data collected by each sensor. Simultaneously, hardware trigger signals are used to ensure that multiple sensors collect data at the same time.

3. The multimodal sporadic anomaly environment generalization perception method for data closure as described in claim 1, characterized in that, Step S1, which involves preprocessing the environmental data, includes: For camera data, Gaussian filtering for noise reduction, white balance color correction, and camera calibration for distortion correction are performed sequentially. For LiDAR data, a voxel grid filter is used to remove irrelevant point clouds; For millimeter-wave radar data, Fourier transform and wavelet transform are used to denoise the echo signal and extract frequency domain features.

4. The multimodal sporadic anomaly environment generalization perception method for data closure as described in claim 3, characterized in that, The mathematical models used in the preprocessing process include: Gaussian filtering process for camera images: in, pixel coordinates The filtered image pixel values, Coordinates in the original image Pixel value at that location, For Gaussian kernel function, and These are the x and y coordinates of the target pixel, respectively. and These are the horizontal and vertical offset indices within the filter window, respectively. The radius of the filtering window. The standard deviation of the Gaussian kernel; Voxel filtering process for lidar data: in, These are the coordinates of the center point of the voxel after filtering. For voxels The coordinate vector of a point, Indices of voxel interior points The number of points within a voxel. This indicates that the coordinates of points within a voxel are averaged.

5. The multimodal sporadic anomaly environment generalization perception method for data closure as described in claim 1, characterized in that, In step S1, the high-dimensional feature representation after encoding is obtained through the feature extraction module as follows: Extracting image features from camera data using convolutional neural networks ; Extracting geometric features from LiDAR data using point cloud extraction networks The feature extraction process is represented as follows: in, For the geometric features of the point cloud of lidar, For the first point in the point cloud set One point, For point index, The number of points in the point cloud set. and These are the first-layer perceptron and the second-layer perceptron, respectively. For max pooling operators, This represents a feature set composed of the features of each point; Extracting temporal features from millimeter-wave radar data using recurrent neural networks Spatial features of millimeter-wave radar data are extracted using convolutional neural networks. and the timing features Spatial features By splicing or weighted fusion, the comprehensive characteristics of millimeter-wave radar are obtained. .

6. The multimodal sporadic anomaly environment generalization perception method for data closure as described in claim 5, characterized in that, Step S1, obtaining the encoded high-dimensional feature representation, also includes: Image features Geometric characteristics of lidar Comprehensive characteristics of millimeter-wave radar Perform fully connected layer mapping separately, with the mapping formula as follows: in, For modal indexing, , , and These represent camera mode, lidar mode, and millimeter-wave radar mode, respectively. For modality The mapped high-dimensional feature vector, For modality The input feature vector, For modality The corresponding weight matrix, For modality The corresponding bias vector; And the mapped features Perform batch normalization: in, For modality The normalized high-dimensional feature representation is used as the basis for subsequent anomaly detection and fusion processing. ; For feature vectors The mean, For feature vectors standard deviation This is a modal index.

7. A multimodal, sporadic anomaly environment generalization perception method for data closure as described in claim 1, characterized in that, In step S2, determining the abnormal state using reconstruction differences specifically involves: The high-dimensional feature representations of each modality are input into the encoder to obtain the low-dimensional latent space representation. : in, This is a low-dimensional latent space representation. For encoder mapping functions, , and These are high-dimensional feature representations for camera mode, lidar mode, and millimeter-wave radar mode, respectively. The latent space is then represented as a feature set consisting of multiple modal features; subsequently, the latent space is represented... The features are re-decoded by the decoder. And calculate the reconstruction error : in, For reconstruction error, For modality Corresponding reconstruction features; when When it is judged as abnormal, among which, For reconstruction error, This is a preset threshold.

8. The multimodal sporadic anomaly environment generalization perception method for data closure as described in claim 1, characterized in that, The adaptive adjustment mechanism described in step S2 specifically includes: Sensor data quality adjustment: The impact of abnormal data can be mitigated by adjusting the scanning frequency, sensitivity, or filtering parameters of the sensing algorithm of abnormal sensors. Fusion weight adjustment: Based on the reconstruction error of each mode, reduce the fusion weight of abnormal modes and increase the fusion weight of non-abnormal modes; Model parameter update: Based on the incremental learning mechanism, the parameters of the perception model are updated using abnormal data that has been verified by labeled or pseudo-labeled data.

9. A multimodal, sporadic anomaly environment generalization perception method for data closure as described in claim 1, characterized in that, In step S4, the environment-related constraint loss is the domain adversarial loss. The calculation formula is as follows: in, For domain confrontation losses, For expectation operator, For multimodal fusion features, For domain tags, For domain classifiers, The probability of the environment source is output by the domain classifier; the goal of the perceptual model is to minimize the task loss while maximizing the domain adversarial loss, so that the domain classifier cannot distinguish the fused features. The environmental source.

10. A multimodal, sporadic anomaly environment generalization perception system oriented towards data closure, characterized in that, include: The data acquisition module is used to collect environmental data in real time using multiple sensors, and to perform time consistency alignment and preprocessing on the environmental data; The feature extraction module is used to encode and process the environmental data to obtain high-dimensional feature representations after encoding each modality; An anomaly detection module is used to identify occasional anomalies in the encoded high-dimensional feature representation. When an anomaly is detected, it enters the data closed-loop feedback stage and adjusts the anomaly sensor parameters, fusion weights, or model parameters through an adaptive adjustment mechanism to handle the occasional anomalies. The anomaly detection module reconstructs the high-dimensional feature representation of each modality and determines the anomaly state based on the reconstruction differences. The multimodal fusion module is used to perform weighted fusion of the high-dimensional feature representations of each modality according to the fusion weights corresponding to each modality, so as to obtain the fused features; The generalization learning module is used to receive data from different environments and train or update the perception model based on the environment generalization learning algorithm to optimize the perception algorithm.

Images