Heterogeneous data feature extraction and fusion method and system for intelligent automobile environment perception sensor, storage medium and program

Through data preprocessing and GRU-Transformer hybrid model, the problem of data heterogeneity of different sensors in intelligent connected vehicles is solved, more accurate environmental perception is achieved, and the environmental perception accuracy and efficiency of intelligent connected vehicles are improved, making it automatic Driving technology provides new support.

CN118887633APending Publication Date: 2024-11-01CHANGSHU INSTITUTE OF TECHNOLOGY
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410791252.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

There is obvious heterogeneity in data acquired by different sensors (visual cameras, lidars and millimeter wave radars) in intelligent connected vehicles, resulting in the effective extraction and fusion of data becoming a technical problem.

Method used

Using a method of extracting and fusion of heterogeneous data features of intelligent automotive environment perception sensors, the features of different sensors are extracted and fused through data preprocessing and a hybrid model based on GRU and Transformer, including unified time reference, consistent coordinate system, data cleaning, and features. Vector extraction and sequence window division, using the GRU layer to capture timing dependencies, and the Transformer layer to capture global dependencies, generating heterogeneous fusion spatiotemporal interaction features.

Benefits of technology

It realizes more accurate perception of the surrounding environment of intelligent connected vehicles, improves the accuracy and efficiency of environmental perception, and provides new means for innovation in autonomous driving technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887633A_ABST
    Figure CN118887633A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent automobile environment perception sensor heterogeneous data feature extraction and fusion method. The method comprises the following steps: collecting original data acquired by a camera, a laser radar and a millimeter wave radar of an intelligent network connection automobile under various driving conditions; the method comprises the following steps: performing data preprocessing on original data, during feature vector extraction, extracting geometric features and density features from point cloud data acquired by a laser radar and a millimeter wave radar in the original data, performing dimension alignment by using principal component analysis after feature vectors of the original data are extracted, and performing end-to-end splicing; segmenting the spliced feature vector according to a fixed time length; and inputting the segmented feature vector sequence into the trained hybrid model based on the GRU and the Transform to obtain the space-time interaction feature of heterogeneous fusion. According to the method, a multi-modal feature extraction mode is adopted, and environment information can be reflected more comprehensively by combining GRU and Transform, so that the sensing precision is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of traffic control, and in particular to a method and system for extracting and fusing heterogeneous data features from intelligent vehicle environment perception sensors. Background Art

[0002] In the road test data of intelligent connected vehicles, the raw data from visual cameras, lidars and millimeter-wave radars have obvious heterogeneity. For example, visual cameras are used to capture and obtain images in the environment, lidars collect point cloud data containing coordinate information in three-dimensional space and echo intensity, and millimeter-wave radars collect data mainly containing point clouds containing coordinate information in space, RCS and Doppler. Therefore, the premise for unified processing of all data is to solve the heterogeneity problem. In this way, effective features that can characterize environmental information can be extracted from the raw data of different sensors. For example, points, lines, surfaces and semantic information are extracted from images, the position, shape and other information of surrounding objects are extracted from laser point clouds, and the position, speed and angle information of surrounding objects are extracted from millimeter-wave radar point clouds. Due to the heterogeneity of different data, how to effectively extract and fuse these features has become a technical problem. Summary of the invention

[0003] In view of the above-mentioned defects of the prior art, the present invention provides a method for extracting and fusing features of heterogeneous data of intelligent automobile environment perception sensors to solve the problem of extracting and fusing heterogeneous data.

[0004] The technical solution of the present invention is as follows: A method for extracting and fusing heterogeneous data features of an intelligent automobile environment perception sensor comprises the following steps:

[0005] Collect raw data from cameras, lidar, and millimeter-wave radars of connected vehicles under various driving conditions;

[0006] The raw data is preprocessed, and the data preprocessing includes time base unification, coordinate system one, data cleaning, feature vector extraction and sequence window division in sequence; when extracting the feature vector, geometric features and density features are extracted from the point cloud data acquired by the laser radar and millimeter wave radar in the raw data, and after extracting the feature vector of the raw data, principal component analysis is used to align the dimensions, and multiple feature vectors are spliced ​​in the order of head to tail connection, and the sequence window division segments the spliced ​​feature vectors according to a fixed time length;

[0007] The segmented feature vector sequence is input into the trained hybrid model based on GRU and Transformer to obtain heterogeneously fused spatiotemporal interaction features. The hybrid model based on GRU and Transformer includes a GRU layer and a Transformer layer. The GRU layer converts the segmented feature vector sequence into a high-dimensional feature representation and passes it to the Transformer layer to output the heterogeneously fused spatiotemporal interaction features.

[0008] Another technical solution of the present invention is a device for extracting and fusing heterogeneous data features of an intelligent automobile environment perception sensor, comprising:

[0009] Acquisition module: collects raw data obtained by cameras, lidar and millimeter-wave radar of intelligent connected vehicles under various driving conditions;

[0010] Data preprocessing module: preprocessing the original data, the data preprocessing includes time base unification, coordinate system one, data cleaning, feature vector extraction and sequence window division in sequence; when extracting the feature vector, extract geometric features and density features from the point cloud data acquired by the laser radar and millimeter wave radar in the original data, after extracting the feature vector of the original data, use principal component analysis to align the dimensions, and splice multiple feature vectors in the order of head to tail connection, and the sequence window division segments the spliced ​​feature vectors according to a fixed time length; and,

[0011] Model output module: The segmented feature vector sequence is input into the trained hybrid model based on GRU and Transformer to obtain heterogeneous fusion spatiotemporal interaction features. The hybrid model based on GRU and Transformer includes a GRU layer and a Transformer layer. The GRU layer converts the segmented feature vector sequence into a high-dimensional feature representation and passes it to the Transformer layer to output the heterogeneous fusion spatiotemporal interaction features.

[0012] Furthermore, when collecting raw data, a mixture of simulated data from software-simulated road tests of intelligent connected vehicles and real data from road tests of intelligent connected vehicles in a formal environment is used.

[0013] Furthermore, when extracting the feature vector, the ResNet model with the fully connected layer removed is used to output the feature vector of the image data acquired by the camera in the original data, and the point cloud data acquired by the lidar and millimeter wave radar is extracted with PCL to extract geometric features and density features.

[0014] Furthermore, when training the hybrid model based on GRU and Transformer, the mean square error is used as the loss function, and the stochastic gradient descent optimizer is used for training using L1 regularization and dropout strategy. Prevent overfitting and improve the generalization ability of the model

[0015] The present invention also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method for extracting and fusing heterogeneous data features of the intelligent automobile environment perception sensor is implemented.

[0016] A program product is also provided, which, when executed by a processor, implements the above-mentioned method for extracting and fusing heterogeneous data features of intelligent automobile environment perception sensors.

[0017] Compared with the prior art, the advantages of the technical solution provided by the present invention are:

[0018] The present invention extracts key features such as edges, corners and textures of image data, as well as geometric features and density features of laser point clouds and millimeter-wave radar point clouds, and by combining the advantages of GRU and Transformer models, can simultaneously capture temporal dependencies and global dependencies, and more accurately perceive and understand the environment around the intelligent networked vehicle. The multimodal feature extraction method can more comprehensively reflect environmental information, thereby improving the accuracy and efficiency of environmental perception of intelligent networked vehicles, providing a new technical means for environmental perception of intelligent networked vehicles, and helping to promote the innovation and development of autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the module for the feature extraction and fusion method of heterogeneous data from intelligent vehicle environment perception sensors. DETAILED DESCRIPTION

[0020] The present invention is further described below in conjunction with examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading this description, various equivalent modifications to this description by those skilled in the art fall within the scope defined by the claims attached to this application.

[0021] The present embodiment relates to a device for extracting and fusing heterogeneous data features from an intelligent automobile environment perception sensor, comprising:

[0022] Acquisition module: collects raw data obtained by cameras, lidar and millimeter-wave radar of intelligent connected vehicles under various driving conditions;

[0023] Data preprocessing module: preprocessing the original data, the data preprocessing includes time base unification, coordinate system one, data cleaning, feature vector extraction and sequence window division in sequence; when extracting the feature vector, extract geometric features and density features from the point cloud data acquired by the laser radar and millimeter wave radar in the original data, after extracting the feature vector of the original data, use principal component analysis to align the dimensions, and splice multiple feature vectors in the order of head to tail connection, and the sequence window division segments the spliced ​​feature vectors according to a fixed time length; and,

[0024] Model output module: The segmented feature vector sequence is input into the trained hybrid model based on GRU and Transformer to obtain heterogeneous fusion spatiotemporal interaction features. The hybrid model based on GRU and Transformer includes a GRU layer and a Transformer layer. The GRU layer converts the segmented feature vector sequence into a high-dimensional feature representation and passes it to the Transformer layer to output the heterogeneous fusion spatiotemporal interaction features.

[0025] Please combine Figure 1 As shown, the heterogeneous data feature extraction and fusion method performed by the heterogeneous data feature extraction and fusion device of the intelligent automobile environment perception sensor includes the following steps:

[0026] 1. Data Collection

[0027] The collected data include two types: simulation data and intelligent connected vehicle road test data.

[0028] Simulation data acquisition uses CARLA software to simulate the operation of intelligent connected vehicle road tests under various driving conditions and collect raw data obtained by cameras, lidar and millimeter-wave radar.

[0029] The intelligent connected vehicle road test data collection is carried out in a real environment, where the intelligent connected vehicle collects raw data obtained by cameras, lidars and millimeter-wave radars under various driving conditions.

[0030] 2. Data preprocessing

[0031] Data preprocessing includes: time base unification, coordinate system one, data cleaning, feature vector extraction, sequence window division and data set division. The following is an explanation of each part of the processing.

[0032] 2.1 Unified time base: GNSS provides the base time, and the respective clock times are calibrated according to the base time provided by GNSS to achieve time synchronization.

[0033] 2.2 Coordinate system 1. Unify the camera, laser radar and millimeter wave radar into the vehicle body coordinate system (upper right front). Let there be a point P in the camera coordinate systemcamera , there is a point P in the laser radar coordinate system Lidar , there is a point P in the millimeter wave radar coordinate system Radar , can be converted to the vehicle coordinate system by the following formula:

[0034] P vehicle1 =R vc P camera +t vc (1)

[0035] P vehicle2 =R vl P Lidar +t vl (2)

[0036] P vehicle3 =R vr P Radar +t vr (3)

[0037] Where: P vehicle1 , P vehicle2 and P vehicle3 Point P camera , P Lidar and P Radar Coordinate in the vehicle body coordinate system; R vc , R vl and R vr are the rotation matrices from the camera, lidar and millimeter-wave radar coordinate systems to the vehicle body coordinate system; t vc ,t vl and t vr are the translation vectors from the origin of the camera, lidar and millimeter-wave radar coordinate systems to the origin of the vehicle coordinate system.

[0038] 2.3 Data cleaning. Check and delete duplicate recorded data to avoid bias during model training. For missing data, delete the missing images when they have little impact on the overall dataset or the image quality is severely damaged and cannot be restored. For missing laser point clouds or millimeter wave radar data, when the missing part is small and the data has temporal or spatial continuity, the nearest neighbor interpolation can be performed using adjacent data points.

[0039] 2.4 Feature vector extraction.

[0040] For image data, the original image data is adjusted to the input size required by the ResNet model and normalized. The output of the convolutional layer before the fully connected layer in the ResNet model is used as a feature because these layers contain key information such as the edges, corners, and textures of the image.

[0041] For laser point cloud and millimeter wave radar point cloud data, PCL is used to extract the geometric features and density features of the point cloud, which include the normal vector, curvature, and point density of the point cloud.

[0042] After extracting the feature vectors of the image and point cloud data, principal component analysis (PCA) is used to align the dimensions and concatenate multiple feature vectors in the order of head to tail connection. PCA dimensionality reduction: PCA is performed on the extracted feature vectors to reduce the dimension of the features and ensure that the features of different data types are consistent in dimension. PCA removes redundant information between features and retains the most important principal components. Feature splicing: The image feature vectors, laser point cloud feature vectors, and millimeter wave radar feature vectors processed by PCA are concatenated in the order of head to tail connection. A comprehensive feature vector that integrates information from multiple sensors is obtained. Save the spliced ​​features: Save the spliced ​​feature vectors to a data structure for subsequent use in model training and testing.

[0043] 2.5 Sequence Window Division: The concatenated continuous feature vectors are divided into sequence windows of fixed length (set to 30 seconds) to ensure that the data in each window is continuous in time and contains data in the complete time period.

[0044] 2.6 Dataset division. All sequences are randomly shuffled to ensure data diversity and randomness and avoid any potential order bias. Divide into training set, validation set and test set. 70% of the data volume is divided into training set for training models; 20% of the data volume is divided into validation set for adjusting model parameters and hyperparameters; the remaining 10% is used as test set for evaluating model performance. Store the divided training set, validation set and test set separately.

[0045] 3. Build a hybrid model based on GRU and Transformer.

[0046] The present invention designs a GRU-Transformer hybrid model structure, which aims to optimize data processing in intelligent automobile environmental perception. In this structure, the data is first processed by the GRU layer, and its gating mechanism is used to effectively capture the dynamic changes and temporal dependencies in the time series data. The design advantage of the GRU layer lies in its small number of parameters and low computational complexity, which enables it to serve as the front end of the model, reducing the computational burden of the subsequent Transformer layer, thereby significantly improving the operating efficiency of the overall model. The GRU layer not only serves as a front-end processing unit for time series analysis, but also plays the role of a feature extractor, converting the original sequence data into a high-dimensional feature representation containing key dynamic information. These high-dimensional features are then passed to the Transformer layer, which captures the global dependencies in the data through its self-attention mechanism, further enriching the expressive power of the features.

[0047] Based on the above design, the model can more effectively process heterogeneous data from different sensors, extract richer and more accurate environmental perception features, and provide strong support for the decision-making of smart cars. In addition, this structural design of the model also allows in-depth analysis of complex environmental perception tasks while maintaining efficient computing.

[0048] In the sequence feature extraction stage, the model inputs the sequence data in the training set into the GRU network for encoding. The GRU network can capture the temporal dependencies in the sequence data through its unique gating mechanism, thereby effectively extracting key features that reflect temporal dynamics.

[0049] In the interactive feature extraction stage, the model further uses the two-layer multi-head attention mechanism of the Transformer encoder to process the encoding vector output by the GRU. This mechanism can weight the information in the encoding vector and strengthen the spatial correlation between sequences, thereby extracting spatial interactive features. Subsequently, the model analyzes the changes of these spatial interactive features over time, captures their temporal dependence, and realizes the comprehensive extraction of spatiotemporal interactive features.

[0050] Through the feature extraction of these two stages, the model can comprehensively consider the information of time series and spatial interaction and generate a set of interactive features containing rich spatiotemporal dynamics. This not only enhances the model's sensitivity to environmental changes, but also provides a more accurate feature representation for the environmental perception of smart cars.

[0051] In the process of building the GRU network, this embodiment first conducts a detailed analysis of the data sequence to determine the optimal configuration of the network structure. Considering the long-term dependencies and complex dynamic characteristics that the data sequence may contain, a GRU network with 64 hidden units is designed. The number of units is selected based on experimental results to ensure that the network can capture key information in the time series while avoiding overfitting.

[0052] In the time series feature extraction stage, the preprocessed time series data is first standardized to eliminate the impact of dimension and accelerate the convergence of the model. After the data is input into the GRU network, the network gradually encodes the input sequence through its hidden units, and uses the gating mechanism to dynamically and selectively retain or forget information, thereby extracting features that reflect the intrinsic structure of the time series. These features include not only the local patterns of the sequence, but also the global trends of the sequence.

[0053] After encoding, the output of the GRU network is a compressed feature vector that contains the time series features of the input sequence. These features are then used as input for further analysis or model training, providing a rich source of information for the environment perception of smart cars.

[0054] In the Transformer layer part, the application of the Transformer encoder is intended to further enhance the model's ability to process data. Based on the output of the GRU model, a Transformer encoder with a two-layer multi-head attention mechanism is used. This encoder processes information in parallel through multiple attention heads, and each head focuses on different aspects of the encoding vector, thereby capturing the complex spatial interaction features between sequences. During the encoding process, each attention head independently weights the input encoding vector, and considers the association between different parts within the sequence through the self-attention mechanism. Subsequently, the encoder layers further process information through residual connections and layer normalization, which helps to avoid the gradient vanishing problem in deep network training. By capturing temporal and spatial information, spatiotemporal interaction features that integrate spatial and temporal information are obtained.

[0055] The decoding module is designed to transform these spatiotemporal interaction features into a final output vector sequence. The Transformer decoder uses a multi-head attention mechanism similar to the encoder to ensure that the decoding process can simultaneously consider the information of the previous output and the current processing features. The output of the decoder is appropriately transformed and mapped to generate the final vector sequence, which can be directly used for the environmental perception task of smart cars.

[0056] For the training and optimization of the hybrid model based on GRU and Transformer, the mean square error (MSE) is selected as the loss function to measure the difference between the model prediction and the actual observation. The model is trained using the stochastic gradient descent (SGD) optimizer, the optimizer Adam, the initial learning rate is set to 0.0005, and the weights of the model are adjusted to minimize the loss function. In addition, in order to prevent overfitting and improve the generalization ability of the model, L1 regularization and dropout strategies are used. During the training process, the performance of the model on the anomaly detection task is judged by calculating indicators such as accuracy, recall, and F1 score. If the performance is poor, the model's hyperparameters (learning rate, batch size, number of encoder layers) are adjusted to optimize the model.

[0057] It should be noted that the specific methods of the above embodiments can form a computer program product. Therefore, the computer program product implemented in this application can be stored on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.).

Claims

1. A method for extracting and fusing heterogeneous data features from intelligent vehicle environment perception sensors, characterized in that: The following steps are involved: Collect raw data from cameras, lidar, and millimeter-wave radars of connected vehicles under various driving conditions; The raw data is preprocessed, and the data preprocessing includes time base unification, coordinate system one, data cleaning, feature vector extraction and sequence window division in sequence; when extracting the feature vector, geometric features and density features are extracted from the point cloud data acquired by the laser radar and millimeter wave radar in the raw data, and after extracting the feature vector of the raw data, principal component analysis is used to align the dimensions, and multiple feature vectors are spliced ​​in the order of head to tail connection, and the sequence window division segments the spliced ​​feature vectors according to a fixed time length; The segmented feature vector sequence is input into the trained hybrid model based on GRU and Transformer to obtain heterogeneously fused spatiotemporal interaction features. The hybrid model based on GRU and Transformer includes a GRU layer and a Transformer layer. The GRU layer converts the segmented feature vector sequence into a high-dimensional feature representation and passes it to the Transformer layer to output the heterogeneously fused spatiotemporal interaction features.

2. The method for extracting and fusing heterogeneous data features of intelligent vehicle environment perception sensors according to claim 1 is characterized in that: When collecting raw data, a mixture of simulated data from software-simulated intelligent connected vehicle road tests and real data from intelligent connected vehicle road tests in formal environments is used.

3. The method for extracting and fusing heterogeneous data features of intelligent vehicle environment perception sensors according to claim 1 is characterized in that: When extracting the feature vector, for the image data acquired by the camera in the original data, the ResNet model with the fully connected layer removed is used to output the feature vector of the image data, and the point cloud data acquired by the lidar and millimeter wave radar is extracted with PCL for geometric features and density features.

4. The method for extracting and fusing heterogeneous data features of intelligent vehicle environment perception sensors according to claim 1 is characterized in that: When training the hybrid model based on GRU and Transformer, the mean square error is used as the loss function, and the stochastic gradient descent optimizer is used for training with L1 regularization and dropout strategy.

5. A device for extracting and fusing heterogeneous data features of intelligent automobile environment perception sensors, characterized in that: include: Acquisition module: collects raw data obtained by cameras, lidar and millimeter-wave radar of intelligent connected vehicles under various driving conditions; Data preprocessing module: preprocess the original data, which includes time base unification, coordinate system one, data cleaning, feature vector extraction and sequence window division in sequence; when extracting the feature vector, extract geometric features and density features from the point cloud data acquired by the laser radar and millimeter wave radar in the original data, and after extracting the feature vector of the original data, use principal component analysis to align the dimensions, and splice multiple feature vectors in the order of connecting head to tail, and the sequence window division segments the spliced ​​feature vectors according to a fixed time length; as well as, Model output module: The segmented feature vector sequence is input into the trained hybrid model based on GRU and Transformer to obtain heterogeneous fusion spatiotemporal interaction features. The hybrid model based on GRU and Transformer includes a GRU layer and a Transformer layer. The GRU layer converts the segmented feature vector sequence into a high-dimensional feature representation and passes it to the Transformer layer to output the heterogeneous fusion spatiotemporal interaction features.

6. The device for extracting and fusing heterogeneous data features of intelligent vehicle environment perception sensors according to claim 1 is characterized in that: When collecting raw data, a mixture of simulated data from software-simulated intelligent connected vehicle road tests and real data from intelligent connected vehicle road tests in formal environments is used.

7. The device for extracting and fusing heterogeneous data features of intelligent vehicle environment perception sensors according to claim 1 is characterized in that: When extracting the feature vector, for the image data acquired by the camera in the original data, the ResNet model with the fully connected layer removed is used to output the feature vector of the image data, and the point cloud data acquired by the lidar and millimeter wave radar is extracted with PCL for geometric features and density features.

8. The device for extracting and fusing heterogeneous data features of intelligent vehicle environment perception sensors according to claim 1 is characterized in that: When training the hybrid model based on GRU and Transformer, the mean square error is used as the loss function, and the stochastic gradient descent optimizer is used for training with L1 regularization and dropout strategy.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the method for extracting and fusing heterogeneous data features of the intelligent automobile environment perception sensor as described in any one of claims 1 to 4 is implemented.

10. A program product, characterized in that When the program product is executed by a processor, the method for extracting and fusing heterogeneous data features of an intelligent automobile environment perception sensor as described in any one of claims 1 to 4 is implemented.

Citation Information

Cited By

  • Vehicle-mounted imaging road information reconstruction method based on deep learning and application thereof

    CN119339348A

  • End-to-end high-dynamic target recognition method based on self-attention mechanism

    CN119805394A