A fusion perception method for autonomous driving of new energy vehicles based on an improved multi-head attention mechanism

By introducing a multi-head attention mechanism and a GRU network, RGB images and LiDAR data are adaptively fused, solving the problems of unified representation of multimodal features and lack of contextual information in the fusion perception of autonomous driving in new energy vehicles, and achieving more accurate trajectory prediction and more efficient autonomous driving decision-making.

CN118627018BActive Publication Date: 2024-11-19GUANGDONG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411103623.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-19
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

Existing fusion perception methods for autonomous driving in new energy vehicles lack a unified representation of features across multiple modalities, cannot effectively integrate features from different modalities, lack reasoning on contextual information, cannot dynamically adjust the degree of attention to perception modalities, and fail to effectively capture long-term dependencies in historical trajectories, resulting in low perception accuracy and stability, and increased computational and storage requirements.

Method used

An autonomous driving fusion perception method based on an improved multi-head attention mechanism is adopted. This method adaptively fuses RGB images and LiDAR data through the multi-head attention mechanism, uses a GRU network to model the long-term dependencies of historical trajectories, and combines feature integration strategy and adaptive perception weight adjustment to achieve flexible adaptation to different driving scenarios.

Benefits of technology

It improves the decision-making and planning capabilities of autonomous driving systems, enhances the accuracy and stability of perception, reduces computing and storage requirements, and improves driving safety and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118627018B_ABST
    Figure CN118627018B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of vehicle intelligent perception technology, and in particular to a new energy vehicle automatic driving fusion perception method based on an improved multi-head attention mechanism, comprising: inputting RGB images and laser radar scanning data in a vehicle driving environment into an automatic driving fusion perception model to obtain a predicted trajectory result; the automatic driving fusion perception model is obtained by training using a training set; wherein the automatic driving fusion perception model comprises: an automatic driving fusion sense encoder based on a multi-head attention mechanism and a trajectory prediction network sub-model; using a ResNet unit of the automatic driving fusion sense encoder in combination with a multi-head attention mechanism unit of the automatic driving fusion sense encoder to perceive each layer of attention in feature data to perform feature extraction, and perform feature fusion to obtain an environmental context encoding vector, and input the environmental context encoding vector into a trajectory prediction network sub-model to perform trajectory prediction, and obtain a predicted trajectory result, thereby controlling the vehicle to perform automatic driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle intelligent perception technology, and in particular to a new energy vehicle automatic driving fusion perception method based on an improved multi-head attention mechanism. Background Art

[0002] With the rapid development of intelligent driving technology for new energy vehicles, integrating multiple sensors and solving various tasks in an end-to-end algorithm is a challenging but crucial topic for autonomous driving. Autonomous driving fusion perception combines multiple sensor data for joint learning representation to minimize uncertainty, thereby achieving more robust and accurate predictions. However, for actual driving tasks, the global context of the 3D scene is key. For example, changes in the state of a traffic light can affect the behavior of vehicles at a geometric distance from the traffic light. Therefore, geometric distance features alone may not be sufficient to effectively fuse representations in the driving model.

[0003] Therefore, designing an accurate, stable and computationally efficient fusion perception method for automatic driving of new energy vehicles to achieve effective trajectory prediction has important theoretical significance and practical application value. It enables the automatic driving system to better understand and adapt to complex traffic environments, promotes the development of vehicle intelligent perception systems, and promotes the implementation of intelligent driving related technologies.

[0004] The Chinese invention patent with application number CN202110880427.5 proposes a robustness test method and system for an autonomous driving multi-sensor fusion perception model, including the following steps: simultaneously collect data from multiple on-board sensors and perform corresponding preprocessing; use the preprocessed data from multiple on-board sensors to train a multi-sensor fusion perception model; select a sensor to be attacked from multiple on-board sensors; generate a mask to add anti-disturbance noise to the data collected by the selected sensor to be attacked, thereby generating fraudulent data; input the fraudulent data and the data normally collected by other sensors into the trained multi-sensor fusion perception model, and calculate the fraudulent data. The loss function generated by the fraud data is calculated and back-propagated; the fraud data is iteratively updated in combination with the gradient of the loss function to obtain the final fraud data; the final fraud data and the data normally collected by other sensors are input into the multi-sensor fusion perception model to obtain the output accuracy of the multi-sensor fusion perception model, and the robustness test result of the multi-sensor fusion perception model is obtained according to the output accuracy; the fraud data and the data normally collected by other sensors are input into the trained multi-sensor fusion perception model, the loss function generated by the fraud data is calculated, and back-propagation is performed; the fraud data is iteratively updated in combination with the gradient of the loss function to obtain the final fraud data.

[0005] The Chinese invention patent with application number CN202311080809.5 proposes a vehicle-road-cloud fusion perception method for an autonomous driving vehicle, including the following steps: the vehicle obtains the image and point cloud information in front of the vehicle through a camera and a lidar, and the road-side equipment of the road section where the vehicle is located obtains the road-side image and point cloud information through a camera and a lidar; and respectively completes the corresponding image-based target detection and point cloud-based target detection; based on multi-sensor fusion, the image targets and point cloud targets in front of the vehicle and on the road side are fused respectively to obtain the vehicle-side fusion features and the road-side fusion features; the road-side fusion features are sent to the vehicle, and the vehicle aligns the vehicle-side fusion features and the road-side fusion features in time and space to achieve synchronization; for objects that can be perceived by both the vehicle side and the road side, the perception results of both ends of the vehicle and the road are fused; for the blind spots of the vehicle-side field of view, the road-side detection results are used as a supplement; the cloud provides global traffic information, high-precision maps and weather information, which are post-fused with the vehicle-road fusion features.

[0006] The Chinese invention patent with application number CN202310153288.5 proposes a vehicle-road collaborative information fusion method and system based on transfer learning, which can fully reflect the basic concept of vehicle-road collaboration, and realize multi-dimensional and multi-angle recognition and observation of driving environment conditions under complex road conditions based on the effective combination of environmental observation information and roadbed equipment transfer learning. It can effectively improve the utilization rate of learning models and significantly improve the accuracy and robustness of vehicle-road collaborative fusion.

[0007] However, the prior art has the following defects:

[0008] 1. Existing methods lack the ability to extend to multiple modalities or interactions between feature pairs, lack a unified representation method for multiple modal features, and lack attention to key information, resulting in low perception accuracy and stability.

[0009] 2. Existing sensor fusion methods lack reasoning about contextual information in high-density dynamic agents and complex scenarios, introduce additional calculations, increase computing and storage requirements, and greatly affect the efficiency and performance of the prediction network.

[0010] 3. The existing new energy vehicle autonomous driving fusion perception system cannot dynamically change the degree of attention to different perception modalities, and thus cannot adapt well to different driving scenarios.

[0011] 4. Existing methods do not consider modeling long-term dependencies in historical trajectories and cannot capture the global information of the path well. Summary of the invention

[0012] The purpose of the present invention is to provide a new energy vehicle automatic driving fusion perception method based on an improved multi-head attention mechanism to solve the problems existing in the above-mentioned prior art, which can improve the decision-making and planning capabilities of the automatic driving system, enable it to more accurately predict the vehicle's driving path, and improve driving safety and efficiency.

[0013] To achieve the above object, the present invention provides the following solutions:

[0014] A fusion perception method for automatic driving of new energy vehicles based on an improved multi-head attention mechanism, comprising:

[0015] Obtain RGB images and LiDAR scanning data in the vehicle driving environment;

[0016] Inputting the RGB image and the laser radar scanning data in the vehicle driving environment into the autonomous driving fusion perception model to obtain the predicted trajectory result; the autonomous driving fusion perception model is obtained by training with the training set; wherein the autonomous driving fusion perception model includes: an autonomous driving fusion sense encoder based on a multi-head attention mechanism and a trajectory prediction network sub-model;

[0017] The ResNet unit of the autonomous driving fusion sense encoder is combined with the multi-head attention mechanism unit of the autonomous driving fusion sense encoder to perceive the attention of each layer in the feature data to perform feature extraction, and perform feature fusion to obtain an environmental context encoding vector, and the environmental context encoding vector is input into the trajectory prediction network sub-model to perform trajectory prediction, and obtain the predicted trajectory result;

[0018] According to the predicted trajectory results, the steering, throttle and braking values ​​of the vehicle are obtained, so as to control the vehicle to perform automatic driving.

[0019] Optionally, using the ResNet unit of the autonomous driving fusion sense encoder in combination with the multi-head attention mechanism unit of the autonomous driving fusion sense encoder to perceive each layer of attention in the feature data for feature extraction includes:

[0020] Processing the RGB image and the laser radar scanning data in the vehicle driving environment to generate a dual-channel pseudo image;

[0021] The multi-head attention mechanism unit is set in front of the convolution layer in each of the ResNet units, a set of queries, keys and values ​​are calculated using linear projection according to the multi-head attention mechanism unit, attention weights are calculated based on the scaled dot product between the queries and the keys, and the values ​​of each query are summarized, and the summarized queries, keys and values ​​are calculated separately Secondary attention, and concatenate all the attention to obtain the attention of each layer;

[0022] The dual-channel pseudo image is input into the ResNet unit and feature extraction is performed in combination with the attention of the corresponding layer to obtain a feature map.

[0023] Optionally, a method for computing a set of queries, keys, and values ​​using linear projection according to the multi-head attention mechanism unit is:

[0024]

[0025] in, is the query matrix, the input matrix and the query weight matrix Multiply them together to get; is the key matrix, which is composed of the input matrix With key weight matrix Multiply them together to get; is a value matrix, which is composed of the input matrix With value weight matrix Multiply them together to get;

[0026] The attention weights are calculated based on the scaled dot product between the query and the key, and the value of each query is summarized as:

[0027]

[0028] in, , , is the weight matrix of the linear projection, , and are the query, key, and value dimensions of each attention head, respectively. Optionally, obtaining the attention of each layer includes:

[0029] The method to obtain the first layer of attention is:

[0030]

[0031]

[0032] in, is a linear transformation, represents the output of the first multi-head attention mechanism, Indicates that the query matrix , key matrix and the numerical matrix Input into the multi-head attention mechanism for processing, It means connecting the outputs of multiple attention heads together.

[0033] Optionally, obtaining the environment context encoding vector includes:

[0034] Embedding trainable parameters in the learnable position of the autonomous driving fusion sense encoder, projecting the current vehicle speed as a scalar value to a C-dimensional vector using a linear layer, obtaining a feature map, summing the elements of the existing feature map obtained by extracting the feature map by the feature extraction unit, and feeding the summation result back to the modal branch in the multi-head attention mechanism unit;

[0035] Extracting a feature map of the first target dimension from each modal branch, reducing the dimension of the feature map of the first target dimension by average pooling, and flattening the feature map to obtain a feature vector, namely, a feature vector of the RGB image and a feature vector of the LiDAR scan;

[0036] The feature vector of the RGB image and the feature vector of the laser radar scan are combined to obtain the environmental context encoding vector.

[0037] Optionally, obtaining the summation result includes:

[0038] The high-resolution feature map in the autonomous driving fusion sense encoder is downsampled to a fixed resolution using average pooling, the fixed resolution is passed to the attention mechanism unit, and the upsampled original resolution is output using bilinear interpolation, and the feature map is summed with the elements of the existing feature map using the original resolution to obtain the summed result.

[0039] Optionally, obtaining the predicted trajectory result includes:

[0040] The environmental context encoding vector is input into the multi-layer perceptron unit in the trajectory prediction network sub-model for dimensionality reduction, and the hidden state of the gated recurrent unit in the trajectory prediction network sub-model is initialized using the reduced-dimensional environmental context encoding vector. The encoding is performed in the hidden state based on the update gate control of the gated recurrent unit, and the information stream of the next time step is output. The predicted trajectory result is obtained in combination with the current position and target position of the vehicle.

[0041] Optionally, the method for obtaining the predicted trajectory result is:

[0042]

[0043]

[0044]

[0045]

[0046] in, Represents the input information at the current moment, Represents the hidden state of the previous moment. The hidden state acts as the memory of the neural network and contains information about the data that the node has seen before. Represents the hidden state passed to the next moment, represents the candidate hidden state, Represents the reset gate, represents the update gate, represents the sigmoid function, Represents the tanh function.

[0047] Optionally, obtaining the autonomous driving fusion perception model includes:

[0048] Input the perceptual modality into the ResNet unit, obtain the features of each perceptual modality, input the features of each perceptual modality into the multi-head attention mechanism unit for adaptive fusion, and calculate the attention weight of each perceptual modality feature;

[0049] The gradient of the loss function to the model parameters is calculated according to the back-propagation algorithm, and the gradient of the model parameters is updated using the loss function according to the Adam optimization algorithm:

[0050]

[0051] in, Indicates the parameters to be updated. is the learning rate, Represents the loss function Parameters The gradient of

[0052] When the loss cannot be reduced after several consecutive iterations, the training process is terminated to obtain the autonomous driving fusion perception model.

[0053] The beneficial effects of the present invention are:

[0054] Traditional perceptual fusion methods are usually based on weighted averaging or feature concatenation, which lacks adaptability to different perceptual modalities. To solve this problem, a multi-head attention mechanism is introduced, which allows the fusion of perceptual data to adaptively focus on the importance of different modalities. In the multi-head attention mechanism, the concept of attention head is introduced, and each attention head can independently focus on different parts of the perceptual data. Each attention head learns weight values. To determine the importance of the corresponding modality, the adaptive fusion of perceptual data can be achieved. The number and weights of attention heads can be determined through training to maximize the fusion effect.

[0055] In order to represent the fused perceptual features more compactly, this method proposes a feature integration strategy to reduce the dimension of the feature map into an environmental context encoding vector . The fused feature map is converted into a compact vector representation using a dimensionality reduction operation. The environmental context encoding vector contains global context information, such as road structure, traffic lights, etc., providing a more effective environmental representation for the autonomous driving system. The compact encoding vector has a lower dimension, which reduces computational and storage overhead and can be transmitted and processed faster.

[0056] Considering that different scenarios have different requirements and importance for perception modalities, this method introduces an adaptive perception weight adjustment mechanism. The perception weight is calculated by learning a parameterized function. The perception weight is automatically adjusted according to the input scene information and importance index. Adaptive adjustment of perception weight can flexibly adapt to the perception needs in different scenarios according to the characteristics of the current driving scene, thereby improving the performance and adaptability of the autonomous driving system.

[0057] In order to better predict the future driving path of the vehicle, a trajectory prediction network based on GRU is proposed. By modeling the long-term dependencies in the historical trajectory through GRU, the global information of the path can be better captured. The gating mechanism of GRU can effectively process the historical sequence data of new energy vehicles. The GRU network can capture the motion patterns and rules in the historical trajectory, and provide more accurate path prediction, providing a more reliable basis for the decision-making and planning of the autonomous driving system.

[0058] The multi-head attention mechanism calculates the weight of each perception modality, and weightedly fuses the features of different modalities according to the importance and reliability of the current environment. This attention mechanism can improve the utilization efficiency of perception data, enhance the focus on key information, and thus improve the accuracy and stability of perception.

[0059] The feature map is reduced to a vector by averaging the values ​​on each channel of the feature map. This method can retain important information in the perceptual feature map and reduce the dimension of the vector. Other environment-related features are introduced into the reduced encoding vector, such as the vehicle's speed, direction, road type, etc. Integrating global context information can better reflect the driving scene in which the vehicle is located and provide a richer and more accurate environmental representation. This compact representation can reduce the requirements for computation and storage and improve the efficiency and performance of the trajectory prediction network.

[0060] By using an adaptive perception weight adjustment method, the new energy vehicle autonomous driving system can dynamically change the degree of attention it pays to different perception modalities under different environments and task requirements, thereby better adapting to different driving scenarios.

[0061] The GRU-based trajectory prediction network can use historical trajectory data to capture the temporal pattern of the trajectory and predict the future path. This time series modeling and prediction capability can improve the accuracy and robustness of trajectory prediction. Combined with global context information, it generates a predicted path and trajectory for the vehicle. This comprehensive prediction method can improve the decision-making and planning capabilities of the autonomous driving system, enabling it to more accurately predict the vehicle's driving path and improve driving safety and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0063] Figure 1 This is a flow chart of a fusion perception method for automatic driving of a new energy vehicle based on an improved multi-head attention mechanism according to an embodiment of the present invention;

[0064] Figure 2 This is a general architecture diagram of an autonomous driving fusion perception method based on a multi-head attention mechanism according to an embodiment of the present invention;

[0065] Figure 3 A detailed structural diagram of the multi-head attention mechanism according to an embodiment of the present invention;

[0066] Figure 4 Detailed structure of the GRU according to the embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] like Figure 1As shown, this embodiment discloses a new energy vehicle automatic driving fusion perception method based on an improved multi-head attention mechanism, including: obtaining RGB images and laser radar scanning data in the vehicle driving environment; inputting the RGB images and laser radar scanning data in the vehicle driving environment into the automatic driving fusion perception model to obtain the predicted trajectory result; the automatic driving fusion perception model is obtained by training using a training set; wherein the automatic driving fusion perception model includes: an automatic driving fusion sense encoder based on a multi-head attention mechanism and a trajectory prediction network sub-model; using the ResNet unit of the automatic driving fusion sense encoder combined with the multi-head attention mechanism unit of the automatic driving fusion sense encoder to perceive each layer of attention in the feature data for feature extraction, and perform feature fusion to obtain an environmental context encoding vector, and input the environmental context encoding vector into the trajectory prediction network sub-model for trajectory prediction to obtain a predicted trajectory result; according to the predicted trajectory result, the steering, throttle and braking values ​​of the vehicle are obtained, thereby controlling the vehicle to perform automatic driving.

[0070] The ResNet unit of the autonomous driving fusion sense encoder is combined with the multi-head attention mechanism unit of the autonomous driving fusion sense encoder to perceive the attention of each layer in the feature data for feature extraction, including: processing the RGB image and lidar scanning data in the vehicle driving environment to generate a dual-channel pseudo image; setting a multi-head attention mechanism unit in front of the convolution layer in each ResNet unit, using linear projection according to the multi-head attention mechanism unit to calculate a set of queries, keys and values, calculating the attention weight based on the scaled dot product between the query and the key, and summarizing the value of each query, and calculating the summarized query, key and value separately Attention is performed again, and all attentions are concatenated to obtain the attention of each layer; the dual-channel pseudo image is input into the ResNet unit and combined with the attention of the corresponding layer for feature extraction to obtain the feature map, specifically:

[0071] S1. Data collection and preprocessing

[0072] S101. Raw data acquisition and preprocessing: Equip sensors and collect raw driving data (RGB images taken by the camera in front of the vehicle, lidar scan data, radar scan data and object bounding boxes) in a real vehicle driving environment to ensure the effectiveness and reliability of the subsequent modeling process. The collected data sets include: 1.4M RGB camera images, 390k lidar scans, 1.4M radar scans and 1.4M object bounding boxes. In order to extract concise and informative feature representations, reduce dimensions and suppress noise, suitable for new energy vehicle autonomous driving tasks, data preprocessing is required before the data set is input into the network, including:

[0073] The lidar point cloud was converted to a histogram on a 2D bird’s eye view (BEV) grid of fixed resolution. Points within 32 m in front of the NEV and within 16 m on both sides were considered, thus comprising a 32m×32m BE grid. The grid was divided into 0.125m×0.125m blocks, resulting in a resolution of 256×256 pixels. For the histogram, the height dimension was discretized into 2 bins, representing points above / below and on the ground plane, respectively. A two-channel pseudo image of size 256×256 pixels was produced.

[0074] For RGB image input, a front-facing camera with a 100° field of view is considered. The front image is extracted with a resolution of 400×300 pixels and cropped to 256×256 to eliminate radial distortion at the edges.

[0075] In this way, it is helpful to further analyze and process the point cloud data, obtain a wider field of view, improve computing efficiency, and eliminate image distortion caused by radial distortion. The processed lidar data is recorded as , the processed RGB image is recorded as . This provides more accurate and reliable input for subsequent image processing and computer vision tasks.

[0076] The weights of each modality in the feature data perceived by the multi-head attention mechanism unit of the autonomous driving fusion sense encoder include: setting a multi-head attention mechanism unit in front of the convolutional layer in each ResNet unit, and using linear projection to calculate a set of queries, keys and values ​​according to the multi-head attention mechanism unit.

[0077] After perceiving the weights of each modality in the feature data through the multi-head attention mechanism unit of the autonomous driving fusion encoder, the following steps are included: Attention is performed again, and all attentions are concatenated to obtain the first layer of attention, specifically:

[0078] S2. Design of fusion perception encoder for autonomous driving based on multi-head attention mechanism

[0079] Considering that the imitation learning strategy based on existing sensor fusion methods performs poorly in high-density dynamic agents and complex scenes, which require global context reasoning. To this end, based on the multi-head attention mechanism and combined with convolutional neural network (CNN), this paper proposes a new multimodal fusion perception method that uses multi-head attention to integrate RGB image and lidar representation.

[0080] The specific design process of the autonomous driving fusion encoder based on the multi-head attention mechanism is as follows:

[0081] S201, using the single-view RGB image and the dual-channel pseudo image generated by the lidar bird's-eye view as the input of the network.

[0082] S202, using a residual neural network (ResNet) for feature extraction, ResNet is composed of multiple residual blocks, each residual block is composed of several convolutional layers and identity mapping. Specifically, ResNet includes 3 stages, each stage includes multiple residual blocks.

[0083] The first stage: contains a sequence of residual blocks, each residual block contains two 3×3 convolutional layers, and the number of channels gradually increases.

[0084] Second stage: Similar to the first stage, but a convolution operation with a stride of 2 is used in the first convolution layer of each residual block to reduce the size of the feature map.

[0085] The third stage: Similar to the first two stages, but the residual block has more channels.

[0086] S203. Use the multi-head attention mechanism to integrate image and lidar representations, as follows:

[0087] The Transformer architecture takes as input a sequence of discrete tokens, each represented by a feature vector. This feature vector is supplemented by a positional encoding to incorporate positional inductive biases. Formally, the input sequence is represented as , where N is the number of tokens in the sequence, and each token consists of dimensions The multi-head attention mechanism in Transformer uses linear projection to compute a set of queries, keys, and values ​​(Q, K, and V):

[0088] (1)

[0089] in, , , is the weight matrix of the linear projection, , and are the dimensions of query, key, and value for each attention head, respectively. It uses the scaled dot product between Q and K to compute the attention weights, and then sums up the values ​​for each query,

[0090] (2)

[0091] Finally, a nonlinear transformation is used to calculate the output features , which is related to the input feature Have the same shape.

[0092] (3)

[0093] For multi-head attention, Q, K, and V are calculated separately Finally, concatenate the attention outputs of all heads to get the first layer of attention representation :

[0094] (4)

[0095] (5)

[0096] in, is a linear transformation used to project the output of the multi-head attention back to the model dimension.

[0097] The intermediate feature maps of each modality are treated as a set rather than a spatial grid, and each element of the set is treated as a token. The convolutional feature extractors for the image and lidar BEV inputs encode different aspects of the scene at different layers. Therefore, these features are fused at multiple scales throughout the encoder ( Figure 2 ), specifically:

[0098] The multi-head attention module is used to integrate the feature representations of images and lidar;

[0099] Multiple scales: As the number of convolutional layers is different, that is, the scale is different, a multi-head attention mechanism is used on convolutions of different numbers of layers, that is, feature fusion is performed at multiple scales.

[0100] Obtaining the environment context encoding vector includes: embedding trainable parameters in a learnable position in the autonomous driving fusion sense encoder, using a linear layer to project the current vehicle speed as a scalar value to a C-dimensional vector, obtaining a feature map, and summing the elements of the existing feature map obtained by extracting the feature map by the feature extraction unit, and feeding the summation result back to the modal branch in the multi-head attention mechanism unit; extracting a feature map of the first target dimension from each modal branch, reducing the dimension of the feature map of the first target dimension by average pooling, and flattening it to obtain a feature vector, namely, a feature vector of the RGB image and a feature vector of the lidar scan; combining the feature vector of the RGB image and the feature vector of the lidar scan to obtain the environment context encoding vector.

[0101] Obtaining the summation result includes: using average pooling to downsample the high-resolution feature map in the autonomous driving fusion encoder to a fixed resolution, passing the fixed resolution to the attention mechanism unit, and using bilinear interpolation to output the upsampled original resolution, and using the original resolution to sum the feature map with the elements of the existing feature map to obtain the summation result, specifically:

[0102] Adding a learnable position embedding , which is a dimension The current vehicle speed is provided as input by projecting the scalar value into a C-dimensional vector using a linear layer. The input sequence, position embedding, and speed embedding are combined using element-wise summation to form a dimension Tensor of .

[0103] The output is then reshaped into S feature maps of dimension H×W×C and fed back into each individual modality branch using element-wise summation with the existing feature maps. The above mechanism constitutes feature fusion at a single scale. This fusion is applied multiple times in the ResNet feature extractors of the image and lidar BEV branches at different resolutions.

[0104] However, processing feature maps at high spatial resolution is computationally expensive. Therefore, the high-resolution feature maps in the early encoder are downsampled to a fixed resolution H=W=8 using average pooling, which is then passed as input to the attention mechanism and the output is upsampled to the original resolution using bilinear interpolation before element-wise summing with the existing feature maps.

[0105] S204, after obtaining the corresponding feature map, it is necessary to integrate the feature maps of different dimensions to obtain a unified representation of the environmental context encoding, as follows:

[0106] For an input with a resolution of 256×256 pixels, 8×8×512 feature maps are obtained from the feature extractor of each modality. These feature maps are reduced to 1×1×512 dimensions by average pooling and flattened into 512-dimensional feature vectors. The 512-dimensional feature vectors from the image and lidar BEV streams are then combined by element-wise summation. This 512-dimensional feature vector constitutes a compact representation of the environment that encodes the global context of the 3D scene. It is then fed to the trajectory prediction network.

[0107] Obtaining the predicted trajectory result includes: inputting the environmental context encoding vector into the multi-layer perceptron unit in the trajectory prediction network sub-model for dimensionality reduction, using the reduced environmental context encoding vector to initialize the hidden state of the gated recurrent unit in the trajectory prediction network sub-model, encoding in the hidden state based on the update gate control of the gated recurrent unit, outputting the information flow of the next time step, and combining the current position and target position of the vehicle to obtain the predicted trajectory result, specifically:

[0108] S3. Trajectory prediction network design

[0109] S301, send the integrated features to the multi-layer perceptron and gated recurrent unit (GRU) for trajectory prediction

[0110] To improve computational efficiency, the 512-dimensional feature vector is reduced to 64 dimensions by a multilayer perceptron with 2 hidden layers of 256 and 128 units, and then passed to an autoregressive trajectory network implemented using GRU. The hidden state of the GRU is initialized using the 64-dimensional feature vector. It also takes the current position and the target position as input, which allows the network to focus on the relevant context in the hidden state to predict the next trajectory.

[0111] (6)

[0112] (7)

[0113] (8)

[0114] (9)

[0115] in, Represents the input information at the current moment. Represents the hidden state at the previous moment. The hidden state acts as the neural network memory, which contains information about the data that the node has seen before. Represents the hidden state passed to the next moment. Represents a candidate hidden state. Reset Gate. Represents the update gate. Represents the sigmoid function, which can be used to convert data into values ​​in the range of 0-1. Represents the tanh function, through which the data can be transformed The value of the range.

[0116] Using the difference between the predicted trajectory and the ground truth trajectory Loss is used to train the network and register it to the current coordinate frame. is the true trajectory of time step t, then the loss function is:

[0117] (10)

[0118] S302: Performing automatic driving control based on the results obtained from trajectory prediction

[0119] Two proportional-integral-derivative (PID) controllers are used for lateral and longitudinal control, from the predicted trajectory Get the steering, throttle and brake values. The longitudinal controller takes the magnitude of the weighted average of the vectors between waypoints at consecutive time steps, while the lateral controller takes their directions.

[0120] Obtaining the autonomous driving fusion perception model includes: inputting the perception modality into the ResNet unit, obtaining the features of each perception modality, inputting the features of each perception modality into the multi-head attention mechanism unit for adaptive fusion, and calculating the attention weight of each perception modality feature; calculating the gradient of the loss function to the model parameters according to the back propagation algorithm, and using the loss function to perform gradient update on the model parameters according to the Adam optimization algorithm; when the loss cannot be reduced after several consecutive iterations, terminating the training process, and obtaining the autonomous driving fusion perception model, specifically:

[0121] S4. Model training

[0122] Based on the design of the S2 encoder and the design of the S3 trajectory prediction network, after obtaining the entire training model, the optimal model architecture and parameter set are determined. The entire network adopts an end-to-end training method, and the network weights are continuously updated until the model converges. Through end-to-end training, the model can autonomously learn the ability of trajectory prediction from the training data.

[0123] The perception model is trained using the prepared dataset. The training process involves the following steps and formulas:

[0124] S401. Forward Propagation

[0125] The perceptual modality is input into the model, and the feature representation of each perceptual modality is obtained through the feature extraction module. Then, the features of different modalities are adaptively fused through the multi-head attention mechanism, and the attention weight of each perceptual modality is calculated.

[0126] S402. Back propagation: Use the back propagation algorithm to calculate the gradient of the loss function to the model parameters, and use the Adam optimization algorithm to calculate the gradient of the loss function according to the loss function. Perform gradient updates on encoder / decoder parameters:

[0127] (11)

[0128] in, Represents the parameters to be updated, which are the weight matrix and bias vector in the encoder and decoder. is the learning rate, which controls the step size of each parameter update. Represents the loss function Parameters The gradient of , which represents the rate of change of the loss function with respect to the parameters. By multiplying by the learning rate , update the parameters according to the direction and magnitude of the gradient , so that the loss function gradually decreases and the model can better adapt to the training data. The updated parameters It will be used in the next iteration of training process.

[0129] S403, training termination condition: according to the loss function value of the validation set, when the loss cannot be reduced after 10 consecutive iterations, the training process is terminated, the optimal model parameters are saved, and the obtained model is used as the final deployment model.

[0130] S5. Model deployment

[0131] Deploy the trained perception model to the actual new energy vehicle autonomous driving system to achieve real-time intelligent perception and decision-making.

[0132] S501. Select a hardware platform suitable for the autonomous driving system, including processors, storage devices, sensors, and communication modules. Ensure that the hardware meets the computing requirements of the model and has sufficient performance and storage capacity. Verify the compatibility of the hardware with the autonomous driving system.

[0133] S502. Integrate the trained perception model into the software architecture of the autonomous driving system. Determine the input and output interfaces of the model and connect it with other modules. Implement data interaction and communication mechanisms to ensure that the model can work seamlessly with other modules.

[0134] S503. Optimize and compress the model to reduce the size and computational complexity of the model in order to meet the real-time performance requirements. Maintain a balance between performance and accuracy to ensure that the model can run efficiently on new energy vehicles.

[0135] S504. Verify and test the entire autonomous driving system to ensure the performance and safety of the model in real-world scenarios. Conduct simulator testing, closed-loop testing, and field testing, etc., covering various driving scenarios and situations. Check whether the output and behavior of the model are as expected, and evaluate the accuracy and robustness of the model under various test cases.

[0136] S505. After deployment, continuously monitor the performance and output results of the model to ensure the stability and reliability of the model in the actual environment. Collect feedback data and use it to improve and optimize the model. Update the model regularly to adapt to new driving scenarios and requirements and improve perception performance and accuracy.

[0137] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A fusion perception method for automatic driving of new energy vehicles based on an improved multi-head attention mechanism, characterized in that: include: Obtain RGB images and LiDAR scanning data in the vehicle driving environment; Inputting the RGB image and the laser radar scanning data in the vehicle driving environment into the autonomous driving fusion perception model to obtain a predicted trajectory result; The autonomous driving fusion perception model is obtained by training with a training set; wherein the autonomous driving fusion perception model includes: an autonomous driving fusion sense encoder based on a multi-head attention mechanism and a trajectory prediction network sub-model; The ResNet unit of the autonomous driving fusion sense encoder is combined with the multi-head attention mechanism unit of the autonomous driving fusion sense encoder to perceive the attention of each layer in the feature data to perform feature extraction, and perform feature fusion to obtain an environmental context encoding vector, and the environmental context encoding vector is input into the trajectory prediction network sub-model to perform trajectory prediction, and obtain the predicted trajectory result; Using the ResNet unit of the autonomous driving fusion sense encoder in combination with the multi-head attention mechanism unit of the autonomous driving fusion sense encoder to perceive the attention of each layer in the feature data for feature extraction includes: Processing the RGB image and the laser radar scanning data in the vehicle driving environment to generate a dual-channel pseudo image; The multi-head attention mechanism unit is set in front of the convolution layer in each of the ResNet units, a set of queries, keys and values ​​are calculated using linear projection according to the multi-head attention mechanism unit, attention weights are calculated based on the scaled dot product between the queries and the keys, and the values ​​of each query are summarized, and the summarized queries, keys and values ​​are calculated separately Secondary attention, and concatenate all the attention to obtain the attention of each layer; Input the dual-channel pseudo image into the ResNet unit and perform feature extraction in combination with the attention of the corresponding layer to obtain a feature map; The method of using linear projection to calculate a set of queries, keys and values ​​according to the multi-head attention mechanism unit is: in, is the query matrix, the input matrix and the query weight matrix Multiply them together to get; is the key matrix, which is composed of the input matrix With key weight matrix Multiply them together to get; is a value matrix, consisting of the input matrix With value weight matrix Multiply them together to get; The attention weights are calculated based on the scaled dot product between the query and the key, and the value of each query is summarized as: in, , , is the weight matrix of the linear projection, , and are the query, key, and value dimensions of each attention head, respectively; Obtaining the predicted trajectory result includes: Input the environmental context encoding vector into the multi-layer perceptron unit in the trajectory prediction network sub-model for dimensionality reduction, use the reduced environmental context encoding vector to initialize the hidden state of the gated recurrent unit in the trajectory prediction network sub-model, encode in the hidden state based on the update gate control of the gated recurrent unit, output the information flow of the next time step, and obtain the predicted trajectory result in combination with the current position and target position of the vehicle; The method for obtaining the predicted trajectory result is: in, Represents the input information at the current moment, Represents the hidden state of the previous moment. The hidden state acts as the memory of the neural network and contains information about the data that the node has seen before. Represents the hidden state passed to the next moment, represents the candidate hidden state, Represents the reset gate, represents the update gate, represents the sigmoid function, represents the tanh function; Obtaining the autonomous driving fusion perception model includes: Input the perceptual modality into the ResNet unit, obtain the features of each perceptual modality, input the features of each perceptual modality into the multi-head attention mechanism unit for adaptive fusion, and calculate the attention weight of each perceptual modality feature; The gradient of the loss function to the model parameters is calculated according to the back-propagation algorithm, and the gradient of the model parameters is updated using the loss function according to the Adam optimization algorithm: in, Indicates the parameters to be updated. is the learning rate, Represents the loss function Parameters The gradient of When the loss cannot be reduced after several consecutive iterations, the training process is terminated to obtain the autonomous driving fusion perception model; According to the predicted trajectory results, the steering, throttle and braking values ​​of the vehicle are obtained, so as to control the vehicle to perform automatic driving.

2. The method for fusion perception of automatic driving of new energy vehicles based on improved multi-head attention mechanism according to claim 1 is characterized in that: Obtaining the environment context encoding vector includes: Embedding trainable parameters in the learnable position of the autonomous driving fusion sense encoder, projecting the current vehicle speed as a scalar value to a C-dimensional vector using a linear layer, obtaining a feature map, summing the elements of the existing feature map obtained by extracting the feature map by the feature extraction unit, and feeding the summation result back to the modal branch in the multi-head attention mechanism unit; Extracting a feature map of the first target dimension from each modal branch, reducing the dimension of the feature map of the first target dimension by average pooling, and flattening the feature map to obtain a feature vector, namely, a feature vector of the RGB image and a feature vector of the LiDAR scan; The feature vector of the RGB image and the feature vector of the laser radar scan are combined to obtain the environmental context encoding vector.

3. The new energy vehicle automatic driving fusion perception method based on the improved multi-head attention mechanism according to claim 2 is characterized in that: Obtaining the summation result includes: The high-resolution feature map in the autonomous driving fusion sense encoder is downsampled to a fixed resolution using average pooling, the fixed resolution is passed to the attention mechanism unit, and the upsampled original resolution is output using bilinear interpolation, and the feature map is summed with the elements of the existing feature map using the original resolution to obtain the summed result.