A multimodal perception and decision-making method and system based on complex roads

Abstract

The present invention discloses a multimodal perception and decision-making method and system based on complex roads. After the large model masters general knowledge through pre-training, it uses data sets of harsh driving environments for cross-domain learning to improve the perception and decision-making capabilities of the autonomous driving system. After the cross-domain learning of the large model is completed, the downstream task is the information source for autonomous driving perception and decision-making. Point cloud segmentation and 3D target detection provide the decision-making system with object information of the driving environment. Road line detection and split detection use the road information decision-making system to ensure that the vehicle runs on a safe and drivable road. After the large model is migrated in a harsh environment, the vehicle's system will be more generalized and can make correct perception and decision-making in the face of complex and unknown scenes. In order to improve the feature extraction capability of the network, the present invention is provided with three transformer encoders connected in series, which solves the problem of partial data offset caused by transfer learning and residual structure.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving technology, and specifically relates to a multimodal perception and decision-making method and system based on complex roads. Background Art

[0002] As a component of Industry 5.0, autonomous driving plays a vital role in accelerating human-machine collaboration and promoting a human-centric industrial world. Humans learn how to make various decisions through social interactions, learning, and working. Therefore, humans have advantages in standard selection, pattern recognition, decision-making, and group perception. However, humans are easily disturbed by emotions, which further affects rational decision-making during manual driving. Based on the perception and decision-making system of autonomous driving, the vehicle first uses sensors such as lidar, cameras, millimeter-wave radar, and ultrasonic radar, which are commonly included in the perception system, to sense the surrounding environment. However, lidar and cameras are easily affected by complex conditions such as strong light, dust, rain, snow, and fog. These scenarios are referred to as complex high-dynamic scenarios in this paper. Especially when the vehicle is driving on unstructured roads such as mines and off-road vehicles, coupled with the interference of complex high-dynamic scenarios, the vehicle's perception system cannot guarantee robustness, and the decision-making system is also easily affected. Millimeter-wave radar can maintain perception capabilities in harsh environments, but it cannot distinguish the elevation of objects. In recent years, 4D millimeter-wave imaging technology has been proposed. In addition to the range, direction, and velocity sensing capabilities of traditional millimeter-wave radar, 4D millimeter-wave radar can also analyze high-dimensional data, achieving full-target, all-condition, and all-weather coverage. This technology enables millimeter-wave radar to generate a point cloud structure similar to that of lidar. However, due to operating frequency limitations, millimeter-wave radar cannot generate the same large number of points as lidar. Typically, in each frame, the number of points generated by millimeter-wave radar is one thousandth or even less than that of lidar. To address the sparse point cloud and insufficient object representation of millimeter-wave radar, methods have been proposed to fuse 4D millimeter-wave radar with cameras, achieving excellent performance in object detection tasks. Perception fusion combines information from multiple sensors (such as cameras, lidar, and radar) to achieve more comprehensive, accurate, and robust environmental perception. Leveraging information collected by multiple cameras, perception fusion can address issues such as target occlusion, environmental fluctuations, and data sparsity from a single sensor. However, due to the huge differences between different driving environments, the perception and decision-making system of cars cannot effectively generalize and adapt to new environments when faced with complex and highly dynamic scenes. In recent years, with the emergence of Chatgpt as a milestone, more and more researchers have begun to try to combine large models in different fields. Large models are neural network models with a large number of parameters and a large number of hierarchical structures. These large models require a lot of computing resources during training and inference, including high-performance hardware (such as graphics processing units - GPUs) and large-scale data sets. More and more work is trying to fine-tune the models for the scenarios they need based on the excellent optimization capabilities and informed cognition of large models. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a multimodal perception and decision-making method and system based on complex roads, which can improve the system's generalization ability in complex unknown scenarios by performing transfer learning on a large model.

[0004] A multimodal perception and decision-making method based on complex roads, including:

[0005] First, input the RGB image x and plane normal image y of the unstructured road;

[0006] Based on the RGB image and the plane normal image, image fragments are obtained respectively, and then fragment encoding is performed to obtain the corresponding fragment vector x1 and fragment vector y1 respectively;

[0007] Input the slice vector x1 and the slice vector y1 into the first transformer encoder respectively to obtain x2 and y2 respectively;

[0008] The slice vectors x1 and y1 are input into a linear layer, which evaluates the importance of each slice feature and outputs a weight. For m*n slice features, m*n feature weights are output. The weights are dot-multiplied with the slice features. The RGB image slice features and the plane normal image slice features after the dot product are added to obtain feature z1.

[0009] Feature z1 is fed into the first correction unit of three series-connected correction units. Each correction unit consists of a transformer encoder and a distribution correction module. Feature z1 is split into two parts. One part passes through the transformer encoder, and the other part is dot-multiplied with the trainable tensor in the distribution correction module. The dot-multiplication result is added to the input result of the parallel transformer encoder to obtain the output result z2, which enters the next correction unit. And so on. After processing in the next two correction units, the output results z3 and z4 are obtained.

[0010] After unifying the size of the results z2, z3 and z4, they are concatenated in the feature dimension and passed through a linear layer to obtain the final output result out, which is the prediction result of the image;

[0011] Among them, when training the trainable tensor, the binary cross entropy loss function is used:

[0012]

[0013] in, is the predicted probability of the pixel at the jth row and kth column in the i-th training sample, and is the true probability label corresponding to the pixel position; batch represents the number of training samples.

[0014] Preferably, the trainable tensor is implemented based on the pytorch framework.

[0015] Preferably, the initial value of the trainable tensor is an all-0 trainable tensor of size c*c, where c is the number of channels of feature z1.

[0016] Preferably, the trainable tensor is trained using a stochastic gradient descent optimizer with momentum.

[0017] The present invention has the following beneficial effects:

[0018] The purpose of this invention is to provide a multimodal perception and decision-making method and system based on complex roads, and to design a multimodal fusion algorithm for point cloud method plane images and images to give full play to the data complementarity of different modalities and improve the robustness of the system; after the large model masters general knowledge through pre-training, it uses data sets of harsh driving environments for cross-domain learning, and uses fine-tuning or knowledge distillation methods to adjust the model, thereby improving the perception and decision-making capabilities of the autonomous driving system in downstream tasks, ensuring that the vehicle can make safe and intelligent decisions in complex and unknown scenarios.

[0019] After the cross-domain learning of the large model is completed, the downstream task is the information source for autonomous driving perception decision-making. The image denoising task improves the accuracy of the perception system in the face of complex situations. Point cloud segmentation and 3D target detection provide the decision-making system with object information about the driving environment. Road line detection and split detection are the perception of road surfaces. The road information decision-making system is used to ensure that the vehicle runs on a safe and drivable road. After the large model is migrated in a harsh environment, the vehicle's system will be more generalized and can make correct perceptions and decisions in the face of complex and unknown scenes. It can achieve safer and more intelligent decisions in applications such as route planning and collaborative driving. In order to improve the feature extraction capability of the network, the present invention is provided with three transformer encoders connected in series, which solves the problem of partial data offset caused by transfer learning and residual structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the multimodal perception and decision-making system framework based on complex roads proposed by the present invention.

[0021] Figure 2 This is a flow chart of the multimodal perception and decision-making method based on complex roads proposed in this invention.

[0022] Figure 3 DCM algorithm architecture for the correction module.

[0023] Figure 4 This is a visualization diagram of the segmentation result obtained by applying the method of the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0025] 1. System Framework

[0026] After the large model masters general knowledge through pre-training, it uses data sets from harsh driving environments for cross-domain learning and uses fine-tuning or knowledge distillation methods to adjust the model, thereby improving the perception and decision-making capabilities of the autonomous driving system in downstream tasks, ensuring that the vehicle can make safe and intelligent decisions in complex and unknown scenarios.

[0027] like Figure 1 As shown, the present invention proposes an adaptive architecture for large models on complex roads. The large model has a huge neural network structure. During the pre-training of the large model, a large amount of training data is also provided for the large model to learn. This enables the large model to have the ability to master general knowledge. When the large model needs to be applied to a specific field, a downstream task model with excellent performance can be obtained with very low training cost. During the cross-domain learning stage of the model, datasets of harsh driving environments (such as rain, sandstorms, fog, or off-roading) are used to adjust the large model. Currently common methods include fine-tuning, knowledge distillation, and transfer learning.

[0028] Transfer learning is the most common downstream training method for large models. Typically, the fine-tuning process involves fixing the large model's massive parameters, designing downstream task modules, and updating only the task modules during the model's backpropagation. After the task modules have been trained for a certain period of time, all parameters are unfrozen, and the entire training is repeated a small number of times to ensure that the entire model learns the distribution patterns of the downstream task dataset as much as possible.

[0029] The knowledge distillation method uses a large model to train data, then uses a smaller model to fit the larger model's output. This allows the smaller model to learn the knowledge of the larger model. In practical applications, after the small model is trained, it is used to learn relevant knowledge about complex roads. Since the inference process does not involve the computational process of the larger model, knowledge distillation can effectively reduce the model's memory usage in downstream tasks.

[0030] After cross-domain learning of the large model is complete, downstream tasks serve as information sources for autonomous driving perception and decision-making. Image denoising improves the perception system's accuracy in complex situations, while point cloud segmentation and 3D object detection provide the decision-making system with information about objects in the driving environment. Road line detection and segmentation detection provide road surface perception, leveraging road information to ensure the vehicle remains on safe and drivable roads. After migrating the large model to harsh environments, the vehicle's system becomes more generalizable, enabling accurate perception and decision-making in complex and unknown scenarios, enabling safer and more intelligent decision-making in applications such as route planning and collaborative driving.

[0031] 2. Segmentation Network

[0032] The network model structure is as follows Figure 2 As shown in the figure, the input is an RGB image x and a surface normal image y of an unstructured road. The RGB image is acquired by a camera. The surface normal image is obtained by post-processing a point cloud acquired by a lidar. A depth map is first generated from the point cloud. The surface normal image is calculated by performing three filtering operations on the inverse depth image or disparity image: horizontal and vertical gradient filtering, and mean / median filtering. The surface normal image represents the angle of the road surface normal vector in space.

[0033] Based on the RGB image and the plane normal image, the image is divided equally along the length m and the width n, resulting in m*n image slices, with no overlap. Slice encoding is performed using a convolutional neural network. In slice encoding, the RGB image slices and the plane normal image slices are first passed through a convolutional neural network layer for feature extraction. Each subsequent slice is then mapped into a multi-channel vector using a linear layer.

[0034] In Transformer Encoder 1, the slice vector x1 of the RGB image is calculated to obtain x2, and the slice vector y1 of the plane normal image is calculated to obtain y2. A linear layer is constructed, in which the training weights are shared. This linear layer takes x1 and y1 as input. It evaluates the importance of each slice feature and outputs a weight. For m*n slice features, it outputs m*n feature weights. These weights are then dot-multiplied with the slice features. The dot-multiplied RGB image slice features and plane normal image slice features are added together to obtain z1.

[0035] Due to the partial data offset problem caused by transfer learning and residual structure, this paper proposes a distribution correction method, which uses a trainable matrix to replace the commonly used regularization operation. The introduction of the residual connection structure has played an excellent role in reducing feature attenuation and enhancing gradient return. In the network structure of the present invention, the present invention uses the residual structure to bridge the Transformer encoder to achieve better training results, but the introduction of residuals will further amplify the contrast:

[0036] x l+1 =x l +F l (x l )

[0037] Where x represents the image feature, l represents the number of layers, and F(.) represents the transformer encoder. Assume that x l and x l (x l ) are independent of each other, and x lThe variance of x l (x l ) has a variance of It is easy to see that each residual connection is constantly amplifying the variance. The general approach is to perform regularization after the connection calculation:

[0038] x l+1 =Norm(x l +F l (x l ))

[0039] This approach alleviates the expansion of variance to a certain extent, but it results in the gradient being smaller in the gradient backpropagation process of the network closer to the front, which deviates from the original intention of the residual connection. In order to improve the feature extraction capability of the network, the present invention sets three transformer encoders connected in series, and each transformer encoder is connected in parallel with a distribution correction module, such as Figure 3 As shown, the input is copied twice: one copy passes through the transformer encoder, and the other is calculated in the distribution correction module, which performs a dot product of the input data with a trainable tensor. A trainable tensor is initialized to 0. Based on the PyTorch framework, the initialization code is: zero_tensor = torch.zeros(c, c, requires_grad = True). c is the number of feature channels. This code initializes an all-zero trainable tensor of size c*c. This tensor is used to balance data drift caused by excessive network depth and transfer learning.

[0040] In each distribution correction module, a separate trainable tensor is initialized. Each input zi is dot-multiplied with the trainable tensor. The result of the dot product is then added to the input of the parallel transformer encoder to produce the output zi+1, which is then fed into the next level of distribution correction module and transformer encoder.

[0041] The computed features z2, z3, z4 of each layer are then output through transformer encoders 2, 3, and 4. Since the sizes of z2, z3, and z4 are different, in order to enable features from different layers to be concatenated, the sizes of features z2, z3, and z4 need to be adjusted.

[0042] The size of z2 is c1*(h / 4)*(w / 4), the size of z3 is c2*(h / 8)*(w / 8), and the size of z3 is c3*(h / 16)*(w / 16), where c1, c2, and c3 represent the number of channels of the feature, h represents the height of the feature, and w represents the width of the feature.

[0043] After dimension mapping, only the channel dimensions of the output of different layers are changed: the size of z2 is c*(h / 4)*(w / 4), the size of z3 is c*(h / 8)*(w / 8), and the size of z3 is c*(h / 16)*(w / 16).

[0044] The upsampling module consists of transposed convolutions, which change the height and width of features. Transposed convolutions upsample z3 and z4 to c*(h / 4)*(w / 4). The resized features z2, z3, and z4 are concatenated with the features output by Transformer Encoder 1 (which have a size of c*(h / 4)*(w / 4)) along the feature dimension. The final output, out, is then passed through a linear layer to produce the predicted image.

[0045] When training a trainable tensor, a binary cross entropy loss function is used:

[0046]

[0047] in, is the predicted probability of the pixel at the jth row and kth column in the i-th training sample, and is the true probability label corresponding to the pixel position; batch represents the number of training samples.

[0048] This model was implemented in PyTorch and trained using the Stochastic Gradient Descent with Momentum (SGDM) optimizer. The initial learning rate was set to 0.001, and the batch size was set to 8. All experiments were performed on a single Nvidia RTX3090 GPU. The image size for training and testing was set to 1280×704.

[0049] The effectiveness of the proposed model is demonstrated in the experiments. Compared with other models, it achieves better evaluation indicators, as shown in the following table:

[0050]

[0051] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal perception and decision-making method based on complex roads, characterized by: include: First, input the RGB image x and plane normal image y of the unstructured road; Based on the RGB image and the plane normal image, image fragments are obtained respectively, and then fragment encoding is performed to obtain the corresponding fragment vector x1 and fragment vector y1 respectively; Input the slice vector x1 and the slice vector y1 into the first transformer encoder respectively to obtain x2 and y2 respectively; The slice vectors x1 and y1 are input into a linear layer, which evaluates the importance of each slice feature and outputs a weight. For m*n slice features, m*n feature weights are output. The weights are dot-multiplied with the slice features. The RGB image slice features and the plane normal image slice features after the dot product are added to obtain feature z1. Feature z1 is fed into the first correction unit of three series-connected correction units. Each correction unit consists of a transformer encoder and a distribution correction module. Feature z1 is split into two parts. One part passes through the transformer encoder, and the other part is dot-multiplied with the trainable tensor in the distribution correction module. The dot-multiplication result is added to the input result of the parallel transformer encoder to obtain the output result z2, which enters the next correction unit. And so on. After processing in the next two correction units, the output results z3 and z4 are obtained. After unifying the size of the results z2, z3 and z4, they are concatenated in the feature dimension and passed through a linear layer to obtain the final output result out, which is the prediction result of the image; Among them, when training the trainable tensor, the binary cross entropy loss function is used: in, is the predicted probability of the pixel at the jth row and kth column in the i-th training sample, and is the true probability label corresponding to the pixel position; batch represents the number of training samples.

2. The multimodal perception and decision-making method based on complex roads according to claim 1, characterized in that: The trainable tensor is implemented based on the pytorch framework.

3. The multimodal perception and decision-making method based on complex roads according to claim 1, characterized in that: The initial value of the trainable tensor is a full-zero trainable tensor of size c*c, where c is the number of channels of feature z1.

4. The multimodal perception and decision-making method based on complex roads according to claim 1, characterized in that: Trainable tensors are trained using a stochastic gradient descent optimizer with momentum.

Images