Multi-modal sensing and decision-making method and system based on complex road

Abstract

The invention discloses a multi-modal perception and decision-making method and system based on a complex road, and the method comprises the steps: enabling a large model to grasp general knowledge through pre-training, carrying out the cross-domain learning through a data set of a severe driving environment, and improving the perception and decision-making capability of an automatic driving system; after cross-domain learning of the large model is completed, a downstream task is an information source of an automatic driving perception decision, and point cloud segmentation and 3D target detection provide object information of a driving surrounding environment for a decision system; the road line detection and the splitting detection use a road information decision-making system to ensure that the vehicle runs on a safe and drivable road; after the large model is migrated in a severe environment, the system of the vehicle is more generalized, and correct perception and decision can be made in a complex and unknown scene; in order to improve the feature extraction capability of the network, three transformers encoders which are sequentially connected in series are arranged, so that the problem of partial data offset caused by transfer learning and a residual structure is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving technology, and specifically relates to a multimodal perception and decision-making method and system based on complex roads. Background Art

[0002] As a component of Industry 5.0, autonomous driving plays an important role in accelerating human-machine cooperation and promoting a people-oriented industrial world. Humans learn how to make various decisions in social, learning, working and other scenarios in a long period of social life, so humans have advantages in standard selection, pattern recognition, decision-making and group perception. However, people are easily disturbed by emotions, which further affects the reasonable decision-making during manual driving. Based on the perception decision system of autonomous driving, the vehicle first uses sensors such as laser radar, camera, millimeter wave radar and ultrasonic radar usually included in the perception system to sense the surrounding environment. However, laser radar and camera are easily affected by complex situations such as strong light, dust, rain, snow and fog. These scenes are called complex high-dynamic scenes in the present invention. Especially when the vehicle's driving environment is in unstructured roads such as mines and off-road, coupled with the interference of complex high-dynamic scenes, the vehicle's perception system cannot guarantee robustness, and the decision system is also easily affected. Millimeter wave radar can maintain perception capabilities in harsh environments, but it cannot distinguish the elevation of objects. In recent years, 4D millimeter wave imaging technology has been proposed. In addition to the distance, direction, and speed that traditional millimeter wave radar can perceive, 4D millimeter wave radar can also analyze data in a high-dimensional way to meet the needs of full-target, full-condition, and all-weather coverage. Based on 4D millimeter wave radar imaging technology, millimeter wave radar can generate a point cloud structure similar to that of laser radar. However, millimeter wave radar is limited by the operating frequency and cannot generate a large number of points like laser radar. Generally speaking, in each frame, the points of millimeter wave radar are often one thousandth or even less than those of laser radar. In order to solve the problem of sparse point cloud and insufficient object expression of millimeter wave radar, there are currently works that have proposed a method of fusion of 4D millimeter wave radar and camera, which has achieved excellent performance in target detection tasks. Perception fusion is a method of combining information from multiple sensors (such as cameras, laser radars, radars, etc.) to obtain a more comprehensive, accurate, and robust environmental perception. Using information collected by multiple cameras, perception fusion can solve the problems of target occlusion, environmental changes, and sparse data from a single sensor. However, due to the huge differences between different driving environments, the perception and decision-making system of cars cannot effectively generalize and adapt to new environments when facing complex and highly dynamic scenes. In recent years, with the emergence of Chatgpt as a milestone, more and more researchers have begun to try to combine large models in different fields. Large models are neural network models with a large number of parameters and a large number of hierarchical structures. These large models require a lot of computing resources during training and reasoning, including high-performance hardware (such as graphics processing units-GPUs) and large-scale data sets. More and more work is trying to fine-tune the models for the scenarios they need based on the excellent optimization capabilities and notification cognition of large models. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a multimodal perception and decision-making method and system based on complex roads, which can improve the generalization ability of the system in complex and unknown scenarios by performing transfer learning on a large model.

[0004] A multimodal perception and decision-making method based on complex roads, comprising:

[0005] First, input the RGB image x and plane normal image y of the unstructured road;

[0006] Based on the RGB image and the plane normal image, image slices are obtained respectively, and then the slice encoding is performed to obtain the corresponding slice vector x1 and the slice vector y1 respectively;

[0007] Input the slice vector x1 and the slice vector y1 into the first transformer encoder to obtain x2 and y2 respectively;

[0008] The slice vectors x1 and y1 are input into a linear layer to evaluate the importance of each slice feature input and output a weight. For m*n slice features, m*n feature weights are output. The weight is dot-multiplied with the slice feature. The RGB image slice feature and the plane normal image slice feature after the dot multiplication are added to obtain feature z1.

[0009] Feature z1 is sent to the first correction unit of the three series-connected correction units; each correction unit consists of a transformer encoder and a distribution correction module; feature z1 is divided into two parts, one part passes through the transformer encoder, and the other part is point-multiplied with the trainable tensor in the distribution correction module. The result after the point multiplication is added to the input result of the parallel transformer encoder to obtain the output result z2, which enters the next correction unit, and so on. After being processed in the next two correction units, the output results z3 and z4 are obtained;

[0010] After the results z2, z3 and z4 are unified in size, they are concatenated in the feature dimension and passed through a linear layer to obtain the final output result out, which is the prediction result of the image;

[0011] Among them, when training the trainable tensor, the binary cross entropy loss function is used:

[0012]

[0013] in, is the predicted probability of the pixel at the jth row and kth column in the ith training sample, and is the true probability label corresponding to the pixel position; batch represents the number of training samples.

[0014] Preferably, the trainable tensor is implemented based on the pytorch framework.

[0015] Preferably, the initial value of the trainable tensor is an all-0 trainable tensor of size c*c, where c is the number of channels of feature z1.

[0016] Preferably, the trainable tensors are trained using a stochastic gradient descent optimizer with momentum.

[0017] The present invention has the following beneficial effects:

[0018] The purpose of the present invention is to provide a multimodal perception and decision-making method and system based on complex roads, and to design a multimodal fusion algorithm for point cloud method plane images and images, so as to give full play to the data complementarity of different modalities and improve the robustness of the system; after the large model masters general knowledge through pre-training, it uses the data set of harsh driving environment for cross-domain learning, and adjusts the model by fine-tuning or knowledge distillation methods, so as to improve the perception and decision-making ability of the autonomous driving system in downstream tasks, and ensure that the vehicle can make safe and intelligent decisions in complex and unknown scenarios.

[0019] After the cross-domain learning of the large model is completed, the downstream task is the information source for autonomous driving perception decision-making. The image denoising task improves the accuracy of the perception system in the face of complex situations. Point cloud segmentation and 3D target detection provide the decision-making system with object information about the driving environment. Road line detection and split detection are the perception of road surfaces. The road information decision system is used to ensure that the vehicle runs on a safe and drivable road. After the large model is migrated in a harsh environment, the vehicle's system will be more generalized, and it can make correct perceptions and decisions in the face of complex and unknown scenes, and can achieve safer and smarter decisions in applications such as route planning and collaborative driving. In order to improve the feature extraction capability of the network, the present invention is provided with three transformer encoders connected in series, which solves the problem of partial data offset caused by transfer learning and residual structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the multimodal perception and decision-making system framework based on complex roads proposed by the present invention.

[0021] Figure 2 This is a flow chart of the multimodal perception and decision-making method based on complex roads proposed in the present invention.

[0022] Figure 3 DCM algorithm architecture for the correction module.

[0023] Figure 4 This is a visualization diagram of the segmentation result obtained by applying the method of the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0025] 1. System Framework

[0026] After the large model has mastered general knowledge through pre-training, it uses data sets from harsh driving environments for cross-domain learning and uses fine-tuning or knowledge distillation methods to adjust the model, thereby improving the perception and decision-making capabilities of the autonomous driving system in downstream tasks, ensuring that the vehicle can make safe and intelligent decisions in complex and unknown scenarios.

[0027] like Figure 1 As shown, the present invention proposes an adaptive architecture of a large model under complex roads. The large model has a huge neural network structure. When pre-training the large model, a large amount of training data is also provided for the large model to learn, which enables the large model to have the ability to master general knowledge. When the large model needs to be applied to a specific field, a downstream task model with excellent performance can be obtained with a very small training cost. In the cross-domain learning stage of the model, the data set of harsh driving environments (such as rain, sandstorms, fog or off-road, etc.) is used to adjust the large model. The current common methods are fine-tuning, knowledge distillation and transfer learning.

[0028] Transfer learning is the most common downstream training method for large models. Under normal circumstances, the fine-tuning process is to fix the huge parameters of the large model, design the downstream task module, and only update the task module during the back propagation process of the model. After the task module has been trained for a certain period of time, all parameters are unfrozen, and then a small number of rounds of training are performed throughout the entire model so that the entire model can learn the distribution pattern of the downstream task data set as much as possible.

[0029] The method of knowledge distillation is to use a large model to train data, and then use a small model to fit the output of the large model. The small model can learn the knowledge of the large model. In practical applications, after the small model is trained, it is used to learn the relevant knowledge of complex roads, and the reasoning does not include the calculation process of the large model, so knowledge distillation can effectively reduce the memory usage of the model in downstream tasks.

[0030] After the cross-domain learning of the large model is completed, the downstream task is the information source for autonomous driving perception decision-making. The image denoising task improves the accuracy of the perception system in complex situations. Point cloud segmentation and 3D target detection provide the decision-making system with object information about the driving environment. Road line detection and split detection are the perception of the road surface. The road information decision-making system ensures that the vehicle runs on a safe and drivable road. After the large model is migrated in a harsh environment, the vehicle's system will be more generalized, and it can make correct perceptions and decisions in complex and unknown scenes, and can achieve safer and smarter decisions in applications such as route planning and collaborative driving.

[0031] 2. Segmentation Network

[0032] The network model structure is as follows Figure 2 As shown, first input the RGB image x and plane normal image y of the unstructured road. The RGB image is collected by the camera. The plane normal image is obtained by post-processing the point cloud collected by the lidar. First, the depth map is generated from the point cloud, and the surface normal image is calculated by performing three filtering operations on the inverse depth image or disparity image: two image gradient filtering in the horizontal and vertical directions and mean / median filtering. The plane normal image represents the normal vector angle of the road surface in space.

[0033] Based on the RGB image and the plane normal image, the image is divided into m equal lengths and n equal widths to obtain m*n image slices, and each slice does not overlap. Slice coding is composed of convolutional neural networks. In slice coding, the RGB image slices and the plane normal image slices are first extracted through the convolutional neural network layer. Each subsequent slice is mapped into a multi-channel vector by a linear layer.

[0034] In transformer encoder 1, the slice vector x1 of the RGB image is calculated to obtain x2, and the slice vector y1 of the plane normal image is calculated to obtain y2. Construct a linear layer in which the training weights are shared. The inputs of this linear layer are x1 and y1. This linear layer will evaluate the importance of each slice feature of the input and output a weight, so for m*n slice features, it will output m*n feature weights, which are dot-multiplied with the slice features. The dot-multiplied RGB image slice features and plane normal image slice features are added together to obtain z1.

[0035] Due to the partial data offset problem caused by transfer learning and residual structure, the present invention proposes a distribution correction method, which uses a trainable matrix to replace the commonly used regularization operation. The residual connection structure has played an excellent effect in reducing feature attenuation and enhancing gradient return. In the network structure of the present invention, the present invention uses the residual structure to bridge the Transformer encoder to achieve better training results, but the introduction of residuals will lead to further amplification of the contrast:

[0036] x l+1 =x l +F l (x l )

[0037] Where x represents the image feature, l represents the number of layers, and F(.) represents the transformer encoder. Assume that x l and F l (x l ) are independent of each other, and x lThe variance of F l (x l ) has a variance of It is easy to see that each residual connection is constantly amplifying the variance. The general approach is to regularize after the connection calculation:

[0038] x l+1 =Norm(x l +F l (x l ))

[0039] This method alleviates the expansion of variance to a certain extent, but it causes the gradient obtained by the network closer to the front to be smaller during the gradient backpropagation process, which deviates from the original intention of the residual connection. In order to improve the feature extraction ability of the network, the present invention is provided with three transformer encoders connected in series, and each transformer encoder is connected in parallel with a distribution correction module, such as Figure 3 As shown, the input is copied twice, one is passed through the transformer encoder, and the other is calculated in the distribution correction module, that is, the input data is multiplied by the trainable tensor; the trainable tensor is a tensor initialized to 0. Based on the pytorch framework, the initialization code is: zero_tensor = torch.zeros(c,c,requires_grad = True). c is the number of channels of the feature, and a trainable tensor of size c*c with all zeros is initialized through this code. This tensor is used to balance the problem of partial data offset caused by too deep network modules and transfer learning.

[0040] In each distribution correction module, a trainable tensor is initialized separately. For each input zi, it is dot-multiplied with the trainable tensor, and the result of the dot multiplication is added to the input result of the transformer encoder connected in parallel to obtain the output result zi+1, which is sent to the next level distribution correction module and transformer encoder.

[0041] Then the calculated features z2, z3, z4 of each layer are output through transformer encoders 2, 3, and 4. Since the sizes of z2, z3, and z4 are different, in order to enable the features of different layers to be spliced, the sizes of features z2, z3, and z4 need to be adjusted.

[0042] The size of z2 is c1*(h / 4)*(w / 4), the size of z3 is c2*(h / 8)*(w / 8), and the size of z3 is c3*(h / 16)*(w / 16), where c1, c2, c3 represent the number of channels of the feature, h represents the height of the feature, and w represents the width of the feature.

[0043] After dimension mapping, only the channel dimensions of the outputs of different layers are changed: the size of z2 is c*(h / 4)*(w / 4), the size of z3 is c*(h / 8)*(w / 8), and the size of z4 is c*(h / 16)*(w / 16).

[0044] The upsampling module is composed of transposed convolutions, which change the height and width of the features. The transposed convolution upsamples the size of z3 and z4 to c*(h / 4)*(w / 4). The unified features z2, z3, z4 and the features output by transformer encoder 1 (size c*(h / 4)*(w / 4)) are concatenated in the feature dimension and passed through the linear layer to obtain the final output out, which is the prediction result of the image.

[0045] When training a trainable tensor, a binary cross entropy loss function is used:

[0046]

[0047] in, is the predicted probability of the pixel at the jth row and kth column in the ith training sample, and is the true probability label corresponding to the pixel position; batch represents the number of training samples.

[0048] This model is implemented in Pytorch, and the model is trained using the Stochastic Gradient Descent with Momentum (SGDM) optimizer. The initial learning rate is set to 0.001 and the batch size is set to 8. All experiments are performed on a Nvidia RTX3090 GPU device. The image size for training and testing is set to 1280×704.

[0049] The effectiveness of the proposed model is proved in the experiment. Compared with other models, it achieves better evaluation indicators, as shown in the following table:

[0050]

[0051] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal perception and decision-making method based on complex roads, characterized in that: include: First, input the RGB image x and plane normal image y of the unstructured road; Based on the RGB image and the plane normal image, image slices are obtained respectively, and then the slice encoding is performed to obtain the corresponding slice vector x1 and the slice vector y1 respectively; Input the slice vector x1 and the slice vector y1 into the first transformer encoder to obtain x2 and y2 respectively; The slice vectors x1 and y1 are input into a linear layer to evaluate the importance of each slice feature input and output a weight. For m*n slice features, m*n feature weights are output. The weight is dot-multiplied with the slice feature. The RGB image slice feature and the plane normal image slice feature after the dot multiplication are added to obtain feature z1. Feature z1 is sent to the first correction unit of the three series-connected correction units; each correction unit consists of a transformer encoder and a distribution correction module; feature z1 is divided into two parts, one part passes through the transformer encoder, and the other part is point-multiplied with the trainable tensor in the distribution correction module. The result after the point multiplication is added to the input result of the parallel transformer encoder to obtain the output result z2, which enters the next correction unit, and so on. After being processed in the next two correction units, the output results z3 and z4 are obtained; After the results z2, z3 and z4 are unified in size, they are concatenated in the feature dimension and passed through a linear layer to obtain the final output result out, which is the prediction result of the image; Among them, when training the trainable tensor, the binary cross entropy loss function is used: in, is the predicted probability of the pixel at the jth row and kth column in the ith training sample, and is the true probability label corresponding to the pixel position; batch represents the number of training samples.

2. A multimodal perception and decision-making method based on complex roads as claimed in claim 1, characterized in that: The trainable tensor is implemented based on the pytorch framework.

3. The multimodal perception and decision-making method based on complex roads as claimed in claim 1, characterized in that: The initial value of the trainable tensor is a full-zero trainable tensor of size c*c, where c is the number of channels of feature z1.

4. The multimodal perception and decision-making method based on complex roads as claimed in claim 1, characterized in that: Trainable tensors are trained using a stochastic gradient descent optimizer with momentum.

Images