Intelligent vehicle end-to-end behavior decision method and system based on multi-modal information fusion

Abstract

The invention discloses an intelligent vehicle end-to-end behavior decision method and system based on multi-modal information fusion. The method comprises the following steps: acquiring RGB data and point cloud data of surrounding environment information; distortion compensation is carried out on the RGB data; performing translation and rotation operation on the cloud data, converting the cloud data into a camera coordinate system, and obtaining a front view of the point cloud; obtaining the fused dense depth information; inputting the dense depth information as environmental perception data into a behavior decision network, and outputting a decision action of the intelligent vehicle; and obtaining a reference instant return of each decision moment, comparing the reference instant return with the cumulative return of the decision action, and when the cumulative return of the decision action is lower than the reference instant return, constraining and correcting the decision action based on the rule backup strategy. The decision-making action of the intelligent vehicle is generated by adopting the behavior decision-making network, and the safety constraint is implemented based on the rule standby strategy, so that the decision-making effectiveness and the driving safety of the intelligent vehicle in the complex dynamic road environment are ensured.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle autonomous driving behavior decision-making technology, specifically relating to an end-to-end behavior decision-making method and system for intelligent vehicles based on multimodal information fusion. Background Technology

[0002] With the rapid development of the automotive industry, intelligent driving has become a key research focus. Intelligent vehicles generally follow a "perception-decision-execution" technical architecture. In this framework, the perception system first acquires multi-source environmental information through sensors such as onboard cameras and LiDAR and performs fusion processing; the decision-making system, based on the perceived information and combined with road structure and traffic constraints, formulates safe behavior strategies and plans reasonable decision trajectories; finally, the underlying execution mechanism, based on the decision trajectory, uses tracking control algorithms to solve for specific front wheel steering angles and accelerator / pedal signals to track the target trajectory.

[0003] As the "brain" of intelligent vehicles, the decision-making system plays a core role in behavioral decision-making and trajectory planning. In recent years, end-to-end behavioral decision-making technology has gradually become one of the mainstream research directions in the field of intelligent driving due to its enormous potential to improve the autonomy and driving safety of intelligent vehicles. However, in highly dynamic scenarios such as urban roads, the environmental information acquired by a single type of onboard sensor often suffers from insufficient robustness, especially under complex conditions such as rain, snow, occlusion, and nighttime. For example, while cameras have advantages in color recognition, they are prone to information acquisition failure in occluded scenarios due to blurred boundaries. Similarly, LiDAR performs excellently in spatial ranging with point cloud data containing rich depth information, but it is difficult to accurately identify object categories due to the lack of texture information. These inherent limitations of sensors make it difficult for end-to-end decision-making systems to acquire reliable and comprehensive environmental perception information, thereby affecting the safety and effectiveness of behavioral decisions. Therefore, an end-to-end behavioral decision-making method for intelligent vehicles based on multimodal information fusion is needed to improve perception robustness and decision-making safety in dynamic scenarios. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, the present invention aims to provide an end-to-end behavior decision-making method and system for intelligent vehicles based on multimodal information fusion, so as to solve the problem of increased decision-making risks caused by the perception limitations of a single on-board sensor in intelligent vehicles in the existing technologies.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] The present invention provides an end-to-end behavior decision-making method for intelligent vehicles based on multimodal information fusion, comprising the following steps:

[0007] Step 1): Acquire RGB data and point cloud data of the surrounding environment using the vehicle's onboard camera and LiDAR;

[0008] Step 2): Perform distortion compensation on the RGB data to complete the preprocessing of the RGB data;

[0009] Step 3): Perform translation and rotation operations on the cloud data, transform it to the camera coordinate system, and use the forward projection method to obtain the front view of the point cloud to complete the point cloud data preprocessing;

[0010] Step 4): Perform multimodal fusion on the preprocessed RGB data and point cloud data from Steps 2) and 3) to obtain fused dense depth information;

[0011] Step 5): Input the dense depth information as environmental perception data into the trained behavior decision network and output the intelligent vehicle's decision action;

[0012] Step 6): Obtain the baseline instant reward at each decision moment and compare it with the cumulative reward of the decision action in Step 5). When the cumulative reward of the decision action is lower than the baseline instant reward, constrain and correct the decision action based on the rule-based backup strategy.

[0013] Furthermore, the distortion compensation process for RGB data in step 2) considers both radial and tangential distortions, as shown below:

[0014] ;

[0015] In the formula, There are 5 distortion parameters; is the position coordinate of the target; r is the radial distance from the target to the optical center in the camera coordinate system; The corrected position coordinates of the perceived target.

[0016] Furthermore, the specific steps for preprocessing cloud data in step 3) are as follows:

[0017] 31) Define the coordinate system of the lidar as O L The camera coordinate system is defined as O C The spatial coordinates of the same perceived target T in the two coordinate systems are denoted as follows: and The three-dimensional spatial transformation of target T is represented as follows:

[0018] ;

[0019] In the formula, T L For O L A point in space within a coordinate system; TC For O C A point in space within a coordinate system; R C S is a spatial rotation matrix; C It is a translation matrix;

[0020] 32) Rotate the spatial rotation matrix R C Decompose the rotation into counterclockwise angular variables about the X, Y, and Z axes, resulting in three rotation matrices. , and The expression is as follows:

[0021] ;

[0022] In the formula, , and These are the rotation angles around the X-axis, Y-axis, and Z-axis, respectively.

[0023] Rotating a point in space in the order of the X, Y, and Z axes, and multiplying the three rotation matrices together to represent the transformation of the three-dimensional spatial state, is as follows:

[0024] ;

[0025] 33) Applying the translation matrix S C Translate a point in space from point O L Coordinate system transformation to O C The coordinate system is represented as follows:

[0026] ;

[0027] 34) In point cloud data from O L Coordinate system transformation to O C After establishing the coordinate system, the point cloud data is projected forward, and the coordinates of each spatial point in O are calculated. C The projection position in the coordinate system is determined, and its position is checked to see if it overflows the image boundary. If it does, it is cropped accordingly.

[0028] Furthermore, in step 4), a deep information fusion network is used to perform multimodal fusion of RGB data and point cloud data, as detailed below:

[0029] 41) Global context information is obtained from RGB data and point cloud data using a global transform encoder, and the encoded features are represented as follows:

[0030] ;

[0031] In the formula, and These are global encoding features obtained from RGB data and point cloud data, respectively; and is the input to the encoder; n is the number of spatial points; Enc is the global transform encoder; The encoded feature of the nth spatial point in RGB data; The encoded feature of the nth spatial point in the point cloud data;

[0032] 42) The encoded features are aggregated bidirectionally using both top-down and bottom-up paths, and the encoder output is divided into four groups. The outputs of the last block of the first, second, third, and fourth groups are respectively represented as follows: , , and The coding features it constitutes As input to a multi-level decoder;

[0033] 43) The encoded features of each group in the multi-level decoder are fed back to the deconvolution block, which contains two types of deconvolution layers: 4×4 and 16×16. After normalization, high-resolution detail features are obtained through the connection layer. The detail features are then fused into a whole feature to obtain dense depth information, as shown below:

[0034] ;

[0035] ;

[0036] In the formula, and These are high-resolution detail features obtained from RGB data and point cloud data, respectively. This represents the feature obtained by fusing detailed features from RGB data and point cloud data; Dec is a multi-level decoder; Conv is a convolutional layer.

[0037] Furthermore, the specific steps in step 5) of generating the intelligent vehicle's decision-making actions using the trained behavioral decision network are as follows:

[0038] The behavioral decision network consists of a deep neural network, including an input layer, hidden layers, normalization layers, and an output layer. The input layer takes multimodal fusion features as input, the normalization layer uses the Softmax activation function, and the output layer outputs the intelligent vehicle behavior strategy, which is the decision action.

[0039] 51) The training objective of the behavior decision network is to minimize the cross-entropy loss function, as follows:

[0040] ;

[0041] In the formula, These are the parameters of the decision network; The probability of the action output by the decision network; This is the one-hot encoded value for the output action;

[0042] 52) During training, the state-action pairs in the experience pool are randomly assigned to multiple batches, and the parameters of the behavior decision network... Directly using batch loss gradient Update, loss gradient The calculation formula is as follows:

[0043] ;

[0044] In the formula, The number of random action trajectories;

[0045] The parameters of the behavior decision network are represented by the loss gradient update process as follows:

[0046] ;

[0047] In the formula, The learning rate of the decision network; and These are the decision network parameters before and after the update in the k-th training iteration, respectively;

[0048] 53) After the network parameters are updated iteratively once, the state-action pairs in the experience pool are shuffled and randomly reassigned to new batches. At the same time, the gradient of the batch loss is further calculated and the network parameters are updated accordingly. When the loss stabilizes at a low level, the iterative training ends.

[0049] 54) The current state of the traffic scene The trained behavior decision network is input, and the intelligent vehicle outputs its decision actions, represented as follows:

[0050] ;

[0051] In the formula, For the decision-making actions of intelligent vehicles; This is the network's transfer function.

[0052] Furthermore, the specific steps of step 6) are as follows:

[0053] 61) Obtain the current state of the traffic scenario using the rule-based backup strategy. Decision-making actions and according to The current instantaneous return is used as a benchmark, and is expressed as follows:

[0054] ;

[0055] ;

[0056] In the formula, The transfer function for the rule-based backup strategy; For the best immediate return;

[0057] 62) Based on decision-making actions Sampling a decision trajectory from time t to time t+T, the cumulative reward calculation expression for the rule backup strategy during the decision-making process is as follows:

[0058] ;

[0059] In the formula, The cumulative reward for the rule-based backup strategy; For decision intervals; As a discount factor, The optimal immediate reward at time t+T;

[0060] 63) Decision actions obtained based on behavioral decision networks Sampling a decision trajectory from time t to time t+T, the corresponding cumulative reward is calculated as follows:

[0061] ;

[0062] In the formula, The cumulative reward for the behavioral decision-making network; The reward for the action at time t; The state value corresponding to the final state; for Momentary action feedback; The action reward at time t+T;

[0063] 64) Determine whether the cumulative reward of the decision action output by the behavioral decision network is greater than or equal to the baseline instantaneous reward obtained by the rule backup strategy. If it is greater, it is considered a valid and safe output; otherwise, the decision action output by the behavioral decision network will be modified based on the rule backup strategy.

[0064] The present invention also provides an end-to-end behavior decision-making system for intelligent vehicles based on multimodal information fusion, comprising:

[0065] The data acquisition module is used to acquire RGB data and point cloud data of the surrounding environment through the vehicle's onboard camera and LiDAR.

[0066] The first data processing module is used to perform distortion compensation on the RGB data in order to complete the preprocessing of the RGB data;

[0067] The second data processing module is used to perform translation and rotation operations on the cloud data, transform it to the camera coordinate system, and use the forward projection method to obtain the front view of the point cloud to complete the point cloud data preprocessing.

[0068] The data fusion module is used to perform multimodal fusion on preprocessed RGB data and point cloud data to obtain fused dense depth information;

[0069] The decision-making action output module is used to input the dense depth information as environmental perception data into the trained behavior decision network and output the decision-making action of the intelligent vehicle.

[0070] The decision optimization module is used to obtain the baseline instant reward at each decision moment and compare it with the cumulative reward of the decision action. When the cumulative reward of the decision action is lower than the baseline instant reward, the decision action is constrained and corrected based on the rule-based backup strategy.

[0071] The beneficial effects of this invention are:

[0072] This invention utilizes a deep information fusion network to acquire deep fusion information from vehicle-mounted cameras and LiDAR, employs a behavioral decision network to generate decision-making actions for the intelligent vehicle, and implements safety constraints based on rule-based backup strategies. This addresses the problem of increased decision-making risk caused by the perception limitations of a single vehicle-mounted sensor, ensuring the effectiveness of the intelligent vehicle's decisions and driving safety in complex and dynamic road environments. This invention has strong practicality and feasibility, and is conducive to promoting the development of intelligent vehicles towards higher levels of autonomous driving.

[0073] 1. This invention utilizes a deep information fusion network to extract deep fusion information from vehicle-mounted cameras and LiDAR, thereby solving the perception limitations of a single vehicle-mounted sensor and providing a global information foundation for end-to-end decision-making systems;

[0074] 2. This invention uses a behavioral decision network to generate the decision actions of the intelligent vehicle and implements safety constraints based on a rule-based backup strategy, thereby ensuring the decision-making effectiveness and driving safety of the intelligent vehicle in complex and dynamic road environments. Attached Figure Description

[0075] Figure 1 This is a flowchart of the method of the present invention;

[0076] Figure 2 This is a schematic diagram of the structural principle of the deep information fusion network in this invention;

[0077] Figure 3 This is a schematic diagram of the decision network structure in this invention. Detailed Implementation

[0078] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0079] Reference Figures 1 to 3 As shown, the present invention provides an end-to-end behavior decision-making method for intelligent vehicles based on multimodal information fusion, comprising the following steps:

[0080] Step 1): Acquire RGB data and point cloud data of the surrounding environment using the vehicle's onboard camera and LiDAR;

[0081] Step 2): Perform distortion compensation on the RGB data to complete the preprocessing of the RGB data;

[0082] In step 2), the distortion compensation process for RGB data considers both radial and tangential distortions, as shown below:

[0083] ;

[0084] In the formula, There are 5 distortion parameters; is the position coordinate of the target; r is the radial distance from the target to the optical center in the camera coordinate system; The corrected position coordinates of the perceived target.

[0085] Step 3): Perform translation and rotation operations on the cloud data, transform it to the camera coordinate system, and use the forward projection method to obtain the front view of the point cloud to complete the point cloud data preprocessing;

[0086] The specific steps for preprocessing cloud data in step 3) are as follows:

[0087] 31) Define the coordinate system of the lidar as O L The camera coordinate system is defined as O C The spatial coordinates of the same perceived target T in the two coordinate systems are denoted as follows: and The three-dimensional spatial transformation of target T is represented as follows:

[0088] ;

[0089] In the formula, T L For O L A point in space within a coordinate system; T C For O C A point in space within a coordinate system; R C S is a spatial rotation matrix; C It is a translation matrix;

[0090] 32) Rotate the spatial rotation matrix RC Decompose the rotation into counterclockwise angular variables about the X, Y, and Z axes, resulting in three rotation matrices. , and The expression is as follows:

[0091] ;

[0092] In the formula, , and These are the rotation angles around the X-axis, Y-axis, and Z-axis, respectively.

[0093] Rotating a point in space in the order of the X, Y, and Z axes, and multiplying the three rotation matrices together to represent the transformation of the three-dimensional spatial state, is as follows:

[0094] ;

[0095] 33) Applying the translation matrix S C Translate a point in space from point O L Coordinate system transformation to O C The coordinate system is represented as follows:

[0096] ;

[0097] 34) In point cloud data from O L Coordinate system transformation to O C After establishing the coordinate system, the point cloud data is projected forward, and the coordinates of each spatial point in O are calculated. C The projection position in the coordinate system is determined, and its position is checked to see if it overflows the image boundary. If it does, it is cropped accordingly.

[0098] Step 4): Perform multimodal fusion on the preprocessed RGB data and point cloud data from Steps 2) and 3) to obtain fused dense depth information;

[0099] In step 4), the multimodal fusion of RGB data and point cloud data is performed using a deep information fusion network. The specific steps are as follows:

[0100] 41) Global context information is obtained from RGB data and point cloud data using a global transform encoder, and the encoded features are represented as follows:

[0101] ;

[0102] In the formula, and These are global encoding features obtained from RGB data and point cloud data, respectively; and is the input to the encoder; n is the number of spatial points; Enc is the global transform encoder; The encoded feature of the nth spatial point in RGB data; The encoded feature of the nth spatial point in the point cloud data;

[0103] 42) The encoded features are aggregated bidirectionally using both top-down and bottom-up paths, and the encoder output is divided into four groups. The outputs of the last block of the first, second, third, and fourth groups are respectively represented as follows: , , and The coding features it constitutes As input to a multi-level decoder;

[0104] 43) The encoded features of each group in the multi-level decoder are fed back to the deconvolution block, which contains two types of deconvolution layers: 4×4 and 16×16. After normalization, high-resolution detail features are obtained through the connection layer. The detail features are then fused into a whole feature to obtain dense depth information, as shown below:

[0105] ;

[0106] ;

[0107] In the formula, and These are high-resolution detail features obtained from RGB data and point cloud data, respectively. This represents the feature obtained by fusing detailed features from RGB data and point cloud data; Dec is a multi-level decoder; Conv is a convolutional layer.

[0108] Step 5): Input the dense depth information as environmental perception data into the trained behavior decision network and output the intelligent vehicle's decision action;

[0109] The specific steps in step 5) of generating the intelligent vehicle's decision-making actions using the trained behavioral decision network are as follows:

[0110] The behavioral decision network consists of a deep neural network, including an input layer, hidden layers, normalization layers, and an output layer. The input layer takes multimodal fusion features as input, the normalization layer uses the Softmax activation function, and the output layer outputs the intelligent vehicle behavior strategy, which is the decision action.

[0111] 51) The training objective of the behavior decision network is to minimize the cross-entropy loss function, as follows:

[0112] ;

[0113] In the formula, These are the parameters of the decision network; The probability of the action output by the decision network; This is the one-hot encoded value for the output action;

[0114] 52) During training, the state-action pairs in the experience pool are randomly assigned to multiple batches, and the parameters of the behavior decision network... Directly using batch loss gradient Update, loss gradient The calculation formula is as follows:

[0115] ;

[0116] In the formula, The number of random action trajectories;

[0117] The parameters of the behavior decision network are represented by the loss gradient update process as follows:

[0118] ;

[0119] In the formula, The learning rate of the decision network; and These are the decision network parameters before and after the update in the k-th training iteration, respectively;

[0120] 53) After the network parameters are updated iteratively once, the state-action pairs in the experience pool are shuffled and randomly reassigned to new batches. At the same time, the gradient of the batch loss is further calculated and the network parameters are updated accordingly. When the loss stabilizes at a low level, the iterative training ends.

[0121] 54) The current state of the traffic scene (That is, environmental information obtained from onboard sensors, including the position and speed of the vehicle and surrounding vehicles), inputted into the trained behavior decision network, outputting the intelligent vehicle's decision action, represented as:

[0122] ;

[0123] In the formula, For the decision-making actions of intelligent vehicles; This is the network's transfer function.

[0124] Step 6): Obtain the baseline instantaneous return for each decision moment and compare it with the cumulative return of the decision action in Step 5). When the cumulative return of the decision action is lower than the baseline instantaneous return, constrain and correct the decision action based on the rule-based backup strategy; the specific steps are as follows:

[0125] 61) Obtain the current state of the traffic scenario using the rule-based backup strategy. Decision-making actions and according to The current instantaneous return is used as a benchmark, and is expressed as follows:

[0126] ;

[0127] ;

[0128] In the formula, The transfer function for the rule-based backup strategy; For the best immediate return;

[0129] 62) Based on decision-making actions Sampling a decision trajectory from time t to time t+T, the cumulative reward calculation expression for the rule backup strategy during the decision-making process is as follows:

[0130] ;

[0131] In the formula, The cumulative reward for the rule-based backup strategy; For decision intervals; As a discount factor, The optimal immediate reward at time t+T;

[0132] 63) Decision actions obtained based on behavioral decision networks Sampling a decision trajectory from time t to time t+T, the corresponding cumulative reward is calculated as follows:

[0133] ;

[0134] In the formula, The cumulative reward for the behavioral decision-making network; The reward for the action at time t; The state value corresponding to the final state; for Momentary action feedback; The action reward at time t+T;

[0135] 64) Determine whether the cumulative reward of the decision action output by the behavioral decision network is greater than or equal to the baseline instantaneous reward obtained by the rule backup strategy. If it is greater, it is considered a valid and safe output; otherwise, the decision action output by the behavioral decision network will be modified based on the rule backup strategy.

[0136] The present invention provides an end-to-end intelligent vehicle behavior decision-making system based on multimodal information fusion, comprising:

[0137] The data acquisition module is used to acquire RGB data and point cloud data of the surrounding environment through the vehicle's onboard camera and LiDAR.

[0138] The first data processing module is used to perform distortion compensation on the RGB data in order to complete the preprocessing of the RGB data;

[0139] The second data processing module is used to perform translation and rotation operations on the cloud data, transform it to the camera coordinate system, and use the forward projection method to obtain the front view of the point cloud to complete the point cloud data preprocessing.

[0140] The data fusion module is used to perform multimodal fusion on preprocessed RGB data and point cloud data to obtain fused dense depth information;

[0141] The decision-making action output module is used to input the dense depth information as environmental perception data into the trained behavior decision network and output the decision-making action of the intelligent vehicle.

[0142] The decision optimization module is used to obtain the baseline instant reward at each decision moment and compare it with the cumulative reward of the decision action. When the cumulative reward of the decision action is lower than the baseline instant reward, the decision action is constrained and corrected based on the rule-based backup strategy.

[0143] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered within the scope of protection of this invention.

Claims

1. An end-to-end behavior decision-making method for intelligent vehicles based on multimodal information fusion, characterized in that, The steps are as follows: Step 1): Acquire RGB data and point cloud data of the surrounding environment using the vehicle's onboard camera and LiDAR; Step 2): Perform distortion compensation on the RGB data; Step 3): Perform translation and rotation operations on the cloud data, transform it to the camera coordinate system, and use the forward projection method to obtain the front view of the point cloud; Step 4): Perform multimodal fusion on the preprocessed RGB data and point cloud data from Steps 2) and 3) to obtain fused dense depth information; Step 5): Input the dense depth information as environmental perception data into the trained behavior decision network and output the intelligent vehicle's decision action; Step 6): Obtain the baseline instant reward at each decision moment and compare it with the cumulative reward of the decision action in Step 5). When the cumulative reward of the decision action is lower than the baseline instant reward, constrain and correct the decision action based on the rule-based backup strategy.

2. The intelligent vehicle end-to-end behavior decision-making method based on multimodal information fusion according to claim 1, characterized in that, The distortion compensation process for RGB data in step 2) considers both radial and tangential distortions, as shown below: ； In the formula, There are 5 distortion parameters; is the position coordinate of the target; r is the radial distance from the target to the optical center in the camera coordinate system; The corrected position coordinates of the perceived target.

3. The intelligent vehicle end-to-end behavior decision-making method based on multimodal information fusion according to claim 2, characterized in that, The specific steps for preprocessing cloud data in step 3) are as follows: 31) Define the coordinate system of the lidar as O L The camera coordinate system is defined as O C The spatial coordinates of the same perceived target T in the two coordinate systems are denoted as follows: and The three-dimensional spatial transformation of target T is represented as follows: ； In the formula, T L For O L A point in space within a coordinate system; T C For O C A point in space within a coordinate system; R C S is a spatial rotation matrix; C It is a translation matrix; 32) Rotate the spatial rotation matrix R C Decompose the rotation into counterclockwise angular variables about the X, Y, and Z axes, resulting in three rotation matrices. , and The expression is as follows: ； In the formula, , and These are the rotation angles around the X-axis, Y-axis, and Z-axis, respectively. Rotating a point in space in the order of the X, Y, and Z axes, and multiplying the three rotation matrices together to represent the transformation of the three-dimensional spatial state, is as follows: ； 33) Applying the translation matrix S C Translate a point in space from point O L Coordinate system transformation to O C The coordinate system is represented as follows: ； 34) In point cloud data from O L Coordinate system transformation to O C After establishing the coordinate system, the point cloud data is projected forward, and the coordinates of each spatial point in O are calculated. C The projection position in the coordinate system is determined, and its position is checked to see if it overflows the image boundary. If it does, it is cropped accordingly.

4. The intelligent vehicle end-to-end behavior decision-making method based on multimodal information fusion according to claim 3, characterized in that, In step 4), a deep information fusion network is used to perform multimodal fusion of RGB data and point cloud data. The specific steps are as follows: 41) Global context information is obtained from RGB data and point cloud data using a global transform encoder, and the encoded features are represented as follows: ； In the formula, and These are global encoding features obtained from RGB data and point cloud data, respectively; and is the input to the encoder; n is the number of spatial points; Enc is the global transform encoder; The encoded feature of the nth spatial point in RGB data; The encoded feature of the nth spatial point in the point cloud data; 42) The encoded features are aggregated bidirectionally using both top-down and bottom-up paths, and the encoder output is divided into four groups. The outputs of the last block of the first, second, third, and fourth groups are respectively represented as follows: , , and The coding features it constitutes As input to a multi-level decoder; 43) The encoded features of each group in the multi-level decoder are fed back to the deconvolution block, which contains two types of deconvolution layers: 4×4 and 16×16. After normalization, high-resolution detail features are obtained through the connection layer. The detail features are then fused into a whole feature to obtain dense depth information, as shown below: ； ； In the formula, and These are high-resolution detail features obtained from RGB data and point cloud data, respectively. This refers to the feature obtained by fusing detailed features from RGB data and point cloud data; Dec is a multi-level decoder; Conv is a convolutional layer.

5. The intelligent vehicle end-to-end behavior decision-making method based on multimodal information fusion according to claim 1, characterized in that, The specific steps in step 5) of generating the intelligent vehicle's decision-making actions using the trained behavior decision network are as follows: The behavioral decision network consists of a deep neural network, including an input layer, hidden layers, normalization layers, and an output layer. The input layer takes multimodal fusion features as input, the normalization layer uses the Softmax activation function, and the output layer outputs the intelligent vehicle behavior strategy, which is the decision action. 51) The training objective of the behavior decision network is to minimize the cross-entropy loss function, as follows: ； In the formula, These are the parameters of the decision network; The probability of the action output by the decision network; This is the one-hot encoded value for the output action; 52) During training, the state-action pairs in the experience pool are randomly assigned to multiple batches, and the parameters of the behavior decision network... Directly using batch loss gradient Update, loss gradient The calculation formula is as follows: ； In the formula, The number of random action trajectories; The parameters of the behavior decision network are represented by the loss gradient update process as follows: ； In the formula, The learning rate of the decision network; and These are the decision network parameters before and after the update in the k-th training iteration, respectively; 53) After the network parameters are updated iteratively once, the state-action pairs in the experience pool are shuffled and randomly reassigned to new batches. At the same time, the gradient of the batch loss is further calculated and the network parameters are updated accordingly. When the loss stabilizes at a low level, the iterative training ends. 54) The current state of the traffic scene The trained behavior decision network is input, and the intelligent vehicle outputs its decision actions, represented as follows: ； In the formula, For the decision-making actions of intelligent vehicles; This is the network's transfer function.

6. The intelligent vehicle end-to-end behavior decision-making method based on multimodal information fusion according to claim 5, characterized in that, The specific steps of step 6) are as follows: 61) Obtain the current state of the traffic scenario using the rule-based backup strategy. Decision-making actions and according to The current instantaneous return is used as a benchmark, and is expressed as follows: ； ； In the formula, The transfer function for the rule-based backup strategy; For the best immediate return; 62) Based on decision-making actions Sampling a decision trajectory from time t to time t+T, the cumulative reward calculation expression for the rule backup strategy during the decision-making process is as follows: ； In the formula, The cumulative reward for the rule-based backup strategy; For decision intervals; As a discount factor, The optimal immediate reward at time t+T; 63) Decision actions obtained based on behavioral decision networks Sampling a decision trajectory from time t to time t+T, the corresponding cumulative reward is calculated as follows: ； In the formula, The cumulative reward for the behavioral decision-making network; The reward for the action at time t; The state value corresponding to the final state; for Momentary action feedback; The action reward at time t+T; 64) Determine whether the cumulative reward of the decision action output by the behavioral decision network is greater than or equal to the baseline immediate reward obtained by the rule-based backup strategy. If it is greater, it is considered a valid and safe output. Otherwise, the rule-based backup strategy will modify the decision actions output by the behavioral decision network.

7. An end-to-end intelligent vehicle behavior decision-making system based on multimodal information fusion, characterized in that, include: The data acquisition module is used to acquire RGB data and point cloud data of the surrounding environment through the vehicle's onboard camera and LiDAR. The first data processing module is used to perform distortion compensation on the RGB data in order to complete the preprocessing of the RGB data; The second data processing module is used to perform translation and rotation operations on the cloud data, transform it to the camera coordinate system, and use the forward projection method to obtain the front view of the point cloud to complete the point cloud data preprocessing. The data fusion module is used to perform multimodal fusion on preprocessed RGB data and point cloud data to obtain fused dense depth information; The decision-making action output module is used to input the dense depth information as environmental perception data into the trained behavior decision network and output the decision-making action of the intelligent vehicle. The decision optimization module is used to obtain the baseline instant reward at each decision moment and compare it with the cumulative reward of the decision action. When the cumulative reward of the decision action is lower than the baseline instant reward, the decision action is constrained and corrected based on the rule-based backup strategy.

Images