Cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation

Abstract

The present invention belongs to the field of deep reinforcement learning autonomous driving and involves vehicle safety decision-making technology. Specifically, it is a cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation. It first constructs a multimodal spatiotemporal perception encoder to jointly model spatial and motion information from multimodal continuous input to obtain the current perception representation of the dynamic driving scene; then, it introduces a future prediction encoder to capture the interaction between different traffic participants from the current perception representation to obtain the future prediction representation; then, the current perception representation and the future prediction representation are connected to form a multimodal spatiotemporal representation and used as the state input of reinforcement learning to fully grasp the scene. In combination with the distributed PPO algorithm, the safety decision-making task is achieved under the guidance of the reward function designed for safety decision-making. The present invention has high environmental adaptability and decision-making success rate, and can achieve active safety decision-making tasks for smart cars in dense traffic scenes and emergencies.

Description

Technical Field

[0001] The present invention belongs to the field of deep reinforcement learning autonomous driving and relates to vehicle safety decision-making technology, specifically to a cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation. Background Art

[0002] In the field of autonomous driving, ensuring that vehicles can make safe decisions in a wide variety of driving scenarios is crucial, as it directly impacts the safety of drivers and passengers. Traditional autonomous driving systems primarily utilize a modular approach, where each function, such as perception, prediction, and decision-making, is developed and integrated into the system separately. The most common decision-making approach within this modular approach is rule-based, which is generally ineffective for the vast array of situations that arise during driving. Consequently, existing approaches primarily focus on data-based learning strategies, such as imitation learning and deep reinforcement learning, to achieve safe decision-making.

[0003] A safe decision-making approach for autonomous driving based on deep reinforcement learning (DRL) represents long-sequence driving tasks as a Markov decision process. Through continuous interaction with the environment, intelligent vehicles learn driving policies guided by a reward function, resulting in adaptive optimal decisions based on current state observations. This approach allows intelligent vehicles to optimize their decision-making through trial and error, without relying on manually designed rules or human driving data. Current deep reinforcement learning autonomous driving approaches fall into two main categories: end-to-end and decoupled approaches. End-to-end approaches directly learn the mapping from raw sensor data to control commands. Because sensor data is often complex and high-dimensional, containing noise and redundant information, deep networks are required to learn good driving policies. However, the gradients generated by DRL are often insufficient to effectively train deep neural networks, making the training process difficult. Decoupled approaches typically divide the autonomous driving system into two components: perception and decision-making. First, the perception component uses supervised learning to train a deep network to understand the environment and generate an intermediate representation. Then, the decision-making component uses reinforcement learning to train a shallower network to learn the driving policy from the intermediate representation. There are two main approaches. One is to train the perception model to directly map raw observations to custom perception results as reinforcement learning states, such as relevant perception indicators (the angle of the car relative to the road, the distance to the lane markings, etc.) or a semantic segmentation mask that parses the entire scene. The other is to use some auxiliary task heads to train the perception encoder to obtain driving-related latent features from the raw observations, which are then fed into the reinforcement learning network to decode the optimal driving policy. The former transfers custom results between perception and decision-making. As the sequential process proceeds, it is easy to cause error accumulation and information loss, thereby limiting the performance of the decision. The latter solves this problem by transferring feature representations and has received widespread attention.

[0004] Currently, methods that use latent features as reinforcement learning states lack safety in high-traffic density scenarios involving a large number of dynamic objects, especially in rare emergencies. There are many factors that contribute to this safety issue, two of which are the main flaws: 1) Lack of comprehensive scene perception. A single sensor usually cannot provide enough information to perceive the driving scene. Single-image methods cannot provide accurate 3D information of the scene, and single-lidar methods cannot provide semantic information. 2) Lack of temporal dimension information of the traffic scene. It can be seen that for dynamic driving scenes, not only spatial information should be captured, but also the dynamic variability between continuous inputs. In addition, predicting the future behavior of surrounding traffic participants is also crucial for autonomous vehicles to make safe and reliable decisions. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose a cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation. Its perception and prediction encoder not only considers information in the spatial dimension but also introduces information in the temporal dimension, achieving a comprehensive understanding of dynamic scenes, thereby improving driving safety and better meeting the needs of practical applications.

[0006] In order to achieve the above object, the technical solutions specifically adopted by the present invention are as follows:

[0007] A cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation first constructs a multimodal spatiotemporal perception encoder to jointly model spatial and motion information from multimodal continuous input to obtain the current perception representation of dynamic driving scenarios. Then, a future prediction encoder is introduced to capture the interactions between different traffic participants from the current perception representation to obtain the future prediction representation. Then, the current perception representation and the future prediction representation are connected to form a multimodal spatiotemporal representation and used as the state input of reinforcement learning to fully grasp the scenario. Combined with the distributed PPO algorithm, the safety decision-making task is achieved under the guidance of the reward function designed for safety decision-making. Specifically, the following steps are included:

[0008] S1. Collection and preprocessing of raw sensor data. The raw sensor data includes the current forward-looking RGB color image, LiDAR point cloud, speed data, and data on the distance and angle of deviation from the lane center. For the LiDAR point cloud, the LiDAR point cloud from the past five frames is first realigned to the current vehicle coordinate system. The point clouds from these six consecutive frames are then voxelized into a 2D BEV grid with a fixed resolution. Finally, these grids are concatenated to obtain a six-channel LiDAR BEV projected pseudo image. The speed data and the distance and angle of deviation from the lane center are normalized.

[0009] S2. Combine spatial perception and motion perception with multi-task head supervision to train a multimodal spatiotemporal perception encoder to extract the current perception representation from a single frame of forward-looking RGB color image and six consecutive frames of lidar BEV projection pseudo images. The network of the multimodal spatiotemporal perception encoder consists of an image feature extraction backbone network, a lidar BEV feature extraction backbone network, a multimodal feature fusion network, and a multi-task head network.

[0010] S3. Train the future prediction encoder to learn the lidar BEV features f output from the multimodal spatiotemporal perception encoder lidar_fusion The network of the future prediction encoder is composed of a position attention network, a channel attention network, an attention fusion network and a future prediction task head network.

[0011] S4. After completing the supervised training of steps S2 and S3, design a reward function and use the distributed PPO reinforcement learning algorithm to train the deep reinforcement learning decision model to learn the optimal safety decision-making strategy from the multimodal spatiotemporal representation composed of the lidar BEV features and future prediction features, as well as the speed data, the deviation distance from the lane center, and the deviation angle data.

[0012] Furthermore, the image feature extraction backbone network extracts features through four residual convolution blocks of a ResNet-34 network pre-trained on ImageNet to obtain four image features with different levels of information. s i Indicates different feature extraction stages.

[0013] Furthermore, the LiDAR BEV feature extraction backbone network takes six consecutive frames of six-channel LiDAR BEV projection pseudo images as input, and extracts features through four spatiotemporal convolution blocks of a VideoResnet-18 network that introduces a spatiotemporal convolution structure to obtain four LiDAR BEV features with different levels of information. where s i Indicates different feature extraction stages.

[0014] Furthermore, the multimodal feature fusion network is used to fuse the image features of four different scales and the laser radar BEV features of four different scales. When fusion is performed, the image features extracted by the backbone network of the two branches are firstly combined. and LiDAR BEV features After dimension reshaping, the sequence vector is obtained by concatenation Then Through a multimodal fusion Transformer module, full information interaction between different modal features is achieved to obtain the global spatiotemporal context features in the 3D scene. Finally, Slice and restore to and Features of the same dimension, and Add elements to obtain fused image features and LiDAR BEV features

[0015] Furthermore, in order to capture the mutual relationship between different branch tasks and enhance the feature expression in different branches, the image branch and the lidar BEV branch are supervised by different multi-task heads respectively; the image branch is supervised by H dep and H sem The BEV point cloud branch consists of two task heads, namely depth estimation and semantic segmentation of the front view image; cross entropy loss is used for front view semantic segmentation, and L1 loss is used to supervise the front view depth estimation task; the BEV point cloud branch consists of task head H bev , H v and H bb It is composed of three parts, which are used for BEV semantic segmentation, surrounding vehicle speed prediction and 2D target detection respectively; cross entropy loss is used for BEV semantic segmentation, L2 loss is used to supervise surrounding vehicle speed prediction, and 2D target detection uses CenterNet decoder to locate other traffic participants in the scene.

[0016] Furthermore, in the step S3, the position attention network combines the multimodal spatiotemporal fusion of the laser radar BEV feature f lidar_fusion Input into three convolutional layers respectively to obtain three feature maps of the same dimension as the original, and then adjust their dimensions to three two-dimensional features of the same dimension and Then, in The transpose and Perform matrix multiplication between and apply the SoftMax layer to calculate the spatial attention map s lo ; then in s lo and Perform matrix multiplication between the transposes of to capture the spatial dependency between any two locations in the feature map and resize the result to get the same value as f lidar_fusion Feature maps of the same dimension Finally and f lidar_fusion Add the elements to get the final output of the position attention network

[0017] Furthermore, in the step S3, the channel attention network combines the multimodal spatiotemporal fusion of the laser radar BEV feature f lidar_fusion Perform dimension adjustment to obtain two two-dimensional features of the same dimension Then The transpose and Perform matrix multiplication between and apply a SoftMax layer to obtain the channel attention map s ch ; then in s ch and f lidar_fusion Perform matrix multiplication between the transposes of to capture the channel dependency between any two channels and reshape the result to get the same value as f lidar_fusion Feature maps of the same dimension Finally and f lidar_fusion Add the elements to get the final output of the channel attention network

[0018] Furthermore, in step S3, the attention fusion network combines the outputs of the position attention network and the channel attention network and After element addition and convolution operations, the future prediction feature f is obtained future ; The future prediction task head network predicts features f from the future future The scene state 0.5 seconds into the future is analyzed, that is, the position and speed prediction of other traffic participants in the current scene 0.5 seconds later.

[0019] Furthermore, in step S4, the reward function includes a sparse reward obtained by triggering a predefined event and a dense reward obtained for each timestamp; the sparse rewards include six types, namely, collision, unexplained parking, speeding, deviation distance greater than a threshold, deviation angle greater than a threshold, and deviation angular velocity greater than a threshold, and penalties are imposed when the above conditions are met; the dense rewards include four types, namely, deviation distance reward, deviation angle reward, angular velocity reward, and speed reward.

[0020] Furthermore, in step S4, a multi-branch network structure with horizontal and vertical separation is adopted. For each navigation command in a set of high-level navigation commands (turn left at the intersection, turn right at the intersection, go straight at the intersection, and drive along the current road), an independent action prediction branch is adopted. The navigation command acts as a switch to select which branch to use at each timestamp, and each branch learns a sub-strategy specific to its navigation command. In each branch, considering that vehicle motion control involves two relatively independent operations: lateral control and longitudinal control, two identical branches are designed to handle lateral control and longitudinal control respectively. The network structures of lateral control and longitudinal control are both composed of a policy network and a value network.

[0021] The strategy network combines the multimodal spatiotemporal fusion of the lidar BEV feature map f lidar_fusion and future prediction feature map f future The channel splicing operation is performed to obtain the feature map f, which is then flattened into a one-dimensional feature vector after four layers of convolution. Finally, the feature vector is connected with the vehicle measurement vector containing speed data, deviation distance from the lane center, and deviation angle data, and then passed through three layers of fully connected layers to output the d-dimensional discrete action prediction probability P(s t );

[0022] The value network combines the multi-modal spatiotemporal fusion of the lidar BEV feature map f lidar_fusion and future prediction feature map f future The channel splicing operation is performed to obtain the feature map f, which is then flattened into a one-dimensional feature vector after four layers of convolution. Finally, the feature vector is connected with the vehicle measurement vector containing speed data, deviation distance from the lane center, and deviation angle data, and then passed through three layers of fully connected layers to output a one-dimensional state value prediction Q(s t ).

[0023] The present invention has the following characteristics and beneficial effects:

[0024] The present invention adopts a cascaded deep reinforcement learning active safety decision-making method based on multimodal spatiotemporal representation to guide the vehicle to learn active safety decision-making skills through a reward function. The designed multimodal spatiotemporal perception encoder jointly models spatial and motion information, providing the current perception representation of the scene for subsequent decision-making, ensuring the richness of the information required for decision-making; the designed future prediction encoder captures the interaction between different traffic participants, providing future prediction representation for subsequent decision-making, and improving the decision success rate. The current perception representation and future prediction representation form a comprehensive understanding of the scene, ensuring the safety of the decision-making process, and have high environmental adaptability and decision success rate. After learning the decision strategy, this method can realize the safe decision-making tasks of intelligent vehicles in dense traffic scenes and emergencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0026] Figure 1 This is an algorithm flow chart of the cascaded deep reinforcement learning security decision-making method based on multimodal spatiotemporal representation in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention is described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0028] like Figure 1 As shown in FIG, a cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation includes the following steps:

[0029] Step (1): Collection and preprocessing of raw sensor data. Specifically: Collect the current forward RGB color image, lidar point cloud, speed data, and deviation distance and deviation angle data from the lane center; for the lidar point cloud, first realign the lidar point cloud of the past five frames to the vehicle coordinate system at the current moment, then voxelize the point cloud of these six consecutive frames into a 2D BEV grid with a fixed resolution, and finally connect them to obtain a six-channel lidar BEV projection pseudo image. For the 2D BEV grid, consider the position of 32 meters in front and behind the vehicle and 32 meters on both sides, and divide the 64m × 64m distance range into multiple areas of 0.25m × 0.25m, so as to obtain a grid with a resolution of 256 × 256. At the same time, delete the point cloud data with a height less than 0.2m to remove the ground plane. For the speed data and the deviation distance and deviation angle data from the lane center, normalization is performed.

[0030] Step (2): Combine the multi-task head supervision of spatial perception and motion perception to train a multimodal spatiotemporal perception encoder to extract the current perception representation from a single frame of forward-looking RGB image and six consecutive frames of laser radar BEV projection pseudo images. The multimodal spatiotemporal perception encoder network consists of an image feature extraction backbone network, a laser radar BEV feature extraction backbone network, a multimodal feature fusion network, and a multi-task head network. During extraction, the forward-looking RGB image of the current frame is input into the image feature extraction backbone network, and four residual convolution blocks of a ResNet-34 network pre-trained on ImageNet are used to extract features to obtain four image features with different levels of information. where s i Indicates different feature extraction stages. Six consecutive frames of six-channel LiDAR BEV projected pseudo images are input into the LiDAR BEV feature extraction backbone network. Four LiDAR BEV features with different levels of information are extracted through four spatiotemporal convolution blocks of a VideoResnet-18 network that introduces a spatiotemporal convolution structure. Among them, s iRepresents different feature extraction stages. Then, the image features of four different scales and the lidar BEV features of four different scales are fused separately through the multimodal feature fusion network. After four rounds of feature fusion, the final image feature f is obtained. rgb_fusion and LiDAR BEV features f lidar_fusion . With the s i Taking the stage as an example, we first extract the image features from the backbone network of the two branches. and LiDAR BEV features After dimension reshaping, the sequence vector is obtained by concatenation Then Through a multimodal fusion Transformer module, full information interaction between different modal features is achieved to obtain the global spatiotemporal context features in the 3D scene. Finally, Slice and restore to and Features of the same dimension, and Add elements to obtain fused image features and LiDAR BEV features

[0031] The operation steps of the multimodal fusion Transformer are as follows:

[0032]

[0033] Among them, LN1(·) and LN2(·) represent Layer Normalization operations, MHA(·) represents the multi-head self-attention layer, and FFN(·) represents the feedforward neural network layer.

[0034] In the multimodal fusion Transformer module, the sequence vector and one with Learnable position embeddings of the same dimension Add the parameter elements to get the position-encoded sequence vector Will After passing through a normalization layer LN1 and a multi-head self-attention layer MHA, Add the elements to realize the attention interaction of the features of the two modalities then After a normalization layer LN2 and a feedforward network layer FFN and The final global spatiotemporal context feature output is obtained by adding the elements

[0035] In this embodiment, the image branch and the lidar BEV branch use different multi-task heads for supervised training, capturing the relationship between different branch tasks and enhancing the feature expression in different branches. dep and H sem The BEV point cloud branch consists of three task heads: depth estimation and semantic segmentation of the front view image. We use cross entropy loss for front view semantic segmentation and L1 loss to supervise the front view depth estimation task. bev , H v and H bb The network is composed of three parts, which are used for BEV semantic segmentation, surrounding vehicle speed prediction, and 2D object detection. We use cross entropy loss for BEV semantic segmentation, L2 loss to supervise surrounding vehicle speed prediction, and CenterNet decoder for 2D object detection to locate other traffic participants in the scene.

[0036] Step (3): Train the future prediction encoder to learn the lidar BEV features f output from the multimodal spatiotemporal perception encoder lidar_fusion The relationship between traffic participants is captured to obtain the future prediction representation of dynamic driving scenes. The future prediction encoder network consists of a position attention network, a channel attention network, an attention fusion network, and a future prediction task head network; among them:

[0037] (I) Position Attention Network

[0038] The lidar BEV feature f after multi-modal spatiotemporal fusion lidar_fusion Input into three convolutional layers respectively to obtain three feature maps of the same dimension as the original, and then adjust their dimensions to three two-dimensional features of the same dimension and Then, in The transpose and Perform matrix multiplication between and apply the SoftMax layer to calculate the spatial attention map s lo Then in s lo and Perform matrix multiplication between the transposes of to capture the spatial dependency between any two locations in the feature map and resize the result to get the same value as f lidar_fusion Feature maps of the same dimension Finally and f lidar_fusion Add the elements to get the final output of the position attention network

[0039] (II) Channel Attention Network

[0040] The lidar BEV feature f after multi-modal spatiotemporal fusion lidar_fusionPerform dimension adjustment to obtain two two-dimensional features of the same dimension Then The transpose and Perform matrix multiplication between and apply a SoftMax layer to obtain the channel attention map s ch . Then in s ch and f lidar_fusion Perform matrix multiplication between the transposes of to capture the channel dependency between any two channels and reshape the result to get the same value as f lidar_fusion Feature maps of the same dimension Finally and f lidar_fusion Add the elements to get the final output of the channel attention network

[0041] (III) Attention Fusion Network

[0042] In order to further enhance the feature representation, the output of the position attention network and the channel attention network are and After element addition and convolution operations, the future prediction feature f is obtained future .

[0043] (IV) Future Prediction Task Head Network

[0044] The future prediction task head predicts features f from the future future The scene state 0.5 seconds into the future is analyzed, that is, the position and speed prediction of other traffic participants in the current scene 0.5 seconds later.

[0045] Step (4): After completing the supervised training of steps (2) and (3), design a reward function and use the distributed PPO reinforcement learning algorithm to train the deep reinforcement learning decision model to learn the optimal safety decision strategy from the multimodal spatiotemporal representation composed of the lidar BEV features and future prediction features, as well as the speed data, the deviation distance from the lane center, and the deviation angle data.

[0046] In this embodiment, the reward function includes sparse rewards obtained by triggering predefined events and dense rewards obtained at each timestamp. There are six types of sparse rewards, namely collision, unexplained parking, speeding, deviation distance greater than a threshold, deviation angle greater than a threshold, and deviation angular velocity greater than a threshold. Penalties are given when the above conditions are met. There are four types of dense rewards, namely deviation distance reward, deviation angle reward, angular velocity reward, and speed reward. Among them:

[0047] The deviation distance bonus is calculated as follows:

[0048] r d =max(k d×(1-d),k d ×(1-d max ))

[0049] Where d is the deviation distance from the center of the road, d max is the maximum threshold set to 2, k d It's the size of the scale.

[0050] Deviation angle rewards are as follows:

[0051] r θ =max(k θ ×(30-θ),k θ ×(30-θ max ))

[0052] Where θ is the course deviation angle, θ max is the maximum threshold set to 60, k θ It's the size of the scale.

[0053] Angular velocity bonuses are as follows:

[0054] r ω =max(k ω ×(10-ω),k ω ×(10-ω max ))

[0055] Where ω is the angular velocity of the vehicle, ω max is the maximum threshold set to 20, k ω It's the size of the scale.

[0056] The speed bonuses are as follows:

[0057]

[0058] Where v is the vehicle speed, k v is the scale, v target is the target speed, range v is the permissible speed range. v target and range v It is calculated based on the distance l to the obstacle in front.

[0059] In this embodiment, the reinforcement learning network design employs a multi-branch, horizontally and vertically separated network structure. For each high-level navigation command (turn left at the intersection, turn right at the intersection, go straight at the intersection, and follow the current road), an independent action prediction branch is employed. The navigation command acts as a selector for which branch to use at each timestamp, and each branch learns a sub-policy specific to its navigation command. Within each branch, considering that vehicle motion control involves two relatively independent operations: lateral control and longitudinal control, two identical branches are designed to handle lateral and longitudinal control, respectively. Both the lateral and vertical control network structures consist of a policy network and a value network.

[0060] (i) Policy Network

[0061] The lidar BEV feature map f after multi-modal spatiotemporal fusion lidar_fusion and future prediction feature map f future The channel splicing operation is performed to obtain the feature map f, which is then flattened into a one-dimensional feature vector after four layers of convolution. Finally, the feature vector is connected with the vehicle measurement vector containing speed data, deviation distance from the lane center, and deviation angle data, and then passed through three layers of fully connected layers to output the d-dimensional discrete action prediction probability P(s t ). For the lateral action space, we evenly discretize the turning space into 33 parts, so d is 33, and select the action a with the largest probability st Execution; For the longitudinal action space, we combine the throttle space and the brake space and discretize them into three different actions, corresponding to acceleration, idling and deceleration, so d in the longitudinal action space is 3, and the action with the highest probability is selected. lo implement.

[0062] (ii) Value Network

[0063] The lidar BEV feature map f after multi-modal spatiotemporal fusion lidar_fusion and future prediction feature map f future The channel splicing operation is performed to obtain the feature map f, which is then flattened into a one-dimensional feature vector after four layers of convolution. Finally, the feature vector is connected with the vehicle measurement vector containing speed data, deviation distance from the lane center, and deviation angle data, and then passed through three layers of fully connected layers to output a one-dimensional state value prediction Q(s t ). Among them, the state value of the horizontal control network output is recorded as v st , the state value of the vertical control network output is recorded as v lo .

[0064] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation, characterized by: The following steps are involved: S1. Collection and preprocessing of raw sensor data; raw sensor data includes the current forward RGB color image, LiDAR point cloud, speed data, and deviation distance and angle from the lane center. For the lidar point cloud, the lidar point cloud of the past five frames is first realigned to the current vehicle coordinate system. Then, the point cloud of these six consecutive frames is voxelized into a 2D BEV grid with a fixed resolution. Finally, they are concatenated to obtain a six-channel lidar BEV projection pseudo image. The speed data and the deviation distance and deviation angle data from the lane center are normalized. S2. Combine spatial perception and motion perception with multi-task head supervision to train a multimodal spatiotemporal perception encoder to extract the current perception representation from a single frame of forward-looking RGB color image and six consecutive frames of lidar BEV projection pseudo images. The network of the multimodal spatiotemporal perception encoder consists of an image feature extraction backbone network, a lidar BEV feature extraction backbone network, a multimodal feature fusion network, and a multi-task head network. S3. Train the future prediction encoder to learn the lidar BEV features f output from the multimodal spatiotemporal perception encoder lidar_fusion The network of the future prediction encoder is composed of a position attention network, a channel attention network, an attention fusion network and a future prediction task head network. S4. After completing the supervised training of steps S2 and S3, design a reward function and use the distributed PPO reinforcement learning algorithm to train the deep reinforcement learning decision model to learn the optimal safety decision-making strategy from the multimodal spatiotemporal representation composed of the lidar BEV features and future prediction features, as well as the speed data, the deviation distance from the lane center, and the deviation angle data.

2. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: The image feature extraction backbone network extracts features through four residual convolution blocks of a ResNet-34 network pre-trained on ImageNet to obtain four image features with different levels of information. Among them S i Indicates different feature extraction stages.

3. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: The LiDAR BEV feature extraction backbone network takes six consecutive frames of six-channel LiDAR BEV projection pseudo images as input, and extracts features through four spatiotemporal convolution blocks of a VideoResnet-18 network that introduces a spatiotemporal convolution structure to obtain four LiDAR BEV features with different levels of information. Among them S i Indicates different feature extraction stages.

4. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: The multimodal feature fusion network is used to fuse the image features of four different scales and the laser radar BEV features of four different scales. When fusion is performed, the image features extracted by the backbone network of the two branches are firstly combined. and LiDAR BEV features After dimension reshaping, the sequence vector is obtained by concatenation Then Through a multimodal fusion Transformer module, full information interaction between different modal features is achieved to obtain the global spatiotemporal context features in the 3D scene. Finally, Slice and restore to and Features of the same dimension, and Add elements to obtain fused image features and LiDAR BEV features 5. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: The image branch and the lidar BEV branch are supervised by different multi-task heads. The image branch is trained by H dep and H sem The BEV point cloud branch consists of two task heads, namely depth estimation and semantic segmentation of the front view image; bev , H v and H bb It is composed of two parts, which are used for BEV semantic segmentation, surrounding vehicle speed prediction and 2D target detection respectively.

6. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: In step S3, the position attention network combines the multimodal spatiotemporal fusion of the lidar BEV feature f lidar_fusion Input into three convolutional layers respectively to obtain three feature maps of the same dimension as the original, and then adjust their dimensions to three two-dimensional features of the same dimension and Then, in The transpose and Perform matrix multiplication between and apply the SoftMax layer to calculate the spatial attention map s lo ; then in s lo and Perform matrix multiplication between the transposes of to capture the spatial dependency between any two locations in the feature map and resize the result to get the same value as f lidar_fusion Feature maps of the same dimension Finally and f lidar_fusion Add the elements to get the final output of the position attention network 7. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: In step S3, the channel attention network combines the multimodal spatiotemporal fusion of the lidar BEV feature f lidar_fusion Perform dimension adjustment to obtain two two-dimensional features of the same dimension Then The transpose and Perform matrix multiplication between and apply a SoftMax layer to obtain the channel attention map s ch ; then in s ch and f lidar_fusion Perform matrix multiplication between the transposes of to capture the channel dependency between any two channels and reshape the result to get the same value as f lidar_fusion Feature maps of the same dimension Finally and f lidar_fusion Add the elements to get the final output of the channel attention network 8. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: In step S3, the attention fusion network combines the outputs of the position attention network and the channel attention network and After element addition and convolution operations, the future prediction feature f is obtained future ; The future prediction task head network predicts features f from the future future parse the scene state 0.5 seconds in the future.

9. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1 is characterized in that: In step S4, the reward function includes a sparse reward obtained by triggering a predefined event and a dense reward obtained for each timestamp; the sparse rewards include six types, namely, collision, unexplained parking, speeding, deviation distance greater than a threshold, deviation angle greater than a threshold, and deviation angular velocity greater than a threshold. When the above conditions are met, a penalty is given; the dense rewards include four types, namely, deviation distance reward, deviation angle reward, angular velocity reward, and speed reward.

10. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 1, characterized in that: In step S4, a multi-branch network structure with horizontal and vertical separation is adopted, and an independent action prediction branch is adopted for each navigation command in a set of high-level navigation commands. The navigation command acts as a selection switch for which branch to use at each timestamp, and each branch learns a sub-strategy specific to its navigation command; in each branch, considering that the vehicle's motion control involves two relatively independent operations: lateral control and longitudinal control, two identical branches are designed to handle lateral control and longitudinal control respectively.

11. The cascaded deep reinforcement learning safety decision-making method based on multimodal spatiotemporal representation according to claim 10, characterized in that: The network structures of both horizontal and vertical control are composed of strategy networks and value networks; The strategy network combines the multimodal spatiotemporal fusion of the lidar BEV feature map f lidar_fusion and future prediction feature map f future The channel splicing operation is performed to obtain the feature map f, which is then flattened into a one-dimensional feature vector after four layers of convolution. Finally, the feature vector is connected with the vehicle measurement vector containing speed data, deviation distance from the lane center, and deviation angle data, and then passed through three layers of fully connected layers to output the d-dimensional discrete action prediction probability P(s t ); The value network combines the multi-modal spatiotemporal fusion of the lidar BEV feature map f lidar_fusion and future prediction feature map f future The channel splicing operation is performed to obtain the feature map f, which is then flattened into a one-dimensional feature vector after four layers of convolution. Finally, the feature vector is connected with the vehicle measurement vector containing speed data, deviation distance from the lane center, and deviation angle data, and then passed through three layers of fully connected layers to output a one-dimensional state value prediction Q(s t ).

Images