Multi-mode perception unmanned vehicle track control method based on reinforcement learning

Abstract

The invention discloses a multi-mode perception unmanned vehicle track control method based on reinforcement learning, and belongs to the technical field of automatic driving and artificial intelligence, and the method comprises the steps: collecting road condition information through a laser radar, collecting image information through a vehicle-mounted camera, assisting obstacle avoidance through a laser scanner, and carrying out the multi-mode perception of an environment, and training and utilizing a reinforcement learning agent for navigation, and controlling the moving direction and speed of the unmanned vehicle, so that the unmanned vehicle completes tasks and avoids collision obstacles. According to the method, the environment features are constructed, the self-adaptive reward function is designed, the task execution efficiency and the movement safety are both considered during strategy selection of the training agent, and a complex driving scene can be better coped with. According to the method, a high-precision map does not need to be preset, the sensor data weight can be adaptively adjusted according to the environmental complexity, the multi-scene virtual simulation environment is constructed, and the generalization ability of the model in the automatic driving technology is verified in combination with a real trolley.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving and artificial intelligence technology, and in particular to a multimodal perception unmanned vehicle trajectory control method based on reinforcement learning. Background Art

[0002] As global urbanization accelerates, autonomous driving technology has become crucial for addressing traffic safety and congestion. However, traditional autonomous driving systems rely on pre-set rules or high-precision maps, making them difficult to handle in dynamic environments (such as sudden obstacles and traffic signal changes) and delaying real-time decision-making in complex interactive scenarios. Therefore, autonomous driving technology must ensure real-time and accurate decision-making, ensuring safe and effective driving decisions in the face of unpredictable environmental changes.

[0003] Reinforcement learning control methods are more suitable for intelligently handling autonomous driving decision-making in complex traffic scenarios and effectively improving the intelligence of autonomous driving. For example, Wang Xiao et al., in their paper "Research on Intelligent Architecture for Cognitive Autonomous Driving Based on Parallel Testing," argue that traditional rule-based systems rely on high-precision maps and preset logic, making them difficult to cope with unexpected road conditions in dynamic environments. Reinforcement learning (RL), on the other hand, demonstrates unique advantages through environmental interaction and autonomous optimization. Beijing University of Posts and Telecommunications proposed a method and device using a DQN network to control a vehicle along a predetermined trajectory. This method focuses on tracking, using kinematic parameters as input and outputting vehicle control variables. Beijing Institute of Technology proposed a decision-making and planning method for autonomous driving, which consists of two modules: lane change decision-making and trajectory planning. The lane change decision module is implemented using DQN technology. In international research, Ari Seff's team proposed a "direct perception" paradigm, using deep convolutional networks to extract driving semantic features from images and achieve cross-scenario generalization in virtual environments. Keishi Ishihara used a multi-task attention network to simultaneously optimize path planning and traffic sign recognition, improving intersection pass rates. To overcome data bottlenecks, Xinyu Huang released the ApolloScape multimodal dataset, supporting high-precision environmental perception model training. In the area of ​​security and privacy, Yijing Li combined blockchain with zero-knowledge proofs (ZKPs) to build a privacy-preserving federated learning framework, reducing model training losses. Chen Yanfeng's team proposed a sensor-attack-resistant perception fusion algorithm, reducing measurement error by 40%. Breakthroughs in deep reinforcement learning (DRL) are evident in end-to-end control and multi-objective optimization. For example, a team from Xidian University designed a multi-objective reward function, using the DDPG algorithm to balance safety and comfort in emergency braking scenarios. Domestic research focuses on improving intelligent decision-making. Wu Sifan's team demonstrated the superiority of DRL in complex traffic scenarios and proposed a hierarchical decision-making architecture. Xiang Chaocan's team used a dual-attention mechanism to achieve a 23% improvement in task allocation efficiency for connected cars. Delu Technology Co., Ltd. developed a path planning method that takes safety into account and balances efficiency and safety by optimizing the reward function. Shandong University proposed a "last-in-first-out" approach that significantly reduces cache requirements by replacing low-value experience with high-value experience.

[0004] However, existing reinforcement learning algorithms (such as DQN) in autonomous driving often suffer from low training efficiency and poor generalization due to the simple design of reward functions. They also lack dynamic constraints on safety boundaries and face multiple challenges: the contradiction between trial-and-error learning and safety constraints must be resolved through simulation pre-training and safety barrier functions; the computing power limitations of edge devices require lightweight models (such as MobileNetv3) and hardware acceleration technologies; and the efficiency of multimodal data fusion requires optimized network architectures (such as attention mechanisms). Future development will focus on technological breakthroughs in safety constraint algorithms, real-time computing systems, and cross-modal collaborative learning. Reinforcement learning drives the autonomous optimization of autonomous driving systems through environmental interaction and reward mechanisms, but key technologies such as multimodal fusion, lightweight deployment, and safety constraints are still needed to ensure safety, real-time performance, and generalization.

[0005] In response to the problems of single reward function design in the above-mentioned general reinforcement learning-based autonomous driving technology, which leads to low training efficiency and poor generalization ability, the present invention proposes a multimodal perception unmanned vehicle trajectory control method based on reinforcement learning. Summary of the Invention

[0006] This invention discloses a multimodal perception-based unmanned vehicle trajectory control method based on reinforcement learning. This method uses a lidar to collect road condition information, an onboard camera to capture image information, and a laser scanner to assist in obstacle avoidance. This method uses multimodal perception of the environment, trains and utilizes a reinforcement learning agent to navigate and control the direction and speed of the unmanned vehicle, enabling it to complete its mission and avoid collisions. By constructing environmental features and designing an adaptive reward function, this method trains the agent to balance task execution efficiency and mobility safety when selecting strategies. Compared to existing methods, this method is more capable of handling complex driving scenarios.

[0007] A multimodal perception unmanned vehicle trajectory control method based on reinforcement learning includes the following steps:

[0008] Step 1: Obtain an intelligent unmanned vehicle system. Assume that an intelligent unmanned vehicle system includes an on-board depth camera, an on-board lidar, and the vehicle's power system (fuel / power).

[0009] Step 2: The unmanned vehicle constructs a dual-depth neural network, which includes an Actor network and a Critic network, and initializes the learning rate α, discount factor δ, and weight parameters θ of the Actor network and the Critic network. (0) and

[0010] Step 3: The intelligent unmanned vehicle system observes and constructs the state information s at time t through the vehicle system t .

[0011] Step 4: The intelligent unmanned vehicle selects an action to execute and sends the status information s t Input into the Actor network and Critic network respectively.

[0012] Step 5: Train the network.

[0013] Step 6: Soft update the target network.

[0014] Step 7: Repeat steps 3 to 6 until the vehicle system learns a stable action selection strategy, that is, the reward converges to a stable value.

[0015] Furthermore, the input of the Actor network in step 2 is the vehicle's state information, which includes the vehicle's depth camera information, the vehicle's lidar point cloud information, and the vehicle's power information. The output is the vehicle's action, which is the linear velocity and angular velocity of the entire vehicle. The engine then calculates the rotational speed of each of the four wheel motors. The input of the Critic network is the vehicle's state information and action, and the output is the corresponding reward value Q value. The quality of the current state and action is evaluated by the distance to the end point and the number of collisions.

[0016] Furthermore, the status information in step 3 includes vehicle-mounted depth camera information, vehicle-mounted lidar point cloud information, and vehicle power information.

[0017] Furthermore, step 4 includes the following steps:

[0018] Step 4.1: The Actor network outputs the action a that the vehicle needs to perform in the current state t ;

[0019] Step 4.2: Execute action a t , and observe the next state information s t+1 , reward r t And whether it is finished (done), where the reward r t The construction of is determined by the distance from the end point and the number of collisions;

[0020] Step 4.3: Empirical data (s t ,a t ,r t ,s t+1 ,done) is stored in the experience replay buffer.

[0021] Furthermore, the specific method of step 5 is:

[0022] Step 5.1: Randomly extract a batch of experience data (s t ,a t ,r t ,st+1 ,done);

[0023] Step 5.2: Send status information t and action a t Input into the Critic network to get the predicted Q value Q pred ;

[0024] Step 5.3: Send the next state information s t+1 Input to the Actor_target network to get the target action a t+1 ;

[0025] Step 5.4: Send the next state information s t+1 and target action a t+1 Input to the Critic_target network to get the target Q value Q target .

[0026] Step 5.5: The formula for calculating the target Q value is: If done is True, then Q target =r t ; If done is False, then Q target =r t +γ*Q target . Define the loss function L respectively Critic and L Actor . Where L Critic The mean squared error is used to measure the difference between the predicted Q value and the target Q value: In order to maximize the Q value output by the Critic network, the loss function of the Actor network is defined as The network weights are updated by calculating the losses of the Critic network and the Actor network respectively.

[0027] Furthermore, the specific method of step 6 is: using a soft update formula to slowly update the parameters of the target network so that it keeps synchronization with the main network:

[0028] Actor_target.params=τ*Actor.params+(1-τ)*Actor_target.params;

[0029] Critic_target.params=τ*Critic.params+(1-τ)*Critic_target.params;

[0030] Here, τ is a small constant with a value range of (0, 0.01).

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This invention is designed for a reinforcement learning reward mechanism and proposes a dynamic reward system of "basic reward + safety penalty" to balance path efficiency and safety risks while improving model training efficiency.

[0033] 2. Through reinforcement learning, the present invention does not require the preset high-precision map under the premise of risk avoidance strategies. It can adaptively adjust the sensor data weight according to the complexity of the environment (such as rainy days and high traffic density), build a multi-scenario virtual simulation environment (including weather, lighting, and obstacle mutations), and combine it with a real car to verify the generalization ability of the model in autonomous driving technology.

[0034] 3. This invention uses the distance to the endpoint and the number of collisions as rewards to assess the strategy's risk. An actor network is designed to generate the robot's control actions based on the current state and goal. Combined with a critic network, the value of the actor-generated actions is evaluated, optimizing the vehicle system's action selection strategy and improving driving safety and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the onboard equipment of the smart car.

[0036] Figure 2 Schematic diagram of the experimental topology. DETAILED DESCRIPTION

[0037] In order to more clearly understand the technical content of the present invention, the following embodiments are given and described in detail with reference to the accompanying drawings.

[0038] A multimodal perception unmanned vehicle trajectory control method based on reinforcement learning includes the following steps:

[0039] Step 1: An intelligent unmanned vehicle system includes an on-board depth camera, an on-board lidar, and the vehicle's power system (fuel / power), such as Figure 1 As shown;

[0040] Step 2: The unmanned vehicle constructs a dual deep neural network consisting of an actor network and a critic network. The actor network and the critic network have the same structure, both consisting of fully connected layers, including an input layer, two hidden layers, and an input layer. The input layer consists of 22 neurons, the two hidden layers consist of 64 and 128 neurons respectively, and the output layer consists of 36 neurons. Set the learning rate α to 0.5 and the discount factor δ to 0.7. Initialize the weight parameters of the actor network and the critic network. and

[0041] Step 3: Construct the current system state vector s of the unmanned vehicle t At time t, the unmanned vehicle's onboard system observes the current state information, including the onboard depth camera information, the onboard lidar point cloud information, and the vehicle's power information.

[0042] Step 4: Based on the feedback information, the intelligent unmanned vehicle selects an action to perform. The intelligent unmanned vehicle converts the state vector s t Input into the Actor network and Critic network respectively. The specific steps include:

[0043] Step 4.1: The Actor network outputs the action a that the vehicle needs to perform in the current state t ;

[0044] Step 4.2: Execute action a t , and observe the next state information s t+1 , reward r t and whether it is finished (done);

[0045] Step 4.3: Empirical data (s t ,a t ,r t ,s t+1 ,done) is stored in the experience replay buffer.

[0046] Among them, the reward r t The construction of is determined by the distance from the end point and the number of collisions that occur.

[0047] Step 5: Train the network. The specific method is:

[0048] Step 5.1: Randomly extract a batch of experience data (s t ,a t ,r t ,s t+1 ,done);

[0049] Step 5.2: Send status information t and action a t Input into the Critic network to get the predicted Q value Q pred ;

[0050] Step 5.3: Send the next state information s t+1 Input to the Actor_target network to get the target action a t+1 ;

[0051] Step 5.4: Send the next state information s t+1 and target action a t+1 Input to the Critic_target network to get the target Q value Qtarget .

[0052] Step 5.5: The formula for calculating the target Q value is: If done is True, then Q target =r t ; If done is False, then Q target =r t +γ*Q target In order to train the Critic network and the Actor network, the loss function L is defined respectively. Critic and L Actor . Where L Critic The mean squared error is used to measure the difference between the predicted Q value and the target Q value: In order to maximize the Q value output by the Critic network, the loss function of the Actor network is defined as The network weights are updated by calculating the losses of the Critic network and the Actor network respectively.

[0053] Step 6: The intelligent unmanned vehicle system uses a soft update formula to slowly update the parameters of the Actor and Critic target networks to keep them synchronized with the main network:

[0054] Actor_target.params=τ*Actor.params+(1-τ)*Actor_target.params;

[0055] Critic_target.params=τ*Critic.params+(1-τ)*Critic_target.params;

[0056] Here, τ is set to 0.001.

[0057] Step 7: Repeat steps 3 to 6 until the vehicle system learns a stable action selection strategy, that is, the reward converges to a stable value, such as Figure 2 Shown is a schematic diagram of the experimental topology based on the method of the present invention.

[0058] The above embodiments are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A multimodal perception unmanned vehicle trajectory control method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Obtain an intelligent unmanned vehicle system. Assume that an intelligent unmanned vehicle system includes an on-board depth camera, an on-board lidar, and a vehicle power system. Step 2: The unmanned vehicle constructs a dual-depth neural network model, which includes an Actor network and a Critic network. Initialize the learning rate α, discount factor δ, and the weight parameters θ of the Actor network and the Critic network. (0) and Step 3: The intelligent unmanned vehicle system observes and constructs the state information s at time t through the vehicle system t ; Step 4: The intelligent unmanned vehicle selects an action to execute and sends the status information s t Input into the Actor network and Critic network respectively; Step 5: Train the network; Step 6: Soft update target network; Step 7: Repeat steps 3 to 6 until the vehicle system learns a stable action selection strategy, that is, the reward converges to a stable value.

2. The multimodal perception unmanned vehicle trajectory control method based on reinforcement learning according to claim 1 is characterized in that: The input of the Actor network in step 2 is the vehicle's state information, and the output is the vehicle's action. The vehicle's action is the linear velocity and angular velocity of the entire vehicle, and the engine subsequently calculates the rotational speed of the four wheel motors. The input of the Critic network is the vehicle's state information and action, and the output is the corresponding reward value Q value. The quality of the current state and action is evaluated by the distance to the end point and the number of collisions.

3. The multimodal perception unmanned vehicle trajectory control method based on reinforcement learning according to claim 1 is characterized in that: The status information in step 3 includes vehicle-mounted depth camera information, vehicle-mounted lidar point cloud information, and vehicle power information.

4. The multi-modal perception unmanned vehicle trajectory control method based on reinforcement learning according to claim 1 is characterized in that: The step 4 comprises the following steps: Step 4.1: The Actor network outputs the action a that the vehicle needs to perform in the current state t ; Step 4.2: Execute action a t , and observe the next state information s t+1 , reward r t And whether it is finished (done), where the reward r t The construction of is determined by the distance from the end point and the number of collisions; Step 4.3: Empirical data (s t ,a t ,r t ,s t+1 ,done) is stored in the experience replay buffer.

5. The multi-modal perception unmanned vehicle trajectory control method based on reinforcement learning according to claim 1 is characterized in that: The step 5 comprises the following steps: Step 5.1: Randomly extract a batch of experience data (s t ,a t ,r t ,s t+1 ,done); Step 5.2: Send status information t and action a t Input into the Critic network to get the predicted Q value Q pred ; Step 5.3: Send the next state information s t+1 Input to the Actor_target network to get the target action a t+1 ; Step 5.4: Send the next state information s t+1 and target action a t+1 Input to the Critic_target network to get the target Q value Q target ; Step 5.5: The formula for calculating the target Q value is: If done is True, then Q target =r t ; If done is False, then Q target =r t +γ*Q target , respectively define the loss function L Critic and L Actor , where L Critic The mean squared error is used to measure the difference between the predicted Q value and the target Q value: In order to maximize the Q value output by the Critic network, the loss function of the Actor network is defined as The network weights are updated by calculating the losses of the Critic network and the Actor network respectively.

6. The multi-modal perception unmanned vehicle trajectory control method based on reinforcement learning according to claim 1 is characterized in that: The specific method of step 6 is: use the soft update formula to slowly update the parameters of the target network to keep it synchronized with the main network: Actor_target.params=τ*Actor.params+(1-τ)*Actor_target.params; Critic_target.params=τ*Critic.params+(1-τ)*Critic_target.params; Here, τ is a small constant with a value range of (0, 0.01).

Images