A multi-modal perception unmanned vehicle trajectory control method based on reinforcement learning

Abstract

The application discloses a kind of multi-modal perception unmanned vehicle trajectory control methods based on reinforcement learning, belong to automatic driving and artificial intelligence technical field, the method is by with laser radar collection road condition information, through vehicle-mounted camera image information is collected, through laser scanner auxiliary obstacle avoidance, the environment is perceived with multi-modal, training and using reinforcement learning intelligent agent to navigate, control the moving direction and speed of unmanned vehicle, so that it completes task and avoids collision barrier.The application constructs environmental characteristics, designs adaptive reward function, when the agent selects strategy, task execution efficiency and movement safety are considered, which can better cope with complex driving scenarios.The application does not need to preset high-precision map, can adjust sensor data weight according to environmental complexity, construct multi-scene virtual simulation environment, and verify the generalization ability of the model in the automatic driving technology by combining the real car.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and artificial intelligence technology, and in particular to a multimodal perception unmanned vehicle trajectory control method based on reinforcement learning. Background Technology

[0002] With the acceleration of global urbanization, autonomous driving technology has become crucial for solving traffic safety and congestion problems. However, traditional autonomous driving systems rely on pre-set rules or high-precision maps, making it difficult to cope with dynamic environments (such as sudden obstacles and changes in traffic signals) and real-time decision-making delays in complex interactive scenarios. Therefore, autonomous driving technology must ensure the real-time nature and accuracy of decision-making, selecting safe and effective driving decisions in the face of unknown environmental changes.

[0003] Reinforcement learning control methods are better suited for intelligently handling autonomous driving decision-making problems in complex traffic scenarios and effectively improving the intelligence of autonomous driving. For example, Wang Xiao et al., in their research on "Research on Cognitive Autonomous Driving Intelligent Architecture Based on Parallel Testing," argue that traditional rule-based systems rely on high-precision maps and preset logic, making it difficult to cope with sudden road conditions in dynamic environments. Reinforcement learning (RL), on the other hand, demonstrates unique advantages through environmental interaction and autonomous optimization. Beijing University of Posts and Telecommunications proposed a method and device for controlling vehicles to travel along a predetermined trajectory using a DQN network. Its focus is on line following, using kinematic parameters as input and the control quantity of the vehicle as output. Beijing Institute of Technology provided an autonomous driving decision-making and planning method, which consists of two modules: lane-changing decision and trajectory planning. The lane-changing decision module is implemented using DQN technology. In international research, Ari Seff's team proposed a "direct perception" paradigm, using deep convolutional networks to extract driving semantic features from images and achieve cross-scenario generalization in virtual environments. Keishi Ishihara, meanwhile, improved intersection throughput by simultaneously optimizing path planning and traffic signal recognition through a multi-task attention network. To overcome data bottlenecks, Xinyu Huang released the ApolloScape multimodal dataset, supporting high-precision environmental perception model training. In the field of security and privacy, Yijing Li combined blockchain with zero-knowledge proofs (ZKP) to construct a privacy-preserving federated learning framework, reducing model training loss. Chen Yanfeng's team proposed a sensor attack-resistant perception fusion algorithm, reducing measurement errors by 40%. Breakthroughs in deep reinforcement learning (DRL) are reflected in end-to-end control and multi-objective optimization. For example, the Xi'an University of Electronic Science and Technology team designed a multi-objective reward function, using the DDPG algorithm to balance safety and comfort in emergency braking scenarios. Domestic research focuses on improving intelligent decision-making. Wu Sifan's team demonstrated the superiority of DRL in complex traffic scenarios, proposing a hierarchical decision architecture. Xiang Chaocan's team used a dual-attention mechanism to improve the efficiency of task allocation in intelligent connected vehicles by 23%. Delu Technology Co., Ltd. invented a path planning method that incorporates safety considerations, balancing efficiency and safety by optimizing the reward function. Shandong University proposed a "last-place elimination" approach, replacing low-value experience with high-value experience, significantly reducing caching requirements.

[0004] However, existing reinforcement learning algorithms (such as DQN) often suffer from low training efficiency and poor generalization ability in autonomous driving due to their simplistic reward function design. They also lack dynamic constraints on safety boundaries and face multiple challenges: the contradiction between trial-and-error learning and safety constraints needs to be resolved through simulation pre-training and safety barrier functions; the computing power limitations of edge devices necessitate lightweight models (such as MobileNetv3) and hardware acceleration technologies; and the efficiency of multimodal data fusion requires optimization of the network architecture (such as attention mechanisms). Future development will focus on breakthroughs in safety constraint algorithms, real-time computing systems, and cross-modal collaborative learning. Reinforcement learning drives the autonomous optimization of autonomous driving systems through environmental interaction and reward mechanisms, but breakthroughs are still needed in key technologies such as multimodal fusion, lightweight deployment, and safety constraints to improve safety, real-time performance, and generalization ability.

[0005] To address the issues of low training efficiency and poor generalization ability caused by the single reward function design in the aforementioned autonomous driving technology based on general reinforcement learning, this invention proposes a multimodal perception unmanned vehicle trajectory control method based on reinforcement learning. Summary of the Invention

[0006] This invention discloses a multimodal perception-based trajectory control method for unmanned vehicles (UAVs) based on reinforcement learning. It utilizes LiDAR to collect road condition information, onboard cameras to acquire image information, and a laser scanner to assist obstacle avoidance, enabling multimodal environmental perception. A reinforcement learning agent is then trained and used for navigation, controlling the UAV's direction and speed to complete tasks and avoid collisions. This method constructs environmental features, designs an adaptive reward function, and trains the agent to balance task efficiency and mobility safety when selecting strategies. Compared to existing methods, this approach better handles complex driving scenarios.

[0007] A multimodal perception-based trajectory control method for autonomous vehicles based on reinforcement learning includes the following steps:

[0008] Step 1: Obtain the intelligent autonomous vehicle system. Assume that an intelligent autonomous vehicle system includes an onboard depth camera, an onboard LiDAR, and the vehicle's power system (fuel / electricity).

[0009] Step 2: The autonomous vehicle constructs a dual deep neural network model, which includes an Actor network and a Critic network, and initializes the learning rate α, the discount factor δ, and the weight parameters θ of the Actor network and the Critic network. (0) and

[0010] Step 3: The intelligent unmanned vehicle system observes and constructs the state information s at time t through the onboard system. t .

[0011] Step 4: The intelligent unmanned vehicle selects the action to execute and sends its status information to the system. t The inputs are fed into the Actor network and the Critic network, respectively.

[0012] Step 5: Train the network.

[0013] Step 6: Soft update the target network.

[0014] Step 7: Repeat steps 3 to 6 until the vehicle system learns a stable action selection strategy, that is, the reward converges to a stable value.

[0015] Furthermore, in step 2, the input to the Actor network is the vehicle's state information, which includes information from the onboard depth camera, the onboard LiDAR point cloud, and the vehicle's power information. The output is the vehicle's motion, which consists of the linear velocity and angular velocity of the entire vehicle. Subsequently, the rotational speeds of the four wheel motors are calculated using the engine. The input to the Critic network is the vehicle's state information and motion, and the output is the corresponding reward value Q. The quality of the current state and motion is evaluated based on the distance to the finish line and the number of collisions.

[0016] Furthermore, the status information in step 3 includes vehicle-mounted depth camera information, vehicle-mounted LiDAR point cloud information, and vehicle power information.

[0017] Furthermore, step 4 includes the following steps:

[0018] Step 4.1: The Actor network outputs the action 'a' that the vehicle needs to perform in the current state. t ;

[0019] Step 4.2: Perform action a t And observe the next state information s t+1 Rewards r t And whether it's finished (done), where the reward r t The construction of the target is determined by the distance from the endpoint and the number of collisions.

[0020] Step 4.3: Transfer the empirical data (s) t ,a t ,r t ,s t+1 ,done) is stored in the experience replay buffer.

[0021] Furthermore, the specific method for step 5 is as follows:

[0022] Step 5.1: Randomly select a batch of empirical data (s) from the empirical replay buffer. t ,a t ,r t ,st+1 ,done);

[0023] Step 5.2: Transfer the status information s t and action a t The input is fed into the Critic network to obtain the predicted Q-value Q. pred ;

[0024] Step 5.3: Send the next status information s t+1 The input is fed into the Actor_target network to obtain the target action a. t+1 ;

[0025] Step 5.4: Send the next status information s t+1 and target action a t+1 The input is fed into the Critic_target network to obtain the target Q-value Q. target .

[0026] Step 5.5: The formula for calculating the target Q value is: If done is True, then Q... target =r t If done is False, then Q target =r t +γ*Q target Define the loss function L respectively. Critic and L Actor L Critic Mean squared error is used to measure the difference between the predicted Q value and the target Q value: To maximize the Q-value of the Critic network output, the loss function of the Actor network is defined as follows: The weights of the network are updated by calculating the losses of the Critic network and the Actor network, respectively.

[0027] Furthermore, the specific method for step 6 is as follows: The parameters of the target network are slowly updated using a soft update formula to keep it synchronized with the main network.

[0028] Actor_target.params=τ*Actor.params+(1-τ)*Actor_target.params;

[0029] Critic_target.params=τ*Critic.params+(1-τ)*Critic_target.params;

[0030] Where τ is a small constant with a value range of (0, 0.01).

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This invention is designed for reinforcement learning reward mechanisms and proposes a dynamic reward system of "basic reward + safety penalty" to improve model training efficiency while balancing path efficiency and safety risks.

[0033] 2. This invention, through reinforcement learning, can adaptively adjust the weights of sensor data based on environmental complexity (such as rainy days or high traffic density) without the need for a pre-set high-precision map, while avoiding dangerous strategies. It can construct a multi-scenario virtual simulation environment (including weather, lighting, and obstacle changes) and combine it with a real car to verify the generalization ability of the model in autonomous driving technology.

[0034] 3. This invention uses the distance to the finish line and the number of collisions as rewards to evaluate the risk value of the strategy. An Actor network is designed to generate the robot's control actions based on the current state and the goal. A Critic network is then used to evaluate the value of the actions generated by the Actor network, optimizing the action selection strategy of the onboard system and improving driving safety and efficiency. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the onboard equipment of a smart car.

[0036] Figure 2 This is a schematic diagram of the experimental topology. Detailed Implementation

[0037] To better understand the technical content of this invention, the following embodiments are provided and described in detail with reference to the accompanying drawings.

[0038] A multimodal perception-based trajectory control method for autonomous vehicles based on reinforcement learning includes the following steps:

[0039] Step 1: An intelligent autonomous vehicle system includes an onboard depth camera, an onboard LiDAR, and the vehicle's power system (fuel / electricity), such as... Figure 1 As shown;

[0040] Step 2: Construct a dual deep neural network for the autonomous vehicle, consisting of an Actor network and a Critic network. The Actor and Critic networks have the same structure, both composed of fully connected layers, including one input layer, two hidden layers, and one output layer. The input layer consists of 22 neurons, the two hidden layers consist of 64 and 128 neurons respectively, and the output layer consists of 36 neurons. Set the learning rate α to 0.5 and the discount factor δ to 0.7. Initialize the weight parameters of the Actor and Critic networks. and

[0041] Step 3: Construct the current system state vector s of the autonomous vehicle t At time t, the autonomous vehicle's onboard system observes current status information, including information from the onboard depth camera, point cloud information from the onboard LiDAR, and vehicle dynamics information.

[0042] Step 4: Based on the feedback information, the intelligent autonomous vehicle selects the action to execute. The intelligent autonomous vehicle will then transfer its state vector s... t The inputs are fed into the Actor network and the Critic network, respectively. The specific steps include:

[0043] Step 4.1: The Actor network outputs the action 'a' that the vehicle needs to perform in the current state. t ;

[0044] Step 4.2: Perform action a t And observe the next state information s t+1 Rewards r t And whether it's finished;

[0045] Step 4.3: Transfer the empirical data (s) t ,a t ,r t ,s t+1 ,done) is stored in the experience replay buffer.

[0046] Among them, the reward r t The construction of the target is determined by the distance from the endpoint and the number of collisions.

[0047] Step 5: Train the network, the specific method is as follows:

[0048] Step 5.1: Randomly select a batch of empirical data (s) from the empirical replay buffer. t ,a t ,r t ,s t+1 ,done);

[0049] Step 5.2: Transfer the status information s t and action a t The input is fed into the Critic network to obtain the predicted Q-value Q. pred ;

[0050] Step 5.3: Send the next status information s t+1 The input is fed into the Actor_target network to obtain the target action a. t+1 ;

[0051] Step 5.4: Send the next status information s t+1 and target action a t+1 The input is fed into the Critic_target network to obtain the target Q-value Q.target .

[0052] Step 5.5: The formula for calculating the target Q value is: If done is True, then Q... target =r t If done is False, then Q target =r t +γ*Q target To train the Critic network and the Actor network, loss functions L are defined respectively. Critic and L Actor L Critic Mean squared error is used to measure the difference between the predicted Q value and the target Q value: To maximize the Q-value of the Critic network output, the loss function of the Actor network is defined as follows: The weights of the network are updated by calculating the losses of the Critic network and the Actor network, respectively.

[0053] Step 6: The intelligent autonomous vehicle system uses a soft update formula to slowly update the parameters of the Actor and Critic target networks, keeping them synchronized with the main network.

[0054] Actor_target.params=τ*Actor.params+(1-τ)*Actor_target.params;

[0055] Critic_target.params=τ*Critic.params+(1-τ)*Critic_target.params;

[0056] Where τ is taken as 0.001.

[0057] Step 7: Repeat steps 3-6 until the vehicle system learns a stable action selection strategy, i.e., the reward converges to a stable value, such as... Figure 2 The diagram shows an experimental topology based on the method of this invention.

[0058] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A trajectory control method for multimodal perception autonomous vehicles based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain the intelligent unmanned vehicle system. Assume that an intelligent unmanned vehicle system includes an onboard depth camera, an onboard LiDAR, and the vehicle's power system. Step 2: Construct a dual deep neural network model for the autonomous vehicle. The dual deep neural network model includes an Actor network and a Critic network. Initialize the learning rate. Discount Factor and the weight parameters of the Actor network and the Critic network. and ; Step 3: The intelligent unmanned vehicle system observes and constructs data through the onboard system. t Status information at any time ; Step 4: The intelligent unmanned vehicle selects the action to execute and sends its status information. The inputs are fed into the Actor network and the Critic network, respectively. Step 5: Train the network; Step 6: Soft update the target network; Step 7: Repeat steps 3-6 until the vehicle system learns a stable action selection strategy, that is, the reward converges to a stable value; Step 4 includes the following steps: Step 4.1: The Actor network outputs the actions that the vehicle needs to perform in the current state. ; Step 4.2: Perform the action And observe the next state information. ,award And whether it's finished, i.e., whether it's done, including the reward. The construction of the target is determined by the distance from the endpoint and the number of collisions. Step 4.3: Transfer the empirical data ( , , , (done) is stored in the experience replay buffer; Step 5 includes the following steps: Step 5.1: Randomly select a batch of empirical data from the empirical replay buffer. , , , (done); Step 5.2: Transfer status information and actions The input is fed into the Critic network to obtain the predicted Q-value. ; Step 5.3: Send the next status information The input is fed into the Actor_target network to obtain the target action. ; Step 5.4: Send the next status information and target action The input is fed into the Critic_target network to obtain the target Q-value. ; Step 5.5: The formula for calculating the target Q value is: If done is True, then = If done is False, then = + γ * Define loss functions respectively and ,in Mean squared error is used to measure the difference between the predicted Q value and the target Q value: To maximize the Q-value of the Critic network output, the loss function of the Actor network is defined as follows: The weights of the network are updated by calculating the losses of the Critic network and the Actor network, respectively.

2. The method for trajectory control of an unmanned vehicle based on reinforcement learning according to claim 1, characterized in that, In step 2, the Actor network takes the vehicle's state information as input and outputs the vehicle's actions, which are the linear and angular velocities of the entire vehicle. Subsequently, the rotational speeds of the four wheel motors are calculated using the engine. The Critic network takes the vehicle's state information and actions as input and outputs the corresponding reward value Q. The quality of the current state and actions is evaluated based on the distance to the finish line and the number of collisions.

3. The method for trajectory control of an unmanned vehicle based on reinforcement learning according to claim 1, characterized in that, The status information in step 3 includes information from the vehicle-mounted depth camera, point cloud information from the vehicle-mounted LiDAR, and vehicle power information.

4. The method for trajectory control of an unmanned vehicle based on reinforcement learning according to claim 1, characterized in that, The specific method for step 6 is as follows: The parameters of the target network are slowly updated using a soft update formula to keep it synchronized with the main network. Actor_target.params = τ * Actor.params + (1 - τ) * Actor_target.params; Critic_target.params = τ * Critic.params + (1 - τ) * Critic_target.params; Where τ is a small constant with a value range of (0, 0.01).

Images