Proximal policy optimization method for autonomous driving decision-making based on distributed exploration enhancement

Abstract

The present invention belongs to the field of autonomous driving technology, and discloses a proximal policy optimization method for autonomous driving decision-making based on distributed exploration enhancement. The present invention designs a distributed architecture of the reinforcement learning decision-making module of the autonomous driving vehicle. Through parallelization and distributed processing, multiple agents can simultaneously perform policy training in different environments, and share policy updates and experience data in real time. In addition, the distributed architecture allows the method to operate on larger data sets and complex environments, improving the adaptability and generalization ability of autonomous driving decisions. The present invention also combines a random network exploration RND module to further enhance the diversity and effectiveness of exploration, and improves the proximal policy optimization algorithm used by the reinforcement learning decision-making module. By combining the distributed architecture with the RND module, the present invention significantly accelerates the training process, improves the exploration efficiency, and comprehensively improves the decision-making quality of the autonomous driving system, and improves the reliability and efficiency of autonomous driving decisions.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to a proximal strategy optimization method for autonomous driving decision-making based on distributed exploration enhancement, which is suitable for autonomous driving decision-making in urban traffic. Background Art

[0002] Autonomous driving provides a more convenient way of travel because it can improve road safety, reduce traffic accidents, improve traffic efficiency, and save energy. The end-to-end approach is a commonly used method in autonomous driving, especially in complex urban traffic. It simplifies the system architecture and improves overall performance by learning directly from perception data to control strategies. At present, deep reinforcement learning can achieve more efficient and flexible autonomous decision-making and control processes in end-to-end methods by learning complex decision-making and control strategies, directly mapping from perception data to behavioral outputs, especially in complex environments and tasks. Reinforcement learning accumulates experience to optimize strategies through continuous interaction with the environment.

[0003] Most existing end-to-end autonomous driving methods use a single agent to train in a single virtual environment. Although this method can simulate driving scenarios to a certain extent, the diversity and complexity of the environment often cannot fully reflect the changes in the real world. Due to the lack of sufficient scenarios, multi-agent interactions, and emergencies in the training data, the efficiency and quality of experience collection are limited, resulting in poor performance of the agent when faced with uncertainties in the real world. In addition, large-scale actual road testing is costly, which limits the way to gain experience. Therefore, how to effectively expand and diversify training scenarios in a virtual environment and improve the efficiency of experience collection has become a major challenge for the application of reinforcement learning in the field of autonomous driving.

[0004] For the reinforcement learning algorithm itself, the exploration-exploitation balance is the key, but in the autonomous driving scenario, the problem of insufficient exploration is particularly prominent. When encountering unknown driving environments or complex scenarios, existing algorithms usually tend to use previously learned suboptimal strategies, which makes it difficult for them to jump out of the local optimal solution. Especially when faced with sparse rewards or dynamic environments, the exploration ability of reinforcement learning algorithms is insufficient, and the intelligent agent is prone to fall into the trap of "premature convergence". In addition, because exploring new strategies may be accompanied by high risks, especially in applications such as autonomous driving that require extremely high safety, intelligent agents are more inclined to conservative strategies, further exacerbating the lack of exploration. Therefore, how to enhance the exploration ability of intelligent agents, balance exploration and safety, and avoid falling into local optimality is another important problem of reinforcement learning in autonomous driving decision-making. Summary of the invention

[0005] The purpose of the present invention is to propose a proximal policy optimization method for autonomous driving decision-making based on distributed exploration enhancement, aiming to improve the training efficiency and exploration strategy of deep reinforcement learning in autonomous driving, accelerate the learning process through distributed computing, and improve the exploration mechanism to more comprehensively cover the state space, improve the optimization effect of the strategy, so that the vehicle can use the proximal policy optimization method based on distributed exploration enhancement to effectively respond to changes in the urban traffic environment and maintain good obstacle avoidance performance.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] A method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement includes the following steps:

[0008] Step 1. The autonomous driving vehicle obtains environmental observation information in the CARLA simulation platform, including RGB images collected by the on-board camera, autonomous driving vehicle information, and path point sequence;

[0009] The RGB images collected by the on-board camera contain elements of roads, obstacles, other vehicles, pedestrians, and traffic signs;

[0010] The autonomous driving vehicle information includes the position, direction and speed of the autonomous driving vehicle; the waypoint sequence is the waypoint sequence provided by the built-in path planner of the CARLA simulation platform, along which the vehicle will travel;

[0011] The image perception module is used to extract the potential features of environmental observation information for use by the reinforcement learning decision module;

[0012] Step 2. Design a distributed architecture for the reinforcement learning decision module of the autonomous vehicle, which uses multiple agents to explore and learn in the environment in parallel and multiple workers to jointly process tasks; where the agent represents the autonomous vehicle in the autonomous driving task;

[0013] Step 3. Based on the distributed architecture in step 2, design the core part of the reinforcement learning decision module, including defining the state space based on the potential features extracted by the image perception module, and defining the action space and reward function of the autonomous driving vehicle;

[0014] Step 4. Design an exploration enhancement mechanism based on the random network distillation algorithm, introduce the exploration enhancement mechanism into the reinforcement learning decision module in the autonomous driving vehicle, and improve the proximal strategy optimization algorithm used by the reinforcement learning decision module;

[0015] Step 5. Apply the improved proximal strategy optimization algorithm to the actual urban traffic environment to ensure that the vehicle completes the driving task according to the preset path points, and monitor the surrounding traffic conditions in real time during the vehicle's driving process to improve the obstacle avoidance capability.

[0016] The present invention has the following advantages:

[0017] As described above, the present invention relates to a method for optimizing proximal strategies for autonomous driving decisions based on distributed exploration enhancement. Among them, the image perception module introduced in the present invention extracts spatial and channel features in the image through a dual attention mechanism (position attention and channel attention), which can effectively capture the relationship between visual information and control information, thereby focusing on environmental information that is critical to decision-making. This feature extraction method improves the ability of the reinforcement learning decision module to understand complex observations, and helps autonomous driving vehicles make more accurate decisions in dynamic environments. Secondly, the present invention adopts a distributed architecture, using multiple worker nodes for parallel processing to improve decision-making efficiency and response speed; the learner summarizes and optimizes the global strategy to ensure that the training process converges quickly to the optimal solution; in addition, the distributed architecture adopts a synchronous gradient transfer protocol to ensure the consistency of global strategy updates and effectively reduce local strategy deviations caused by communication delays. This distributed architecture improves the training speed and performance of reinforcement learning in complex environments. In addition, the present invention introduces an exploration enhancement mechanism, which quantifies the novelty of the environment through the random network distillation (RND) method, thereby improving the exploration ability of the agent. The RND method evaluates the familiarity of the state by comparing the output difference between the target network and the prediction network. The larger the difference, the more novel the state, thereby generating intrinsic rewards. The intrinsic reward guides the agent to pay attention to unfamiliar states during the exploration process and avoid falling into local optimality. At the same time, combined with external rewards, it enhances the exploration depth and breadth of the reinforcement learning decision module in complex environments and promotes strategy optimization. The present invention significantly accelerates the training process, improves the exploration efficiency, and comprehensively improves the decision quality and reliability of the autonomous driving system by applying the proximal strategy optimization method of autonomous driving decision-making based on distributed exploration enhancement in the actual urban traffic environment. The urban traffic environment is complex and diverse, including straight roads, right and left turn intersections, intersections, roundabouts, congested areas, pedestrian crossings, non-motorized vehicle lanes and bus stations, accompanied by high-density traffic flow, random pedestrians and non-motorized vehicles, signal light systems and emergency situations (such as sudden stops or obstacles). The method of the present invention ensures that the autonomous driving vehicle can follow the traffic rules while driving according to the preset path points, and show good path completion effect and obstacle avoidance ability in the changeable urban traffic, so as to achieve stable and safe driving performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1is a flow chart of a method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement in an embodiment of the present invention;

[0019] Figure 2 is a network structure diagram of an image perception module in an embodiment of the present invention;

[0020] Figure 3 It is an architecture diagram of a proximal strategy optimization method based on distributed exploration enhancement in an embodiment of the present invention;

[0021] Figure 4 It is a processing flow chart of the exploration enhancement mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0023] Example 1

[0024] The present invention describes a method for optimizing proximal policies for autonomous driving decisions based on distributed exploration enhancement, in which a distributed architecture of a reinforcement learning decision module is designed, wherein the distributed architecture utilizes parallel worker nodes to improve decision efficiency and response speed, and achieves rapid convergence through global policy optimization of the learner, ensuring consistent updates to reduce deviations caused by delays. In addition, the present invention combines the random network distillation method with the exploration enhancement mechanism to generate intrinsic rewards, guide the agent to focus on novel states during the exploration process, improve the exploration depth, and effectively optimize the strategy. In addition, the present invention also introduces an image perception module, which strengthens the extraction of the association between visual and control information, and improves the vehicle's ability to understand key environmental information.

[0025] like Figure 1 As shown, the proximal strategy optimization method for autonomous driving decision-making based on distributed exploration enhancement includes the following steps:

[0026] Step 1. The autonomous driving vehicle obtains environmental observation information in the CARLA simulation platform, which includes three parts: RGB images collected by the on-board camera, autonomous driving vehicle information, and path point sequence.

[0027] RGB images collected by vehicle-mounted cameras: RGB images obtained using a forward-looking camera, which mainly contain elements such as roads, obstacles, other vehicles, pedestrians, and traffic signs.

[0028] Autonomous driving vehicle information: including the position, orientation and speed of the autonomous driving vehicle; Waypoint sequence: the waypoint sequence provided by the built-in path planner, along which the vehicle will travel.

[0029] At the same time, the present invention introduces an image perception module, whose structure is as follows: Figure 2The present invention extracts potential features of environmental observation information through an image perception module for use by a reinforcement learning decision module of an autonomous driving vehicle.

[0030] This step 1 includes the following steps:

[0031] Step 1.1. Preprocessing of environmental observation information.

[0032] First, the autonomous driving vehicle information and path point sequence are preprocessed into a path image of the same size as the RGB image captured by the vehicle camera. Secondly, the autonomous driving vehicle is fixed at the center of the path image, the coordinates of each path point are calculated, and two adjacent path points are connected with a line segment. Then, the path image and the RGB image captured by the vehicle camera are spliced ​​on the channel. The path point sequence is the path point sequence provided by the built-in path planner of the CARLA simulation platform, and the autonomous driving vehicle information is obtained in real time through various sensors and systems in the simulation environment.

[0033] Step 1.2. The image perception module extracts latent features of environmental observation information.

[0034] like Figure 2 As shown in the figure, the image perception module uses a dual attention network as the backbone network, and extracts features containing spatial and channel relationships from observations by utilizing position attention and channel attention.

[0035] Subsequently, the self-attention mechanism is used to bridge visual information (such as roads, obstacles, traffic signs, etc.) with control information (such as driving decisions such as acceleration, braking, and steering), focusing on information that is critical to vehicle decision-making (for example, obstacles in the middle of the road have a greater impact on the decision than obstacles on both sides of the road).

[0036] The path image and RGB image spliced ​​in step 1.1 are used for feature extraction using the image perception module, with the goal of providing representative features of the original complex observations for the reinforcement learning decision module in step 2.

[0037] Step 2. Design a distributed architecture for the reinforcement learning decision module of the autonomous vehicle, which uses multiple agents to explore and learn in the environment in parallel and multiple workers to jointly process tasks.

[0038] like Figure 3 As shown, step 2 includes the following steps:

[0039] Step 2.1. The architecture used by the reinforcement learning decision module is a learner-worker distributed architecture, which uses multiple worker nodes for parallel processing to improve decision efficiency and response speed.

[0040] The learner is responsible for processing data and updating model parameters, while the worker is responsible for executing tasks, collecting data, and providing local gradients. The collaborative work of the two enables the distributed architecture to effectively train models in complex tasks.

[0041] Step 2.2. Design the learner in the distributed architecture of the reinforcement learning decision module.

[0042] In the distributed architecture, the learner acts as the central optimizer, responsible for synchronously aggregating gradient data from all worker nodes and updating parameters to optimize the global strategy.

[0043] By coordinating feedback from multiple workers, the learner ensures that the global policy converges quickly to the optimal solution.

[0044] Step 2.3. Design the workers in the distributed architecture of the reinforcement learning decision module.

[0045] The worker acts as an actor in the environment, interacting with the environment through strategies to calculate intrinsic rewards, extrinsic rewards, and advantage functions; the worker then stores the collected experience data in its own experience buffer.

[0046] Based on the collected empirical data, the worker calculates the gradient and transfers it to the learner's gradient buffer; the worker then waits for the learner to update the parameters and repeats the training process until the training round ends.

[0047] Step 2.4. Design the communication relationship between the learner and the worker in the distributed architecture of the reinforcement learning decision module.

[0048] The distributed architecture is based on a synchronous gradient transfer protocol, which allows each computing node to synchronously report its local gradient, thereby ensuring the consistency of global policy updates and reducing local policy deviations caused by communication delays.

[0049] Step 3. Based on the distributed architecture of step 2, design and improve the core components of the reinforcement learning decision module, which includes defining the state space based on the potential features extracted by the image perception module in step 1, and defining the action space and reward function of the autonomous driving vehicle.

[0050] Step 3.1. Based on the image perception module in step 1.2, the potential features of the environmental observation information are extracted and the state set of the reinforcement learning decision module in the autonomous driving vehicle is defined.

[0051] The state set of the reinforcement learning decision module mainly includes two aspects:

[0052] On the one hand, the image perception module extracts potential features of environmental observation information, which include spatial information, channel information, vehicle behavior characteristics and environmental context information, which help the vehicle make correct decisions in complex environments.

[0053] On the other hand, there is the autonomous driving vehicle information obtained in step 1, including the linear speed, route deviation and distance of the autonomous driving vehicle, as well as the steering, throttle and braking values ​​of the autonomous driving vehicle.

[0054] The state set of the reinforcement learning decision module is defined by the latent features and the autonomous driving vehicle information.

[0055] Step 3.2. The reinforcement learning decision module generates executable actions for the autonomous vehicle through the policy network.

[0056] The policy network outputs a probability distribution through the state set, indicating the execution probability of each possible action in that state. The autonomous vehicle selects specific actions based on this probability distribution, thereby affecting the environment and proceeding to the next step of learning.

[0057] The action set of the autonomous driving vehicle is defined as: ;in, Represents discrete actions. Indicates turning, Indicates the throttle. Indicates brakes. It is a discrete value between 0 and 1, which is used to control the steering angle, acceleration force and braking force of the vehicle respectively.

[0058] Step 3.3. To achieve safe driving in urban traffic, the reinforcement learning decision module needs to design an external reward function. The external reward function provides feedback in the reinforcement learning decision module to guide the autonomous driving vehicle to optimize action decisions to achieve the mission goal and maximize the cumulative reward. The external reward function includes sparse reward function and dense reward function.

[0059] Among them, the sparse reward function is used to encourage the agent to explore over a long period of time and implement effective long-term strategies; the dense reward function is used to instantly correct the agent's behavioral problems at each moment.

[0060] The intensive reward setting is as follows:

[0061] (1) To ensure that the autonomous vehicle remains within the predefined route, the Momentary deviation reward , the formula is as follows:

[0062] ;

[0063] in, Indicates the deviation distance of the vehicle. Indicates the maximum acceptable deviation distance.

[0064] (2) To ensure that the heading of the autonomous vehicle is consistent with the predetermined direction, the Angle Rewards of Moment , and its expression formula is as follows:

[0065] ;

[0066] in, Indicates the vehicle's deviation angle, Indicates the maximum acceptable deviation angle.

[0067] (3) The speed setting of the autonomous vehicle needs to take into account the requirements of different driving conditions. The speed reward function at each moment is designed for two scenarios: with obstacles and without obstacles.

[0068] When there is an obstacle ahead, Speed ​​bonus of the moment The calculation of is as follows:

[0069] ;

[0070] in, Indicates the distance to the obstacle (unit: meter), Indicates the driving speed of the vehicle (unit: meter per second (m / s)), is the maximum speed, is the expected speed.

[0071] When there are no obstacles ahead, The speed bonus at that moment is calculated as follows:

[0072] ;

[0073] in, Indicates the minimum speed (unit: meter per second (m / s)).

[0074] The sparse reward function is defined as follows:

[0075] (1) Collision with a static object, the reward function is set to ;

[0076] (2) Collision with pedestrians or other vehicles, the reward function is set to ;

[0077] (3) The vehicle deviates from the planned route, and the reward function is set to ;

[0078] (4) The vehicle fails to stay in the designated lane, and the reward function is set to ;

[0079] (5) When the vehicle exceeds the maximum allowed speed, the reward function is set to ;

[0080] (6) When the vehicle is blocked, the reward function is set to ;

[0081] (7) When the route is successfully completed, the reward function is set to .

[0082] Give extrinsic rewards The complete expression is:

[0083] .

[0084] in, Indicates deviation from the reward, represents the angle reward, represents the speed reward, Represent the weights of deviation reward, angle reward, speed reward and sparsity reward respectively.

[0085] Step 4. Design an exploration enhancement mechanism and introduce the reinforcement learning decision module in the autonomous driving vehicle, and improve the proximal policy optimization algorithm (PPO) used by the reinforcement learning decision module to enhance the exploration ability and learning efficiency of the intelligent agent.

[0086] like Figure 4 As shown, step 4 includes the following steps:

[0087] Step 4.1. Design an exploration enhancement mechanism. The exploration enhancement mechanism uses the Random Network Distillation (RND) method. The random network distillation method generates the target output by using a fixed random network and trains a learnable prediction network to approximate the output, thereby quantifying the novelty of the environment and improving the exploration ability of the agent.

[0088] Step 4.2. Introduce the exploration enhancement mechanism into the reinforcement learning decision module.

[0089] Set target network and prediction network .

[0090] The target network is randomly initialized, has fixed parameters, and outputs a scalar value based on the state input. The prediction network is trained to predict the output of the target network as accurately as possible. The prediction network output is also a scalar value.

[0091] When the agent encounters a state in the environment, the state will be processed simultaneously by the target network and the prediction network. The difference in the output values ​​of the two networks is used to reflect the familiarity with the current state. The more familiar the state is, the smaller the difference between the two.

[0092] Step 4.3. Based on the target network and prediction network designed in step 4.2, the difference between the output values ​​of the two is used to reflect the familiarity of the state. The difference between the two is used as the intrinsic reward in the reinforcement learning decision module.

[0093] The intrinsic reward is determined by the difference between the output of the target network and the prediction network, which is also used to train the prediction network, guiding the prediction network to gradually approach the output of the target network; Moment intrinsic reward function The definition is as follows:

[0094] ;

[0095] in, Represents the output of the prediction network Status at all times scalar value of , Represents the output of the target network Status at all times Scalar value.

[0096] Step 4.4. Based on the intrinsic reward obtained in step 4.3, combine the intrinsic reward with the extrinsic reward designed in step 3.3.

[0097] At every moment , the total reward obtained by the agent It is expressed as:

[0098] ;

[0099] in and Represent the weights of intrinsic rewards and extrinsic rewards, respectively, to balance the ratio of the two rewards. Represents every moment , the intrinsic reward obtained by the agent, Represents every moment , the extrinsic reward obtained by the agent.

[0100] The process of improving the proximal policy optimization algorithm used by the reinforcement learning decision module is as follows:

[0101] I.1. Design of distributed architecture processing flow of proximal strategy optimization algorithm:

[0102] The learner first generates the initial policy parameters , and distributes these parameters to each worker; after receiving these policy parameters, each worker creates an autonomous driving vehicle to interact with the environment and execute actions To respond to the status So as to get the corresponding next moment state , and collect corresponding external rewards , and according to the next moment state Calculating Intrinsic Rewards .

[0103] The learner usually runs on the CPU, and its main task is to handle parameter updates and gradient calculations, which are related to the model training process, including optimizing network parameters, calculating gradients, and updating the model through back propagation. Since the training process usually requires a lot of calculations, the hardware configuration of the learner needs to have high-performance computing power and large memory to efficiently perform gradient calculations and parameter updates.

[0104] The worker usually runs on the GPU, and its task is to interact with the environment, collect status and rewards, etc. The worker interacts with the environment through multiple parallel environment instances (such as running multiple CARLA simulation platform instances at the same time), obtains experience data (status, action, reward, etc.), and transmits this data to the learner for model training.

[0105] I.2. Design the processing flow of the internal exploration enhancement mechanism in the proximal strategy optimization algorithm;

[0106] During intrinsic reward processing, when the autonomous vehicle encounters a state in the environment When , the state is passed as input to the target network and the prediction network respectively; the scalar value of the target network output is , the scalar value of the prediction network output is , calculate the intrinsic reward If the intrinsic reward is large, it means that the next state It is a novel state that the autonomous vehicle has not fully explored; conversely, a smaller difference indicates that the state is more familiar and the intrinsic reward is reduced; the autonomous vehicle uses this intrinsic reward to adjust its strategy and encourage it to explore more unknown or rare states.

[0107] After the intrinsic reward is calculated, at each moment , the total reward obtained by the autonomous vehicle is expressed as .

[0108] I.3. Design the policy optimization and gradient calculation of the proximal policy optimization algorithm in the reinforcement learning decision module, calculate the advantage function according to the processing flow of the exploration enhancement mechanism described in step I.2 and the overall reward obtained;

[0109] The advantage function measures the superiority of the current action relative to the average action, helping the reinforcement learning agent to evaluate and select strategies more effectively, thereby improving learning efficiency.

[0110] The advantage function is calculated using the generalized advantage estimation (GAE) method. ;

[0111] ;

[0112] in express The total reward of the moment, represents the discount factor, It represents the value network; They represent the state and value network parameters respectively, and k represents the time step, which is used to define the number of time steps within which the cumulative reward and future value estimation are calculated.

[0113] The update of the value network uses a truncation function to control the update amplitude;

[0114] ;

[0115] in, represents the state value function estimate after applying a truncation function, which limits the current value function estimate With the previous value function estimate After the calculation is completed, the generated empirical data is obtained, including the state ,action Total Rewards , the next moment state And the advantage function .

[0116] Gradient calculation is to accumulate these experience data through continuous interaction with the environment and store them in the worker's experience cache; the worker applies the proximal strategy optimization algorithm to update the strategy based on the experience data in the experience cache and calculates the corresponding gradient .

[0117] The process of gradient calculation is to define the loss function and use the back propagation algorithm to calculate the gradient of the network parameters, so as to update the model to improve the strategy and decision-making ability. The loss function is as follows:

[0118] ;

[0119] in, represents the loss function, represents the number of samples that each worker first randomly samples from its experience cache, represents the importance weight, is a hyperparameter that controls the update amplitude, is the advantage function; the worker uses back propagation to calculate the gradient of the loss function for each parameter and passes the gradient to the learner to complete the parameter update.

[0120] After perfecting the proximal policy optimization algorithm used in the reinforcement learning decision module, the effectiveness of the proximal policy optimization method enhanced by distributed exploration is evaluated by path completion rate and obstacle avoidance rate. Specifically, a simulation experiment is designed to evaluate the effectiveness of the proximal policy optimization algorithm enhanced by distributed exploration by path completion rate and obstacle avoidance rate.

[0121] First, the vehicle needs to drive smoothly along the designated lane and make turns when necessary according to traffic conditions or target paths. Second, the system needs to be able to avoid other vehicles to ensure collision avoidance and safe driving in complex urban road environments. Through visual analysis, the real-time and precise control advantages of the proximal policy optimization algorithm enhanced by distributed exploration in these two tasks are demonstrated. The simulation experiment process includes the following steps:

[0122] II.1. In the CARLA urban traffic scenario, ensure that the vehicle successfully completes the driving task according to the set path points.

[0123] By following preset waypoints, autonomous vehicles can use proximal policy optimization methods based on distributed exploration-enhanced autonomous driving decisions to cope with changes in urban traffic environments and ensure smooth completion of the entire driving route.

[0124] II.2. Specific obstacle avoidance scenarios are set up in the CARLA simulator to evaluate the vehicle’s performance in the following scenarios.

[0125] (1) In a busy city center, vehicles must give way to pedestrians who suddenly appear;

[0126] (2) At busy intersections, vehicles must deal with sudden lane changes by other vehicles;

[0127] (3) On the set short route, when the vehicle reaches the trigger point, a stationary tool vehicle is simulated and required to stop within a safe distance to avoid collision;

[0128] Autonomous driving vehicles can use proximal policy optimization methods for autonomous driving decisions based on distributed exploration enhancement to cope with changes in urban traffic environments and maintain good obstacle avoidance performance.

[0129] Step 5. After the exploration enhancement mechanism is introduced in step 4, the proximal strategy optimization algorithm used in the distributed architecture of the reinforcement learning decision module is finally applied to the actual urban traffic environment. The vehicle completes the driving task according to the preset path points; and during the driving process, the vehicle not only follows the planned path, but also monitors the surrounding traffic conditions in real time and takes avoidance measures to improve the obstacle avoidance capability, which is ultimately conducive to achieving the goal of safe and efficient autonomous driving.

[0130] The method of the present invention designs a distributed architecture of the reinforcement learning decision module of the autonomous driving vehicle, which significantly improves the training efficiency of the algorithm. Through parallelization and distributed processing, multiple autonomous driving vehicles can perform policy training simultaneously in different environments, share policy updates and experience data in real time, thereby accelerating the convergence speed of the algorithm and reducing the time cost required for independent training of a single autonomous driving vehicle. In addition, the distributed architecture allows the algorithm to operate in larger data sets and complex environments, further improving the adaptability and generalization ability of autonomous driving decisions. The present invention also combines an exploration enhancement module, which uses a random network distillation method to further enhance the diversity and effectiveness of exploration. The exploration enhancement module encourages autonomous driving vehicles to explore unseen states and behaviors by introducing randomness and additional reward signals, avoiding premature convergence to suboptimal strategies. In the distributed architecture, the exploration enhancement module realizes the collaborative work between autonomous driving vehicles, ensuring that the exploration process is more comprehensive on a global scale. After combining the exploration enhancement module, the proximal policy optimization algorithm adopted by reinforcement learning not only improves the training efficiency, but also enhances the robustness and generalization ability of the strategy, making it perform better in complex autonomous driving decision tasks. By combining the design of a distributed architecture with an exploration enhancement module, the present invention significantly accelerates the training process, improves exploration efficiency, and comprehensively improves the decision-making quality of the autonomous driving system, thereby improving the reliability and efficiency of autonomous driving decisions.

[0131] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A proximal strategy optimization method for autonomous driving decision-making based on distributed exploration enhancement, characterized in that: The steps include: Step 1. The autonomous driving vehicle obtains environmental observation information in the CARLA simulation platform, including RGB images collected by the on-board camera, autonomous driving vehicle information, and path point sequence; The RGB images collected by the on-board camera contain elements of roads, obstacles, other vehicles, pedestrians, and traffic signs; The autonomous driving vehicle information includes the position, direction and speed of the autonomous driving vehicle; the waypoint sequence is the waypoint sequence provided by the built-in path planner of the CARLA simulation platform, along which the vehicle will travel; The image perception module is used to extract the potential features of environmental observation information for use by the reinforcement learning decision module; Step 2. Design a distributed architecture for the reinforcement learning decision module of the autonomous vehicle, which uses multiple agents to explore and learn in the environment in parallel and multiple workers to jointly process tasks; The agent represents the autonomous vehicle in the autonomous driving task; Step 3. Based on the distributed architecture in step 2, design the core part of the reinforcement learning decision module, including defining the state space based on the potential features extracted by the image perception module, and defining the action space and reward function of the autonomous driving vehicle; Step 4. Design an exploration enhancement mechanism based on the random network distillation algorithm, introduce the exploration enhancement mechanism into the reinforcement learning decision module in the autonomous driving vehicle, and improve the proximal strategy optimization algorithm used by the reinforcement learning decision module; Step 5. Apply the improved proximal strategy optimization algorithm to the actual urban traffic environment to ensure that the vehicle completes the driving task according to the preset path points, and monitor the surrounding traffic conditions in real time during the vehicle's driving process to improve the obstacle avoidance capability.

2. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.

1. Preprocessing of environmental observation information; First, the autonomous driving vehicle information and path point sequence are preprocessed into a path image of the same size as the RGB image collected by the vehicle camera. Second, the autonomous driving vehicle is fixed at the center of the path image, the coordinates of each path point are calculated, and two adjacent path points are connected with line segments. Then the path image and the RGB image collected by the on-board camera are spliced ​​on the channel; Step 1.

2. Use the image perception module to extract the potential features of environmental observation information; The image perception module uses a dual attention network as the backbone network to extract features containing spatial and channel relationships from observations by utilizing position attention and channel attention. Then, a self-attention mechanism is used to bridge visual information with control information, focusing on information that is critical to vehicle decision-making; The path image and RGB image spliced ​​in step 1.1 are input into the image perception module, and feature extraction is performed using the image perception module to provide representative features of the original complex observations for the reinforcement learning decision module.

3. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.

1. The architecture used by the reinforcement learning decision module is a learner-worker distributed architecture, which uses multiple worker nodes for parallel processing to improve decision efficiency and response speed; Step 2.

2. Design the learner in the distributed architecture of the reinforcement learning decision module; In the distributed architecture, the learner acts as a central optimizer, responsible for synchronously aggregating gradient data from all worker nodes and updating parameters to optimize the global strategy. By coordinating feedback from multiple workers, the learner ensures that the global strategy can quickly converge to the optimal solution; Step 2.

3. Design the workers in the distributed architecture of the reinforcement learning decision module; The worker acts as an actor in the environment, interacting with the environment through strategies to calculate intrinsic rewards, extrinsic rewards, and advantage functions; the worker then stores the collected experience data in its own experience buffer; Based on the collected empirical data, the worker calculates the gradient and transfers it to the learner's gradient buffer; the worker then waits for the learner to update the parameters and then repeats the training process until the training round ends; Step 2.

4. Design the communication relationship between the learner and the worker in the distributed architecture of the reinforcement learning decision module; The distributed architecture is based on a synchronous gradient transfer protocol, which allows each worker to synchronously report its local gradient to ensure the consistency of global policy updates and reduce local policy deviations caused by communication delays.

4. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 1, characterized in that: The step 3 includes the following steps: Step 3.

1. Based on the potential features of the environmental observation information extracted by the image perception module in step 1, define the state set of the reinforcement learning decision module in the autonomous driving vehicle; The state set of the reinforcement learning decision module includes two aspects: on the one hand, the potential features of the environmental observation information extracted by the image perception module; on the other hand, the autonomous driving vehicle information obtained in step 1; Define the state set of the reinforcement learning decision module through the latent features and the autonomous driving vehicle information; Step 3.

2. The reinforcement learning decision module generates executable actions for the autonomous driving vehicle through the policy network; The policy network outputs a probability distribution through the state set, indicating the execution probability of each possible action in that state; the autonomous vehicle selects a specific action based on the probability distribution, thereby affecting the environment and proceeding to the next step of learning; The action set of the autonomous driving vehicle is defined as: ;in, Represent discrete actions; Indicates turning, Indicates the throttle. Indicates brakes; , , They are all discrete values ​​between 0 and 1, which are used to control the steering angle, acceleration force and braking force of the vehicle respectively; Step 3.

3. The reinforcement learning decision module designs an external reward function, including a sparse reward function and a dense reward function; Among them, the dense reward function is set as follows: (1) To ensure that the autonomous vehicle remains within the predefined route, the Momentary deviation reward , the formula is as follows: ； in, Indicates the deviation distance of the vehicle. Indicates the maximum acceptable deviation distance; (2) To ensure that the heading of the autonomous vehicle is consistent with the predetermined direction, the Angle Rewards of Moment , and its expression formula is as follows: ； in, Indicates the deviation angle of the vehicle. Indicates the maximum acceptable deviation angle; (3) The speed setting of the autonomous vehicle needs to take into account the requirements of different driving conditions. The speed reward function at each moment is designed for two scenarios: with obstacles and without obstacles; When there is an obstacle ahead, Speed ​​bonus of the moment The calculation of is as follows: ； in, Indicates the distance to the obstacle in meters. Indicates the driving speed of the vehicle in meters per second. is the maximum speed, is the expected speed; When there are no obstacles ahead, The speed bonus at that moment is calculated as follows: ； in, Indicates the minimum speed in meters per second; The sparse reward function is defined as follows: (1) Collision with a static object, the reward function is set to ; (2) Collision with pedestrians or other vehicles, the reward function is set to ; (3) The vehicle deviates from the planned route, and the reward function is set to ; (4) The vehicle fails to stay in the designated lane, and the reward function is set to ; (5) When the vehicle exceeds the maximum allowed speed, the reward function is set to ; (6) When the vehicle is blocked, the reward function is set to ; (7) When the route is successfully completed, the reward function is set to ; Give extrinsic rewards The complete expression is: ； in, Indicates deviation from the reward, represents the angle reward, represents the speed reward, Represent the weights of deviation reward, angle reward, speed reward and sparsity reward respectively.

5. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.

1. Design and explore the enhancement mechanism using random network distillation method; The random network distillation method generates the target output by using a fixed random network and trains a learnable prediction network to approximate the output, thereby quantifying the novelty of the environment and improving the exploration ability of the agent. Step 4.

2. Introduce the exploration enhancement mechanism into the reinforcement learning decision module; Set target network and prediction network ; The target network is randomly initialized, has fixed parameters, and outputs a scalar value based on the state input. The prediction network is trained to predict the output of the target network as accurately as possible. The prediction network output is also a scalar value. When the agent encounters a state in the environment, the state will be processed by the target network and the prediction network at the same time. The difference between the output values ​​of the two networks is used to reflect the familiarity with the current state. The more familiar the state is, the smaller the difference between the two. Step 4.

3. Based on the target network and prediction network designed in step 4.2, the difference between the output values ​​of the two is used to reflect the familiarity of the state; the difference between the two is used as the intrinsic reward in the reinforcement learning decision module; The intrinsic reward is determined by the difference between the output of the target network and the prediction network, which is also used to train the prediction network, guiding the prediction network to gradually approach the output of the target network; Moment intrinsic reward function The definition is as follows: ； in, represents a scalar value for the prediction network output, represents the scalar value of the target network output, Indicated in The state of the moment; Step 4.

4. Based on the intrinsic reward obtained in step 4.3, combine the intrinsic reward with the extrinsic reward; At every moment , the total reward obtained by the agent It is expressed as: ； in and Represent the weights of intrinsic rewards and extrinsic rewards, respectively, to balance the ratio of the two rewards. Represents every moment , the intrinsic reward obtained by the agent, Represents every moment , the extrinsic reward obtained by the agent.

6. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 5 is characterized in that: In step 4, the process of the proximal strategy optimization algorithm used by the reinforcement learning decision module is as follows: I.

1. Design the process flow of the distributed architecture of the proximal strategy optimization algorithm; The learner first generates the initial policy parameters , and distribute these parameters to each worker; After receiving these policy parameters, each worker creates an autonomous vehicle to interact with the environment by performing actions. To respond to the status So as to get the corresponding next moment state , and collect corresponding external rewards , and according to the next moment state Calculating Intrinsic Rewards ; I.

2. Design the processing flow of the internal exploration enhancement mechanism in the proximal strategy optimization algorithm; During intrinsic reward processing, when the autonomous vehicle encounters a state in the environment When , the state is passed as input to the target network and the prediction network respectively; the scalar value of the target network output is , the scalar value of the prediction network output is , calculate the intrinsic reward ; After the intrinsic reward is calculated, at each moment , the total reward obtained by the autonomous vehicle is expressed as ; I.

3. Design the policy optimization and gradient calculation process of the proximal policy optimization algorithm in the reinforcement learning decision module; According to the process flow of exploring the enhancement mechanism in step I.2 and the overall reward, the advantage function is calculated; specifically, the advantage function is calculated using the generalized advantage estimation GAE method ; ； in express The total reward of the moment, represents the discount factor, Represent the state and value network parameters respectively, and k represents the time step; It represents the output value of the value network; The update of the value network uses a truncation function to control the update amplitude; ； in, represents the state value function estimate after applying a truncation function, which limits the current value function estimate With the previous value function estimate changes between The function is used to restrict the input value to a specified range and truncate the value outside the range to the minimum or maximum value of the range. represents the hyperparameter that controls the update amplitude; After the advantage function is calculated, the generated empirical data is obtained. The empirical data includes the state ,action Total Rewards , the next moment state And the advantage function ; Gradient calculation is to accumulate experience data through continuous interaction with the environment and store it in the worker's experience cache; the worker interacts with the environment based on the data in the experience cache, and this experience data is used for gradient calculation and model update.

7. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 6 is characterized in that: In step I.3, the gradient calculation process is to define the loss function and use the back propagation algorithm to calculate the gradient of the network parameters, thereby updating the model to improve the strategy and decision-making ability. The loss function is as follows: ； in, represents the loss function, represents the number of samples that each worker first randomly samples from its experience cache, represents the importance weight, is a hyperparameter that controls the update amplitude, is the advantage function; the worker uses back propagation to calculate the gradient of the loss function for each parameter and passes the gradient to the learner to complete the parameter update.

8. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 1, characterized in that: In step 4, after perfecting the proximal policy optimization algorithm used by the reinforcement learning decision module, the effectiveness of the proximal policy optimization method enhanced by distributed exploration is evaluated by the path completion rate and obstacle avoidance rate.

9. The method for optimizing proximal strategy for autonomous driving decision-making based on distributed exploration enhancement according to claim 8, characterized in that: The step 4 comprises the following steps: II.

1. In the CARLA urban traffic scenario, ensure that the vehicle successfully completes the driving task according to the set path points; By following the preset waypoints, the autonomous vehicle can use the proximal policy optimization method based on distributed exploration-enhanced autonomous driving decision-making to cope with changes in the urban traffic environment and ensure the smooth completion of the entire driving route; II.

2. Set up obstacle avoidance scenarios in the CARLA simulator to evaluate the vehicle’s performance in the following scenarios; (1) In a busy city center, vehicles must give way to pedestrians who suddenly appear; (2) At busy intersections, vehicles must deal with sudden lane changes by other vehicles; (3) On the set short route, when the vehicle reaches the trigger point, a stationary tool vehicle is simulated and required to stop within a safe distance to avoid collision; Autonomous driving vehicles can use proximal policy optimization methods for autonomous driving decisions based on distributed exploration enhancement to cope with changes in urban traffic environments and maintain good obstacle avoidance performance.

Images