Reinforcement learning driving decision-making method based on adaptive dynamic topological structure

Abstract

The invention relates to a reinforcement learning driving decision-making method based on a self-adaptive dynamic topological structure, and belongs to the technical field of intelligent driving. According to the method, a reinforcement learning parameter optimization network and a Type-balance experience pool are established, and samples are collected from the experience pool for learning of the reinforcement learning parameter optimization network; vehicle control parameters are generated according to the environment parameters collected in real time; the driving decision parameter is optimized through the reinforcement learning parameter optimization network, the reward values of the optimized driving decision parameter and the experience parameter are compared through the reward prediction network, the reward values are compared to select the better driving decision parameter, and the real vehicle control parameter is further calculated. According to the invention, motion decision-making in an intelligent driving scene is carried out through the self-adaptive dynamic topological structure framework based on reinforcement learning, the path planning capability of a reinforcement learning algorithm can be improved, and the transparency of an automatic driving decision-making process is improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology and relates to a reinforcement learning driving decision-making method based on an adaptive dynamic topology structure. Background Art

[0002] Vehicle motion planning is a crucial component of the development of intelligent driving technology. Local motion planning typically involves creating a decision module that allows the vehicle to adaptively make local motion decisions based on road conditions and the behavior of other vehicles (autonomous vehicles). These decisions are primarily categorized into rule-based and learning-based approaches.

[0003] Rule-based methods have the characteristics of clear logic, model interpretability and scalability. However, they have limited adaptability to complex driving scenarios because they rely on a limited set of rule parameters to make decisions in dynamic environments. Rule-based methods usually divide continuous decision-making behaviors into discrete driving behaviors by adopting predefined rules. The vehicle's state and environmental characteristics are then evaluated according to these rules to make the final driving decision. Finite state machine (FSM) is one of the most representative rule-based decision-making methods. The key components of FSM include state, event, transition and action. Each state responds to a specific event, performs a corresponding action, and transitions to a new state. Rule-based methods face greater challenges in complex driving conditions. For example, traversing a large number of states may result in significant computational overhead. In addition, state division and conflict resolution between states remain key obstacles to improving the efficiency and reliability of these methods.

[0004] Reinforcement learning, a technology with great development potential, is known for its self-learning ability, high decision-making flexibility, and ability to deeply traverse specific scenarios. In recent years, it has been increasingly used in driving decision-making and has shown good performance. Compared with rule-based decision-making methods, reinforcement learning decision-making has greater decision-making flexibility and environmental adaptability, and can also simplify the scale of decision-making modules by leveraging network structures. However, learning-based methods face the challenge of acting in scenarios outside of the training data, which limits their safety in different scenarios. To address this problem, existing technologies have explored hybrid methods that combine rule-based and learning-based methods to enhance decision-making safety through risk constraints and takeover mechanisms. For example, Yang et al. (R. Yang, Z. Li, B. Leng, and L. Xiong, “Safe reinforcement learning for autonomous vehicles to make lane-change decisions: Constraint based on Incomplete Information Game Theory,” in 2023 7th CAA International Conference on Vehicular Control and Intelligence (CVCI): IEEE, pp. 1-6, 2023.) proposed a safe RL framework based on game theory that combines a reinforcement learning network and a safety constraint module. The safety constraint module ensures risk assessment and driving safety during and after training, and it also takes over decision-making when RL brings driving risks. Similarly, Xu et al. (C.Xu, W.Zhao, J.Liu, C.Wang, and C.Lv, “An integrated decision-making framework for highway autonomous driving using combined learning and rule-based algorithm,” IEEE Transactions on Vehicular Technology, vol.71, no.4, pp.3621-3632, 2022.) proposed an integrated decision-making framework for highway automation that combines RL with imitation learning while leveraging rule-based corrective actions to address flawed decisions and improve safety. Despite these advances, learning-based methods suffer from poor interpretability. Even with the integration of rule-based auxiliary tools, the decision-making process of learning-based algorithms often remains opaque, which poses challenges to the verification and deployment of safety-critical scenarios. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a reinforcement learning driving decision-making method based on an adaptive dynamic topological structure. By using a reinforcement learning based self-adaptive dynamic topological structure (RL-SDTS) framework to perform motion decisions in intelligent driving scenarios, the path planning capability of the reinforcement learning algorithm is improved, and the transparency of the reinforcement learning algorithm is enhanced.

[0006] To achieve the above objectives, the present invention provides a reinforcement learning driving decision-making method based on an adaptive dynamic topology structure, which includes:

[0007] Build a reinforcement learning parameter optimization network and a typed-balanced experience pool. Run a pre-trained pseudo-expert agent in the environment to generate experience data including environmental parameters and corresponding driving decision parameters. Store the experience data in the typed-balanced experience pool. Collect samples from the typed-balanced experience pool for the reinforcement learning parameter optimization network to learn.

[0008] Generate driving decision parameters based on real-time collected environmental parameters, and calculate vehicle control parameters based on the driving decision parameters through a PID controller;

[0009] The generated driving decision parameters are input into the reinforcement learning parameter optimization network for optimization. The real-time collected environmental parameters are input into the pre-trained pseudo-expert body to generate comparative driving decision parameters, and the vehicle control parameters are calculated based on the driving decision parameters using a PID controller.

[0010] The optimized driving decision parameters and real-time collected environmental parameters are input into the reward prediction network. The predicted reward values ​​are output respectively by the reinforcement learning agent reward predictor and the pseudo-expert reward predictor. The reward values ​​are compared to select the better driving decision parameters, and the real vehicle control parameters are calculated by the PID controller.

[0011] Furthermore, the reinforcement learning parameter optimization network includes a spatiotemporal state feature encoder and a reinforcement learning agent; the spatiotemporal state feature encoder obtains multiple continuous state slices from the real-time collected environmental parameters for encoding and attention mechanism calculation, and then transmits them to the reinforcement learning agent, and the reinforcement learning agent optimizes the driving decision parameters according to the calculation results.

[0012] Among them, the reward item of the reinforcement learning agent at each time step includes the TTC item r TTC, collision term r collision , speed term r velocity and lane change term r lane-changing , then the action reward at each time step is r step =r TTC +r collision +r velocity +r lane-changing +r waiting ; Considering T continuous state slices, the decision reward of the actor-critic reinforcement learning framework at the current time step is Among them, t = 1 represents the current time step, t = T represents the previous T-1 time step, and β is the attenuation coefficient.

[0013] Furthermore, the driving decision parameters and the comparative driving decision parameters both include the lane change decision value, the safe lane change distance, the expected lane change speed when changing lanes, the lane keeping speed when not changing lanes, the driving speed during the waiting period for lane change, and the braking distance; the vehicle control parameters and the comparative vehicle control parameters both include speed, throttle, and brake.

[0014] Furthermore, in the reward prediction network, the reinforcement learning agent reward predictor predicts the reward r after the reinforcement learning agent performs an action. pred (s,p RL )=f pred (s,p RL )+σv dts , the pseudo-expert reward predictor predicts the reward r after the pseudo-expert performs an action pred (s,p exp )=f pred (s,p exp )+σv dts ′, where s represents the environmental parameters collected in real time, p RL represents the optimized driving decision parameters, v dts Indicates that based on p RL The calculated speed, p exp represents the driving decision parameters generated by the pseudo-expert, v dts ′ represents the value based on p exp The calculated speed, σ represents the adjustment parameter; select r pred (s,p RL ) and r pred (s,p exp ) is used as the final output.

[0015] Furthermore, the Typed-balanced experience pool includes road-keeping samples, lane-changing samples, and low-reward samples. The maintenance mechanism for road-keeping and lane-changing samples is a first-in, first-out mechanism, while the maintenance mechanism for low-reward samples is to prioritize low-reward samples. Each time a sample is collected from the Typed-balanced experience pool, the sampling ratios of the three types of samples are as follows:

[0016]

[0017] θ3=ρ

[0018] Where θ1 is the sampling ratio of road-keeping samples, θ2 is the sampling ratio of lane-changing samples, and θ3 is the sampling ratio of low-reward samples. N1 and N2 are the numbers of road-keeping samples and lane-changing samples, respectively. ρ is the preset ratio parameter.

[0019] The sampling method for road keeping samples and lane changing samples is based on sample priority, and the sampling method for low reward value samples is random sampling.

[0020] Furthermore, for any of the road keeping samples and lane changing samples, assuming that the index set of all samples of this type is I, I is sorted in descending order according to the reward value corresponding to the sample index, and I′={i (1) ,i (2) ,...,i (N)}, index i (k) The corresponding reward is r (k) , k=1,2,...,N, where r (1) ≥r (2) ≥...≥r (N) , N is the total number of samples; for the sorted index i (k) Calculating priority weights α is a parameter used to control the priority, converting the priority weight into sampling probability

[0021] From the set {i (1) ,i (2) ,...,i (N)}, and the sampling results are recorded as Each position l m The probability of selection is π lm , m=1,2,...,M, then the sampling process can be expressed as:

[0022]

[0023] in, According to the probability distribution Repeat the sampling M times, sampling a category index l∈{1,2,…,N} independently each time.

[0024] The beneficial effects of the present invention are: the present invention proposes a reinforcement learning driving decision-making method based on an adaptive dynamic topology structure, which makes motion decisions in intelligent driving scenarios through an adaptive dynamic topology structure framework based on reinforcement learning, thereby improving the path planning ability of the reinforcement learning algorithm. In this method, the reinforcement learning algorithm does not directly output the decision-making actions of the intelligent agent, but optimizes the dynamic topology structure parameters so that it can output adaptive decision-making actions, and the overall structure of the dynamic topology structure is interpretable, thereby improving the transparency of the autonomous driving decision-making process.

[0025] In addition, the present invention sets up a typed-balanced experience pool and a pre-trained pseudo-expert body. The pre-trained pseudo-expert body runs in the environment to generate experience data including environmental parameters and corresponding driving decision parameters, and stores them in the typed-balanced experience pool. Samples are collected from the typed-balanced experience pool for the reinforcement learning parameter optimization network to learn. At the same time, a method for sampling samples from the experience pool and a method for maintaining the experience pool are designed, so that the intelligent agent can quickly learn all sample types, thereby improving the adaptability and reaction speed of the intelligent agent in different environments and improving the path planning ability.

[0026] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0028] Figure 1 This is a schematic diagram of the framework structure of the adaptive dynamic topology structure based on reinforcement learning;

[0029] Figure 2 It is a schematic diagram of dynamic topology structure;

[0030] Figure 3 Schematic diagram of the network structure for reinforcement learning parameter optimization;

[0031] Figure 4 This is a schematic diagram of the basic driving test on the composite road section;

[0032] Figure 5 The results of basic driving tests on composite road sections are shown in Figure 2. Figure 5 (a) is the average reward curve, Figure 5 (b) is the cumulative reward curve;

[0033] Figure 6 This is a schematic diagram of the global path navigation experiment;

[0034] Figure 7 is the global path navigation experiment result, Figure 7 (a) is the average reward curve, Figure 7 (b) is the cumulative reward curve. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0036] An embodiment of the present invention provides a reinforcement learning driving decision method based on adaptive dynamic topology structure, which generates driving decisions through a reinforcement learning-based adaptive dynamic topology structure (RL-SDTS) framework, such as Figure 1 The RL-SDTS framework is shown, consisting of five modules: a dynamic topology (DTS), a reinforcement learning parameter optimization network, a reward prediction network, imitation learning, and a typed-balanced experience pool. The dynamic topology is a motion decision framework based on dynamic environment parameters, used to generate driving decisions for the ego vehicle. The reinforcement learning parameter optimization network, consisting of a spatiotemporal state feature encoder and a deep network, optimizes the parameters of the dynamic topology. The reward prediction network predicts the rewards associated with actions taken by the agent and pseudo-expert strategies, selecting the optimal action based on reward comparison. Imitation learning accelerates the agent's learning by inputting pseudo-expert experience into the agent's experience pool. The typed-balanced experience pool labels decision actions, determining the sampling ratio based on the proportion of data with each label in the pool. The sample sampling priority is determined by the corresponding reward value, and a certain proportion of low-reward samples are retained as counterexamples for learning. Each module is described in detail below.

[0037] 1. Dynamic topology

[0038] The dynamic topology structure includes data acquisition module, decision module and control module. Based on the concept of hierarchical reinforcement learning, the dynamic topology structure decomposes the driving task into multiple subtasks that operate in a collaborative manner, including detection, obstacle and lane marking calculation, behavior decision, safety constraints and motion control, etc. Figure 2 shown.

[0039] The data acquisition module performs detection and obstacle and lane marking calculation tasks. The detection task defines the environmental data acceptance range and performs obstacle and lane line detection. The obstacle and lane marking calculation task, based on vehicle and road information acquired from the environment, detects and marks the environment within the own lane and adjacent lanes based on lane position on structured roads with parallel lane lines. On roads with intersecting lane lines or no lane lines, the obstacle and lane marking calculation task uses an ortho-hexagon structure to acquire environmental information from six directions.

[0040] The decision-making module performs behavioral decisions and safety constraint tasks, using environmental information acquired by the data acquisition module to determine safety conditions and make adaptive environmental decisions. The module's main driving decision parameters are input based on a reinforcement learning parameter optimization network. These driving decision parameters are inherent to the decision-making module and are manually initialized. Dynamic optimization is then performed based on the reinforcement learning parameter optimization network, and the decision action is then output by the parameter-optimized decision-making module. The parameters involved include:

[0041] 1)d turn , which is an adjustable threshold used to determine whether to perform a lane change operation, and is expressed as:

[0042]

[0043] Among them, 0<w1<1, w1 is the dynamic coefficient of reinforcement learning output, Curvature(wpt HV ) represents multiple planning points wpt based on the vehicle in the neighborhood HV The curvature of the route is estimated by a quadratic polynomial, d lane-obs Scope for decision making.

[0044] 2)d safe , is the safe lane change distance, which is expressed as:

[0045]

[0046] Among them, 0<w2<1, w2 is the dynamic coefficient of reinforcement learning output, v HV represents the speed of the vehicle, v desired represents the desired speed of the vehicle, Indicates the shortest braking distance, D safe For a safe distance.

[0047] 3)v lc , is the lane-changing speed expected by the ego vehicle when performing a lane-changing operation, which is expressed as:

[0048]

[0049] Among them, 0<μ1<1, μ1 is the dynamic coefficient of reinforcement learning output, v min is the minimum speed limit of the ego vehicle, v OV is the speed of other vehicles in the selected environment, and σ is a constant set according to the state of the surrounding vehicles.

[0050] 4) is the lane keeping speed of the vehicle without considering lane changing, which is expressed as:

[0051]

[0052] Among them, 0<μ2<1, 0<γ1<1, 0<γ2<1, TTC represents the time when the ego vehicle collides with other vehicles in the target lane of the ego vehicle, TTC th is the TTC threshold, 0<μ3<1, μ2, μ3, γ1, γ2 are the dynamic coefficients of reinforcement learning output, k2, k3, k4 are the hyperparameters set to facilitate the reinforcement learning parameter optimization network output calculation, v max is the maximum speed limit of the vehicle, and d is the distance between the vehicle and the nearest vehicle ahead.

[0053] 5) When the ego vehicle needs to change lanes according to the global route planning, the speed of the ego vehicle during the period of waiting for the lane change conditions to be met is It is expressed as:

[0054]

[0055] Among them, 0<γ3<1, γ3 is the dynamic coefficient of reinforcement learning output, k S It is a constant set according to the status of surrounding vehicles.

[0056] 6)d brake is the braking distance of the vehicle, by combining d brake , the ego vehicle can adjust the boundaries of safe driving actions, thereby increasing the operating limits while maintaining safety, d brake Expressed as:

[0057]

[0058] in, is the dynamic coefficient of reinforcement learning output, k6 and k7 are hyperparameters, Indicates the braking distance threshold, Depends on the driving mode of the vehicle. Indicates the shortest braking distance.

[0059] The control module performs motion control tasks, using a control mechanism based on lateral and longitudinal PID to achieve a smoother overall control output. This module is primarily used to implement three main functions: emergency braking, rear-end collision avoidance, and normal driving. The control module outputs control parameters such as speed, throttle, and brake lights, as shown in the following formula:

[0060] dts(s,p)=a=(v dts ,t dts ,b dts ) (7)

[0061] in, dts(·) represents the calculation process of the control module, s represents the environmental information obtained by the data acquisition module, and a represents the control parameters output by the control module, including the speed v dts , throttle t dts , brake b dts wait.

[0062] 2. Reinforcement Learning Parameter Optimization Network

[0063] like Figure 3 As shown in Figure 1, the reinforcement learning parameter optimization network consists of a spatial-temporal state feature encoder and a deep network. It receives environmental parameters collected by the data acquisition module, including relative spatial position and relative velocity. Furthermore, to capture the vehicle's temporal state information, the spatial-temporal state feature encoder further expands the spatial position and relative velocity information into time series information by introducing T consecutive state slices.

[0064] The deep network architecture adopts the actor-critic reinforcement learning framework, that is, the reinforcement learning agent, which passes the spatiotemporal state information obtained by the data acquisition module to the spatial-temporal state feature encoder. After encoding by the encoder and calculation by the attention mechanism, the state feature value is output to the deep network, and finally the driving decision parameters of the decision module with a dynamic topology structure are output.

[0065] The reward setting for a single time step includes the TTC (Time to collision) term r TTC , collision term r collision , speed term r velocity and lane change term r lane-changing . Set them as follows:

[0066]

[0067] Among them, δ1>0, δ2>0, δ3>0, δ4>0, δ5>0, δ1, δ2, δ3, δ4, δ5 are the preset reward coefficients, t safe is the preset safe collision time, ttc is the collision time between the OV and the ego vehicle selected according to the ego vehicle’s driving intention, and N is d lane-obs The number of other vehicles in range, r waiting When a collision occurs, r collision is assigned and ends the current training / testing round.

[0068] The action reward for a single time step is:

[0069] r step =r TTC +r collision +r velocity +r lane-changing +r waiting (13)

[0070] Since driving decisions are decisions with correlation between previous and subsequent actions, the rewards after the action is executed need to consider the time series information. Based on the T continuous state slices introduced by the spatial-temporal state feature encoder, the corresponding T continuous r step The value is used as the decision reward for the current step, recorded as:

[0071]

[0072] Among them, when t = 1, it represents the current step, when t = T, it represents the previous T-1 steps, and β (0 < β < 1) is the attenuation coefficient, that is, the closer the time distance, the higher the reward ratio.

[0073] 3. Reward Prediction Network

[0074] The reward prediction network uses a similar network architecture to the critic. It estimates future rewards based on the current state and corresponding actions. The reward prediction network's input is a state matrix consisting of the driving decision parameters output by the reinforcement learning parameter optimization network and the spatiotemporal state information acquired by the data acquisition module. The reward prediction network ultimately outputs a predicted reward value. This predicted reward value is added to the target speed output by the PID to form the basis for the final action selection. The formula is as follows:

[0075] r pred =f pred (s,p)+σv dts (15)

[0076] Among them, σ is the adjustment parameter, r pred For, f pred (·) is the reward predictor.

[0077] For the agent and pseudo-expert, the reward prediction network is trained based on their respective historical data to obtain the reinforcement learning agent reward predictor (RL-agent rewards predictor) and the pseudo-expert reward predictor (pseudo-expert rewards predictor). The two reward predictors respectively predict the corresponding rewards for the actions taken by the agent and pseudo-expert strategies. By comparing the reward values, the better action is selected as the actual output. The formula is as follows:

[0078]

[0079] Among them, p RL is the output of the reinforcement learning agent, i.e. the optimized driving decision parameters, p exp Dynamic topology parameters set for pseudo-experts.

[0080] 4. Imitation Learning

[0081] The experience gained by the pre-tuned pseudo-expert running in the environment is stored in the agent's experience pool, which is used to accelerate the agent's learning. By setting the ratio of reinforcement learning to behavior cloning learning at 4:1, the agent maintains both action strategy exploration and pseudo-expert experience learning. The randomness of action exploration gradually decreases with the number of training rounds, eventually stabilizing the action strategy.

[0082] 5. Typed-balanced experience pool

[0083] This method designs four experience pools, including the RL-agent experience pool (RL-agent buffer), the pseudo-expert experience pool (pseudo-expert buffer), the RL-agent reward prediction experience pool (rewardspredicting buffer for RL-agent), and the pseudo-expert reward prediction experience pool (rewardspredictingbuffer for pseudo-expert). Among them, the RL-agent experience pool stores the experience data (s, p) of the agent's final choice output. RL ,r); the pseudo-expert experience pool stores the pseudo-expert experience data (s,p exp ,r), which is used to accelerate the early learning of the reinforcement learning agent; the reinforcement learning agent reward prediction experience pool stores the experience data (s,a) used to train the reinforcement learning agent reward predictor RL ,p RL ,r); and the pseudo-expert reward prediction experience pool stores the experience data (s,a exp,p exp ,r).

[0084] Among them, the reinforcement learning agent experience pool and the pseudo-expert experience pool constitute the Typed-balanced experience pool, whose purpose is to ensure comprehensive and effective learning of samples and avoid the influence of data distribution on the learning results of the model for low-proportion samples. It divides the sample data into three categories according to the action type and reward score: road keeping samples, lane changing samples, and low reward value samples. Samples of different categories are maintained separately. The maintenance mechanism of road keeping samples and lane changing samples is FIFO. Low reward value samples will give priority to retaining low reward value samples in the experience pool. At each sampling, the proportion of different types of samples in a batchsize is affected by the number of samples. The sampling ratio of road keeping samples θ1, the sampling ratio of lane changing samples θ2, and the sampling ratio of low reward value samples θ3 are expressed as:

[0085]

[0086] θ3=ρ (19)

[0087] Where N1 and N2 are the number of lane-keeping samples and lane-changing samples, respectively, and ρ = ρ (0 < ρ < 1) is the preset ratio parameter. This setting ensures that all sample types are learned and automatically adjusts the learning weight of different sample types based on actual conditions.

[0088] For the road keeping samples and lane changing samples, assume that the index set of all samples of a certain class is I. According to the reward value corresponding to the sample index, I is sorted in descending order to obtain I′={i (1) ,i (2) ,...,i (N)}, index i (k) (k=1,2,...,N) The corresponding reward is r (k) , where r (1) ≥r (2) ≥...≥r (N) , N is the total number of samples. (k) Calculate priority weights:

[0089]

[0090] Among them, α>0 is a parameter used to control the priority. When α=0, the sampling method is uniform sampling. The larger α is, the more inclined to collect samples with high reward values. Convert the priority weight to sampling probability:

[0091]

[0092] From the set {i (1) ,i (2),...,i (N)}, and the sampling results are recorded as Each position l m The selection probability of (m=1,2,...,M) is It can be expressed as:

[0093]

[0094] in, According to the probability distribution Repeat the sampling M times, sampling a category index l∈{1,2,…,N} independently each time.

[0095] For samples with low reward values, random sampling is used, which is mainly used to provide learning counterexamples.

[0096] The reward prediction experience pool uses standard experience pool maintenance and sampling methods. Samples are not classified, the maintenance mechanism is FIFO, the experience pool capacity is set to a small value, and the sampling method is random. This is primarily intended to provide the reward prediction network with samples that are closer in time, enabling it to predict rewards that are more appropriate for the agent's actions at the current stage.

[0097] In summary, the method described in this embodiment can be expressed as follows:

[0098] Build a reinforcement learning parameter optimization network and a typed-balanced experience pool. Run a pre-trained pseudo-expert agent in the environment to generate experience data including environmental parameters and corresponding driving decision parameters. Store the experience data in the typed-balanced experience pool. Collect samples from the typed-balanced experience pool for the reinforcement learning parameter optimization network to learn.

[0099] Generate driving decision parameters based on real-time collected environmental parameters, and calculate vehicle control parameters based on the driving decision parameters through a PID controller;

[0100] The generated driving decision parameters are input into the reinforcement learning parameter optimization network for optimization. The real-time collected environmental parameters are input into the pre-trained pseudo-expert body to generate comparative driving decision parameters, and the vehicle control parameters are calculated based on the driving decision parameters using a PID controller.

[0101] The optimized driving decision parameters and real-time collected environmental parameters are input into the reward prediction network. The predicted reward values ​​are output respectively by the reinforcement learning agent reward predictor and the pseudo-expert reward predictor. The reward values ​​are compared to select the better driving decision parameters, and the real vehicle control parameters are calculated by the PID controller.

[0102] In this embodiment, the reinforcement learning driving decision-making method based on the adaptive dynamic topology structure is simulated through the Town04 simulation environment of Carla 0.9.14 to compare the performance differences between the present invention and other intelligent driving decision-making methods.

[0103] The simulation environment consists of a complex road section with a length of approximately 990 meters, consisting of straights, curves, and ramps. The desired speed of the ego vehicle is set to 92 km / h (maximum speed is less than 100 km / h). The desired speed of other vehicles is set to 60 km / h, and the initial parking interval is approximately 16 meters. The other vehicles' operation model uses Carla Autopilot. The ego vehicle is required to reach the destination on this road section at the desired speed as much as possible. The mission is considered complete when the ego vehicle reaches the destination or collides with the vehicle. Vehicle driving performance is evaluated based on one-step rewards. The baseline models used in the experiment include TD3, DDPG, SAC, a pre-tuned expert model based on dynamic topology, and a car-following model based on Carla Autopilot. TD3, DDPG, and SAC are basic reinforcement learning models. The adaptive dynamic topology structure implemented based on the basic reinforcement learning algorithms (TD3, DDPG, and SAC) is denoted as RL-SDTS, with the corresponding TD3, DDPG, and SAC algorithms being SDTS-TD3, SDTS-DDPG, and SDTS-SAC, respectively. The differences in settings between the baseline model and the RL-SDTS model proposed in this example are shown in Table 1.

[0104] Table 1

[0105]

[0106] The rewards for a single time step are used to calculate the average and cumulative rewards for each time step, which are used as comparison indicators for model performance. The average reward is the average of the rewards for all steps from the vehicle's starting point to the end of the mission. The cumulative reward is the cumulative reward from the vehicle's starting point to 800 steps or the end of the mission (the number of steps required to complete the mission and achieve a high reward is between 800 and 900).

[0107] (1) Basic driving test on composite road section

[0108] Composite road section basic driving test is used to verify the efficiency and safety of driving. Figure 4 As shown. Each model is trained for 3 rounds and the average curve of the training is taken as the final experimental result. Since there is no specified path end point, r waiting Set it to 0. The experimental results are as follows Figure 5 shown.

[0109] Experimental results show that the RL-SDTS proposed in this embodiment can achieve a significant improvement in the overall driving performance based on the basic RL algorithm and the dynamic topology framework. Compared with the basic Autopilot following model, the dynamic topology model supports the vehicle to implement more complex local operations such as overtaking and lane changing, and can achieve a very significant improvement in reward scores compared to the Autopilot model. This provides a very good decision-making basis for RL algorithms based on dynamic topology structures. The reward score of an untrained RL-SDTS model (including the baseline model) will be lower than that of a dynamic topology model with expert prior parameters, but as the model is trained, the reward curve of the RL-SDTS model will gradually surpass the expert model or be on par with its performance. Compared with the basic baseline model, the performance improvements of the trained TD3, DDPG, and SAC are more obvious. Figure 5 The average reward curve shown in (a) shows that the score of TD3 after stable training is improved by 38.5% compared with the baseline TD3, the score of DDPG after stable training is improved by 33.9% compared with the baseline DDPG, and the score of SAC after stable training is improved by 10.6% compared with the baseline SAC. Figure 5 The cumulative reward curve shown in (b) shows that after stable training, TD3 achieves a 25.9% improvement over the baseline TD3 score, DDPG achieves a 31.6% improvement over the baseline DDPG score, and SAC achieves a 10.3% improvement over the baseline SAC score. Furthermore, after stable training, TD3, DDPG, and SAC also demonstrate improved safety performance. Based on the experimental results for 200 rounds of stable training, the average collision rates for TD3, DDPG, and SAC are 2.2%, 3.3%, and 4.0%, respectively. The average collision rates for the corresponding baseline models are 3.8%, 8.3%, and 5.2%, respectively.

[0110] Among all the RL models tested, SAC achieved the best overall score. In terms of average reward, after stable training, SAC outperformed TD3 and DDPG by 23.6% and 11.3%, respectively. In terms of cumulative reward, SAC outperformed TD3 and DDPG by 17.5% and 7.5%, respectively. The baseline SAC model also achieved a significant score advantage over the baseline TD3 and DDPG models. SAC's strong performance is attributed to its enhanced exploration capabilities. SAC's maximum entropy RL increases its algorithmic exploration, reducing the likelihood of the policy falling into poor local optima. The dynamic topology provides reasonable action constraints for the decision-making action outputs, allowing exploration within a relatively safe range. Therefore, combining SAC with the dynamic topology can greatly exploit the potential of the dynamic topology through SAC's exploration capabilities.

[0111] This experiment was compared with other similar experiments in terms of experimental scenarios, task settings, and task completion, as shown in Table 2. The comparison data is based on the best results from 200 consecutive SDTS-SAC training rounds after stabilization. These results demonstrate that the RL-SDTS proposed in this example achieves a good path completion rate even under more complex road conditions, denser vehicle environments, higher speeds, and greater speed differentials, demonstrating its superior environmental adaptability.

[0112] In Table 2, the relevant detailed information of other experiments used for comparison is as follows:

[0113] [1] D.Chen, L.Jiang, Y.Wang, and Z.Li, "Autonomous driving using safereinforcement learning by incorporating a regret-based human lane-changingdecision model," in 2020American Control Conference (ACC): IEEE, pp.4355-4361, 2020.

[0114] [2]C.-J.Hoel,K.Driggs-Campbell,K.Wolff,L.Laine,and M.J.Kochenderfer,“Combining planning and deep reinforcement learning in tactical decisionmaking for autonomous driving,”IEEE Transactions on Intelligent Vehicles,vol.5,no.2,pp.294-305,2020.

[0115] [3]Z.Feng,W.Song,M.Fu,Y.Yang,and M.Wang,“Decision-making and pathplanning for highway autonomous driving based on spatio-temporal lane-changegaps,”IEEE Systems Journal,vol.16,no.2,pp.3249-3259,2021.

[0116] [4]J.Dong,S.Chen,Y.Li,R.Du,A.Steinfeld,and S.Labi,“Space-weightedinformation fusion using deep reinforcement learning:The context of tacticalcontrol of lane-changing autonomous vehicles and connectivity rangeassessment,”Transportation Research Part C:Emerging Technologies,vol.128,pp.103192,2021.

[0117] [5]Y.Gu,K.Yuan,S.Yang,M.Ning,and Y.Huang,“Mandatory Lane-ChangingDecision-Making in Dense Traffic for Autonomous Vehicles based on DeepReinforcement Learning,”in 2022 6th CAA International Conference on VehicularControl and Intelligence(CVCI):IEEE,pp.1-7,2022.

[0118] [6]G.Li,Y.Yang,S.Li,X.Qu,N.Lyu,and S.E.Li,“Decision making ofautonomous vehicles in lane change scenarios:Deep reinforcement learningapproaches with risk awareness,”Transportation Research Part C:EmergingTechnologies,vol.134,pp.103452,2022.

[0119] [7]S.Alagumuthukrishnan,S.Deepajothi,R.Vani,and S.Velliangiri,“Reliable and efficient lane changing behaviour for connected autonomousvehicle through deep reinforcement learning,”Procedia Computer Science,vol.218,pp.1112-1121,2023.

[0120] [8] R. Yang, Z. Li, B. Leng, and L. Xiong, "Safe reinforcement learning for autonomous vehicles to make lane-change decisions: Constraint based on Incomplete Information Game Theory," in 2023 7th CAA International Conference on Vehicular Control and Intelligence (CVCI): IEEE, pp. 1-6, 2023.

[0121] [9] Y. Wang, L. Liu, M. Wang, and X. Xiong, "Reinforcement learning from human feedback for lane changing of autonomous vehicles in mixed traffic," arXivpreprint arXiv:2408.04447, 2024.

[0122]

[10] Z. Wang, H. Huang, J. Tang, and L. Hu, "A deep reinforcement learning-based approach for autonomous lane-changing velocity control in mixed flow of vehicle group level," Expert Systems with Applications, vol. 238, pp. 122158, 2024.

[0123] Table 2

[0124]

[0125] (2) Global path navigation experiment

[0126] The global path navigation experiment is designed to verify the vehicle's driving performance under the premise of a specified path end point (i.e., off-ramp). Figure 6 This experiment optimizes the model parameters for global path navigation based on the SDTS-SAC model trained in the above composite road basic driving experiment. The dynamic topology structure expert model is used as the experimental comparison model. The experimental results are shown in Figure 7 shown.

[0127] Experimental results show that the SDTS-SAC model can complete the global navigation task while outperforming the expert model. After the parameter optimization, the SDTS-SAC model adds the global path navigation task, although the reward score decreases, but the overall performance is still better than the expert model. Figure 7 As shown in the figure, the SDTS-SAC model's reward curve significantly improved early in training and maintained its score advantage. The collision rate of the SDTS-SAC model also gradually decreased as training progressed, ultimately achieving zero collisions. Regarding the average reward curve, after stable training, the SDTS-SAC model achieved a 56.7% improvement in score compared to the expert model. Regarding the cumulative reward curve, after stable training, the SDTS-SAC model achieved a 52.7% improvement in score compared to the expert model. Table 2 compares these experimental results with other similar studies. These comparative results demonstrate that the SDTS-SAC model can complete global path navigation tasks in more complex scenario settings.

[0128] The reward score of the global path navigation experiment is significantly lower than that of the basic driving experiment. This is mainly because the vehicle needs to ensure that it has switched to the corresponding lane of the exit ramp before reaching the ramp. waiting =-δ5. In addition, in the design of the dynamic topology, reaching the designated lane has a higher execution priority. Therefore, during RL-SDTS training, the reinforcement learning agent tends to prioritize completing the task of reaching the designated lane, which will sacrifice the non-negative rewards from other reward items (especially r velocity ).

[0129] (3) Ablation experiment

[0130] Ablation experiments were conducted using the best-performing SAC model. The mean and standard deviation of the average reward and cumulative reward over 150 rounds after training performance stabilized were recorded as comparison values. The comparison results are shown in Table 3. The experimental results show that all three modules—Temporal State & Reward, Reward Prediction, and Typed-Balanced Experience Pool—improve model performance. Reward prediction showed the most significant improvement. The Typed-Balanced Experience Pool provided less improvement in reward, but significantly improved output stability. The introduction of the Typed-Balanced Experience Pool further stabilized the model's reward output performance. Furthermore, Temporal State & Reward also contributed to the model's performance. By providing temporal state representation and reward signals, the model can access more state space, which is beneficial for both reward scores and stability.

[0131] Table 3

[0132]

[0133] Another embodiment of the present invention provides a reinforcement learning driving decision system based on an adaptive dynamic topology structure, which includes a dynamic topology structure module, a reinforcement learning parameter optimization module, a reward prediction module, an imitation learning module and a typed-balanced experience pool.

[0134] Among them, the dynamic topology structure module collects environmental parameters in real time, generates driving decision parameters based on the environmental parameters, and generates vehicle control parameters based on the driving decision parameters; the reinforcement learning parameter optimization module optimizes the driving decision parameters based on the environmental parameters; the reward prediction module selects the better driving decision parameters by comparing the reward values ​​and outputs them to the dynamic topology structure module to calculate the actual vehicle control parameters; the imitation learning module generates experience samples by deploying pre-trained pseudo-experts and stores them in the typed-balanced experience pool; the typed-balanced experience pool maintains the experience samples and provides samples for the reinforcement learning parameter optimization module to learn.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A reinforcement learning driving decision-making method based on an adaptive dynamic topology structure, characterized in that: The method includes: Build a reinforcement learning parameter optimization network and a typed-balanced experience pool. Run a pre-trained pseudo-expert agent in the environment to generate experience data including environmental parameters and corresponding driving decision parameters. Store the experience data in the typed-balanced experience pool. Collect samples from the typed-balanced experience pool for the reinforcement learning parameter optimization network to learn. Generate driving decision parameters based on real-time collected environmental parameters, and calculate vehicle control parameters based on the driving decision parameters through a PID controller; The generated driving decision parameters are input into the reinforcement learning parameter optimization network for optimization. The real-time collected environmental parameters are input into the pre-trained pseudo-expert body to generate comparative driving decision parameters, and the vehicle control parameters are calculated based on the driving decision parameters using a PID controller. The optimized driving decision parameters and real-time collected environmental parameters are input into the reward prediction network. The predicted reward values ​​are output respectively by the reinforcement learning agent reward predictor and the pseudo-expert reward predictor. The reward values ​​are compared to select the better driving decision parameters, and the real vehicle control parameters are calculated by the PID controller.

2. The method according to claim 1, characterized in that The reinforcement learning parameter optimization network includes a spatiotemporal state feature encoder and a reinforcement learning agent; the spatiotemporal state feature encoder obtains multiple continuous state slices from the real-time collected environmental parameters, encodes and calculates the state feature values ​​through the attention mechanism, and transmits them to the reinforcement learning agent. The reinforcement learning agent optimizes the driving decision parameters according to the calculation results.

3. The method according to claim 2, characterized in that The reward term of the reinforcement learning agent at each time step includes the TTC term r TTC , collision term r collision , speed term r velocity and lane change term r lane-changing , then the action reward at each time step is r step =r TTC +r collision +r velocity +r lane-changing +r waiting ; Considering T continuous state slices, the decision reward of the actor-critic reinforcement learning framework at the current time step is Among them, t = 1 represents the current time step, t = T represents the previous T-1 time step, and β is the attenuation coefficient.

4. The method according to claim 1, wherein Both the driving decision parameters and the comparison driving decision parameters include the lane change decision value, safe lane change distance, expected lane change speed when changing lanes, lane keeping speed when not changing lanes, driving speed during waiting for lane change, and braking distance; both the vehicle control parameters and the comparison vehicle control parameters include speed, throttle, and brake.

5. The method according to claim 4, characterized in that In the reward prediction network, the reinforcement learning agent reward predictor predicts the reward r after the reinforcement learning agent performs an action pred (s,p RL )=f pred (s,p RL )+σv dts , the pseudo-expert reward predictor predicts the reward r after the pseudo-expert performs an action pred (s,p exp )=f pred (s,p exp )+σv dts ′, where s represents the environmental parameters collected in real time, p RL represents the optimized driving decision parameters, v dts Indicates that based on p RL The calculated speed, p exp represents the driving decision parameters generated by the pseudo-expert, v dts ′ represents the value based on p exp The calculated speed, σ represents the adjustment parameter; select r pred (s,p RL ) and r pred (s,p exp ) is used as the final output.

6. The method according to claim 1, characterized in that The Typed-balanced experience pool includes road-keeping samples, lane-changing samples, and low-reward samples. The maintenance mechanism for road-keeping samples and lane-changing samples is a first-in-first-out mechanism, while the maintenance mechanism for low-reward samples is to prioritize low-reward samples. Each time a sample is collected from the Typed-balanced experience pool, the sampling ratios of the three types of samples are as follows: θ3=ρ Where θ1 is the sampling ratio of road-keeping samples, θ2 is the sampling ratio of lane-changing samples, and θ3 is the sampling ratio of low-reward samples. N1 and N2 are the numbers of road-keeping samples and lane-changing samples, respectively. ρ is the preset ratio parameter. The sampling method for road keeping samples and lane changing samples is based on sample priority, and the sampling method for low reward value samples is random sampling.

7. The method according to claim 6, characterized in that For any of the road keeping samples and lane changing samples, assume that the index set of all samples of this type is I, sort I in descending order according to the reward value corresponding to the sample index, and obtain I′={i (1) ,i (2) ,...,i (N) }, index i (k) The corresponding reward is r (k) , k=1,2,...,N, where r (1) ≥r (2) ≥...≥r (N) , N is the total number of samples; for the sorted index i (k) Calculating priority weights α is a parameter used to control the priority, converting the priority weight into sampling probability From the set {i (1) ,i (2) ,...,i (N) }, and the sampling results are recorded as Each position l m The probability of selection is m=1,2,...,M; the sampling process is expressed as: in, According to the probability distribution Repeat the sampling M times, sampling a category index l∈{1,2,…,N} independently each time.

Images