A Method for Continuous Lane Changing Decision-Making in Autonomous Driving Based on Meta-Reinforcement Learning
By constructing a continuous lane-changing decision model based on meta-reinforcement learning, integrating the motion state of the main vehicle and lane-changing gap information, and designing a policy network with discrete and continuous action branches, this method solves the problems of observation limitations and insufficient generalization ability in existing continuous lane-changing decision-making technologies, and achieves safe, efficient and smooth continuous lane changing.
Patent Information
- Application Number
- CN202611597491.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing reinforcement learning models struggle to simultaneously observe traffic conditions in subsequent target lanes and consider the impact of current lane-changing behavior on the feasibility of lane-changing in the next stage during continuous lane-changing decisions in autonomous driving. Furthermore, they lack generalization ability in diverse scenarios involving variations in traffic density, vehicle speed, and lane-changing distance.
A continuous lane-changing decision model is constructed using a meta-reinforcement learning approach. The model integrates the SAC algorithm and Reptile meta-learning method with two-stage Ward clustering, combining the vehicle's motion state, lane-changing process information, lane-changing gap information, and risk value. A policy network with discrete and continuous action branches is designed to achieve adaptive decision-making for the vehicle in continuous lane changes.
It improves the safety, efficiency, and smoothness of autonomous vehicles during continuous lane changes, enhances the model's generalization ability in different scenarios, and enables it to quickly adapt to new lane-changing tasks.
Smart Images

Figure CN123104700A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent transportation and traffic safety control, and in particular relates to a method for continuous lane-changing decision-making in autonomous driving based on meta-reinforcement learning. Background Technology
[0002] With the widespread adoption of autonomous vehicles, their driving environments are gradually expanding from simple, ordinary road sections to multi-lane road sections with diverging, merging, and weaving zones. Vehicles often need to traverse multiple lanes sequentially within a limited longitudinal distance to enter a target lane that matches their predetermined route. In this situation, the timing and entry gap of the previous lane change alters the vehicle's relative position in the middle lane and further impacts the remaining operational space for the next lane change. Autonomous vehicles must consider subsequent lane change needs while executing the current lane change, coordinating safety, traffic efficiency, driving smoothness, and task completion. Therefore, it is necessary to study continuous lane-changing decisions in these scenarios to meet the practical needs of autonomous vehicles performing complete driving tasks.
[0003] Currently, research on autonomous driving decision-making for continuous lane changes is still limited, with most studies focusing on single lane change decisions. Reinforcement learning, due to its ability to autonomously learn strategies through interaction with unknown environments, is widely applied to the lane-changing decision-making problem in autonomous driving. For example, patent application CN122116687A uses the DDQN method, which designs corresponding traffic flow guidance rewards by perceiving the traffic wave propagation speed of the current lane and the left and right lanes within the lane reduction zone, enabling vehicles to spontaneously leave lanes with congestion tendencies. Patent application CN122290357A also uses the DDQN method, which sets rewards such as energy consumption and efficiency based on obtaining the micro-interaction characteristics of vehicles in two lanes and the macro-signal light status, improving the effectiveness of lane changing for vehicles upstream of signalized intersections. Patent application CN122116686A uses the MADDPG method, which constructs the motion state space of the vehicle and the vehicles in front and behind in the target lane, and sets individual and global rewards to achieve multi-vehicle cooperative lane changing in obstacle avoidance situations.
[0004] Although single lane-changing models based on reinforcement learning have demonstrated good autonomous learning performance, their direct application to continuous lane-changing still has two drawbacks:
[0005] On the one hand, such models focus on vehicle information in the current lane and adjacent target lanes, and make decisions about whether to change lanes, the direction of lane change, and the adjustment of vehicle movement status. If directly used in continuous lane-changing scenarios, the model can usually only make local decisions for adjacent lanes, making it difficult to observe the traffic situation in subsequent target lanes at the same time, and also unable to fully consider the impact of the current lane-changing behavior on the feasibility of lane changing in the next stage.
[0006] On the other hand, when making continuous lane-changing decisions, the model's decision requirements vary significantly with changes in traffic density, vehicle speed, and lane-changing distance. Traditional reinforcement learning models are usually trained for fixed scenarios and are difficult to maintain generalization ability in differentiated continuous lane-changing scenarios.
[0007] Meta-learning provides an effective approach to solving this problem. Its core lies in "learning how to learn," which means extracting transferable common knowledge from multiple training tasks so that the model can quickly adapt with only a few updates when facing new lane-changing tasks. Summary of the Invention
[0008] The purpose of this invention is to provide a continuous lane-changing decision-making method for autonomous driving based on meta-reinforcement learning, which enables autonomous vehicles to adaptively select lane-changing timing and adjust motion state according to traffic pressure on multiple lanes, lane-changing gap distribution and remaining lane-changing distance, thereby completing a safe, efficient, smooth and generalizable continuous lane-changing task.
[0009] To achieve one of the aforementioned objectives, one embodiment of the present invention provides a method for continuous lane-changing decision-making in autonomous driving based on meta-reinforcement learning. This method uses the SAC algorithm as the basic decision-making method to construct a continuous lane-changing decision-making model, and then implements continuous lane-changing decisions for autonomous driving based on the trained continuous lane-changing decision-making model; wherein,
[0010] The continuous lane-changing decision-making model includes a policy network, two value networks, and two corresponding target networks;
[0011] The continuous lane-changing decision model takes the continuous lane-changing interaction state as input and the lane-changing gap selection and longitudinal acceleration of the main vehicle as action output.
[0012] As a further improvement of one embodiment of the present invention, the continuous lane-changing interaction state includes the motion state of the main vehicle from the start to the end of the continuous lane-changing, the lane-changing process of the main vehicle, the set of lane-changing gaps, the risks suffered by the main vehicle, and the set of lane-changing gap risks, wherein,
[0013] The motion state of the main vehicle at time t from the start to the end of the continuous lane change. Represented as:
[0014] ,
[0015] In the formula, Let represent the ordinate of the center position of the main vehicle at time t, the abscissa of the center position, the longitudinal velocity, the lateral velocity, the longitudinal acceleration, the lateral acceleration, and the lane number it is in.
[0016] The main train lane-changing process at time t Represented as:
[0017] ,
[0018] In the formula, These represent the next target lane number of the main vehicle at time t, the remaining longitudinal distance from the lane change endpoint, and the final target lane number, respectively.
[0019] Set of lane change gaps at time t Represented as:
[0020] ,
[0021] In the formula, , This represents the total number of lane-changing gaps at time t. This represents the state of the k-th lane change interval at time t. , These represent the longitudinal position of the center of the k-th lane change gap at time t, the lateral position of the center, and the gap length, respectively. These represent the longitudinal distances between the main vehicle and the vehicle in front and behind at time t during the k-th lane change interval, respectively. Let represent the longitudinal relative velocities between the main vehicle and the preceding and following vehicles at time t during the kth lane change interval, respectively; This represents the lane number where the k-th lane change interval is located at time t;
[0022] Risks suffered by the main vehicle at time t By determining the center position of the main vehicle at time t Substituting into the road safety potential field formula yields the following:
[0023] Risk set of lane change gap at time t Represented as:
[0024] ,
[0025] In the formula, This represents the risk of the k-th lane change gap at time t. By determining the center position of the k-th lane change gap at time t Substituting into the road safety potential field formula, we get... These represent the longitudinal and lateral coordinates of the center position of the k-th lane change gap at time t, respectively.
[0026] As a further improvement to one embodiment of the present invention, based on the continuous lane-changing interaction state, the state space of the continuous lane-changing decision model at time t is... Represented as: ;
[0027] Action space of the continuous lane-changing decision model at time t Represented as:
[0028] ,
[0029] In the formula, This indicates the lane change interval number selected by the main vehicle at time t;
[0030] Reward function of the continuous lane-changing decision model at time t Represented as:
[0031] ,
[0032] In the formula, They are respectively Weighting coefficients; These represent the safety reward, efficiency reward, comfort reward, and task completion reward at time t, respectively. , This indicates the risk penalty coefficient suffered by the main vehicle. This represents the lane change clearance risk penalty coefficient. Represents the time t. Risk of lane change gap; , Indicates the desired speed. This indicates the total longitudinal travel distance for consecutive lane changes. Indicates the start time of the lane change. This indicates whether the main vehicle is in a following mode. , and These represent the speed bonus coefficient, the remaining distance penalty coefficient, and the following time penalty coefficient, respectively. , Indicates the acceleration penalty coefficient; , express Whether the main vehicle has completed a lane change at any given time. , express Whether the main vehicle has completed all lane-changing tasks at any given time. , This indicates whether the lane-changing task is terminated at time t due to a collision or exceeding the maximum number of decision steps. , , and These represent the stage lane change reward coefficient, the final task completion reward coefficient, and the task failure penalty coefficient, respectively.
[0033] As a further improvement of one embodiment of the present invention, the policy network consists of a shared feature extraction layer, a discrete action output branch, and a continuous action output branch. The shared feature extraction layer is used to extract common input features in the state space; the discrete action output branch is used to output the corresponding lane change gap selection probability based on the common features; and the continuous action output branch is used to output the longitudinal acceleration that matches the lane change gap based on the common features.
[0034] As a further improvement to one embodiment of the present invention, it also includes,
[0035] Build task distribution and will Training tasks were divided according to a set ratio. With test tasks This is to enable the training and testing of the continuous lane-changing decision model;
[0036] Among them, task distribution The construction steps include:
[0037] 1) Construct a feature vector for a continuous lane-changing scenario, consisting of the remaining longitudinal distance of the main vehicle from the lane-changing endpoint, the risk experienced by the main vehicle, and the average risk of the lane-changing gap in the next target lane; where the average risk of the lane-changing gap in the next target lane at time t is expressed as:
[0038] ,
[0039] In the formula, This represents the set of lane-changing gaps in the next target lane at time t. , Indicates the number of lane change gaps on the next target lane;
[0040] 2) Perform two-stage clustering on continuous lane-changing scenarios, specifically including:
[0041] by Based on the clustering criteria, the Ward minimum variance clustering method was used to implement the first stage of clustering, dividing all consecutive lane-changing samples into... The task urgency categories are represented as follows:
[0042] ,
[0043] In the formula, This represents the results of the first stage of clustering. Indicates the first The tasks are categorized by urgency.
[0044] Within each major category of task urgency, a second-stage clustering process is implemented based on the risk balance index, further dividing the continuous lane-changing samples into... There are several risk balance subcategories; among them, the lane-changing process is... Risk balance index for each sub-interval Represented as:
[0045] ,
[0046] In the formula, This indicates that the main vehicle is in the lane-changing process at the first stage. The risks within each sub-interval,
[0047] , For the lane changing process The duration of each sub-interval, This is represented as the candidate gap for the next target lane in the lane-changing process. Average risk within each sub-interval ;
[0048] The final two-stage clustering results for all consecutive lane-changing samples are represented as follows:
[0049] ,
[0050] In the formula, This represents the results of two-stage clustering. Indicates the first The first of the major categories of task urgency Each risk balancing subcategory;
[0051] 3) Based on the two-stage clustering results, a task distribution is constructed. :
[0052]
[0053] In the formula, This indicates the results of two-stage clustering. The m-th consecutive lane-changing task is formed by the corresponding set of consecutive lane-changing samples. This indicates the total number of tasks.
[0054] As a further improvement to one embodiment of the present invention, the training of the continuous lane-changing decision model consists of two parts: intra-task updates and inter-task updates, specifically including:
[0055] 1) In-task update: Using the SAC method, guided by the overall objective function, the value network is first updated by minimizing the Bellman error. Then, the policy network is updated by maximizing the Q-value of the value network output while maintaining the policy entropy. Finally, the target network is updated using a soft update method based on the corresponding value network. The overall objective function... Represented as:
[0056]
[0057] in, Indicates the expectation. The maximum number of decision steps in a single round. As a discount factor, express The entropy of the policy distribution; For temperature parameters;
[0058] 2) Inter-task updates: From Several training tasks were sampled, and the Reptile method was used to process the meta-parameters. Optimize:
[0059]
[0060] In the formula, M represents the meta-learning step size, and M represents the step size from which data is learned in each round of meta-training. The number of tasks in the sampling process. This indicates that the m-th task is executed. The policy network parameters obtained during the step task are updated.
[0061] As a further improvement to one embodiment of the present invention, the loss function of the value network is expressed as:
[0062] ,
[0063] In the formula, , Represents the i-th value network loss function, Represents the i-th value network Q-value estimation for the current state-action pair. This represents the time-difference objective Q-value. , .
[0064] As a further improvement to one embodiment of the present invention, the loss function of the policy network is expressed as:
[0065] ,
[0066] In the formula, .
[0067] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps in the above-described method for continuous lane-changing decision-making in autonomous driving based on meta-reinforcement learning.
[0068] To achieve one of the above-mentioned objectives, an embodiment of the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps in the autonomous driving continuous lane-changing decision-making method based on meta-reinforcement learning as described above.
[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0070] (1) Unlike traditional decision-making methods that only address single lane changes, this invention addresses the scenario where a vehicle completes two or more lane changes consecutively. It integrates the vehicle's motion state, lane change process information, lane change gap information, and corresponding risk values to construct a continuous lane change interactive state. This interactive state enables the autonomous vehicle to observe the dynamic changes in the lane change gap and its risks of the subsequent target lane while executing the current lane change decision, thereby pre-considering the impact of each decision on the feasibility of subsequent lane changes.
[0071] (2) This invention addresses the correlation between lane-changing stages in continuous lane-changing decisions by setting discrete action branches and continuous action branches in the SAC policy network, which output lane-changing gap selection and longitudinal acceleration of the main vehicle, respectively, enabling the vehicle to rationally plan lane-changing timing. In addition, by introducing a car-following state penalty between two adjacent lane changes into the model's reward function, the vehicle's dwell time in the middle slow lane can be reduced, improving the driving efficiency of continuous lane-changing.
[0072] (3) This invention combines the Reptile meta-learning method with the SAC algorithm, and trains the model on diverse lane-changing tasks obtained through two-stage clustering to obtain the optimal policy initialization parameters. Compared with reinforcement learning models trained only in a single scenario, this invention can quickly adapt to new continuous lane-changing scenarios by using a small number of interaction samples and a limited number of parameter updates, thereby improving the generalization ability of the continuous lane-changing decision model.
[0073] (4) This invention proposes a scenario classification method based on two-stage Ward clustering to address the differences in decision-making needs at different stages in continuous lane-changing scenarios. The first stage clustering uses the remaining longitudinal distance of the lane change as the basis to identify different levels of urgency for vehicles to complete continuous lane-changing tasks. In the second stage clustering, the changing characteristics of the risk experienced by the lead vehicle and the risk of the lane-changing gap in the next target lane are further utilized to classify different risk interaction patterns. Compared to methods that directly perform single-stage clustering on multi-dimensional traffic features, this invention can reduce the mutual interference between different physical meaning indicators, enabling the obtained scenario categories to simultaneously possess both task urgency and risk interaction explanatory capabilities. Attached Figure Description
[0074] Figure 1 It is the overall architecture of a continuous lane-changing decision-making model based on meta-reinforcement learning.
[0075] Figure 2 This is a two-stage Ward clustering flowchart for a continuous lane-changing scenario.
[0076] Figure 3 This is a schematic diagram of SAC parameter updates in reinforcement learning.
[0077] Figure 4 This is a diagram illustrating the update of the Reptile parameter in meta-learning. Detailed Implementation
[0078] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.
[0079] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0080] This invention proposes a continuous lane-changing decision-making method for autonomous driving based on meta-reinforcement learning, specifically including: First, clarifying the continuous lane-changing direction of the vehicle and extracting lane-changing gap information of the main vehicle, surrounding vehicles, and subsequent target lanes, and constructing a continuous lane-changing interaction state in combination with the road safety potential field; then, designing a continuous lane-changing decision-making model including a lane-changing gap selection branch and a longitudinal acceleration output branch, enabling the vehicle to select the lane-changing gap in the next target lane step by step according to the predetermined lane-changing direction; finally, implementing two-stage Ward clustering based on the task urgency and risk balance relationship of the continuous lane-changing scenario, constructing the different scenario categories obtained from the clustering into a task distribution for meta-reinforcement learning, and obtaining policy network parameters that can quickly adapt to different continuous lane-changing scenarios.
[0081] like Figure 1 As shown, the specific steps of this method are as follows:
[0082] Step S1: Construction of Continuous Lane Changing Interaction State. Obtain the future lane sequence of the vehicle and identify the main vehicle for continuous lane changing based on the sequence; extract the state of the main vehicle and the surrounding vehicles within its perception range to construct the lane changing gap of the subsequent target lane; use the road safety potential field to quantify the risk to the main vehicle and the risk of the lane changing gap, and finally form the continuous lane changing interaction state, which serves as the basis for subsequent model design and training.
[0083] Step S2: Design of the continuous lane-changing decision model. Using the continuous lane-changing interaction state obtained in S1 as input, and the lane-changing gap selection and longitudinal acceleration of the main vehicle as action outputs, a reward function is constructed based on safety, efficiency, comfort, and task completion. A dual-branch policy network is designed using the SAC algorithm as the basic decision-making method to achieve lane-changing policy output under specific tasks. The Reptile method is used to train the SAC across tasks, enabling the model to quickly adapt to new continuous lane-changing scenarios with minimal SAC updates.
[0084] Step S3, Model Training. Based on the continuous lane-changing interaction states obtained in S1, construct scene feature vectors; implement Ward minimum variance clustering in stages from two aspects: task urgency and risk balance; output continuous lane-changing scene category labels, and set different scene categories as the task distribution of the model; finally, complete the model training.
[0085] S.1 Continuous Lane Change Interaction State Construction
[0086] S1.1 Future Lane Sequence Acquisition
[0087] This invention first acquires trajectory-level data within an observed road area, which can be a divergence zone, merging zone, weaving zone, or other multi-lane scenario requiring vehicles to complete two or more lane changes consecutively. The trajectory data includes vehicle IDs (…). ), time frame ( Lane number () ), lateral position ( ), longitudinal position ( ), lateral / longitudinal velocity ( ), lateral / longitudinal acceleration ( ) and vehicle dimensions (length) ,width (and other basic elements.)
[0088] Then, by combining vehicle navigation information, road structure information, and road passability status, it is determined whether there is a need for continuous lane changing. This need includes situations such as entering or exiting ramps, avoiding construction areas or road obstacles, responding to lane reduction, and entering a designated lane according to the navigation path. For example, when a vehicle needs to enter or exit a ramp, the lane connecting to the target ramp is determined based on the navigation path and used as the final target lane; when a vehicle needs to avoid construction areas or other road obstacles, the final target lane is determined from the passable lanes that meet the navigation requirements; when lane reduction or lane function changes occur, the final target lane the vehicle should enter is determined based on the road topology.
[0089] After determining the final target lane, based on the vehicle's initial lane, the final target lane, and the adjacency relationships between lanes, an ordered sequence of future lanes is generated according to the principle that the vehicle can enter adjacent lanes step by step:
[0090]
[0091] in, This indicates the initial lane where the main vehicle is located at the start of a continuous lane change. This indicates the final destination lane; adjacent lanes in the sequence are all interchangeable. Indicates the required number of lane changes. When When this happens, the lane-changing behavior is identified as a continuous lane-changing behavior.
[0092] S1.2 Extraction of lane-changing information for the main vehicle
[0093] For the determined future lane sequence of continuous lane changes, a search is performed within the dataset, and vehicles with corresponding trajectories are identified as the main vehicle samples for continuous lane changes. For the main vehicle performing the continuous lane change... ,set up Given the lane number at the current moment, extract its motion state from the start to the end of the continuous lane change:
[0094]
[0095] To characterize the lane-changing process of the main train, let... Number the next target lane; This is the remaining longitudinal distance between the main vehicle and the end point of the lane change. The final target lane is known to be... The lane-changing process of the main vehicle is represented as follows:
[0096]
[0097] S1.3 Extraction of information about surrounding vehicles
[0098] Set the longitudinal sensing range of the main vehicle and lateral sensing range The trajectory of surrounding vehicles is extracted. These surrounding vehicles are not limited to a specific lane, but include vehicles in the current lane and subsequent target lanes that have potential interaction with the main vehicle. The main vehicle then... Surrounding vehicles within the constant perception range The set is represented as:
[0099]
[0100] For surrounding vehicles Calculate its value at each time step The motion state variables relative to the host vehicle, including the longitudinal distance between the host vehicle and the host vehicle. and lateral distance and the longitudinal relative speed with respect to the main vehicle. and lateral relative velocity :
[0101]
[0102]
[0103]
[0104]
[0105] Set surrounding vehicles The body length is Vehicle width to surrounding vehicles The information is represented as:
[0106]
[0107] Get all surrounding vehicle information within the perception range .
[0108] S1.4 Lane Change Clearance Information Construction
[0109] Based on the constructed surrounding vehicle information, potential lane-changing gap information in subsequent target lanes is extracted. For any target lane... According to the longitudinal position of the vehicles in the lane Sort the vehicles. For any adjacent vehicles in the same lane... and the car behind If the following conditions are met:
[0110]
[0111] This creates a lane-changing gap between the two lanes. Therefore, the first lane in that lane... The length of a lane change gap is defined as:
[0112]
[0113] in, Indicates the first Lane change interval length, Indicates the lane change interval where the vehicle in front is The vertical position of time. Indicates the gap after the lane change. The vertical position of time. These represent the lengths of the front and rear vehicles, respectively.
[0114] The longitudinal position of the center of the lane change clearance is represented as follows:
[0115]
[0116] The lateral position of the center of the lane change gap is determined by the position of the centerline of the lane it belongs to, denoted as . .
[0117] The longitudinal distances between the main vehicle and the vehicles in front and behind it during lane changing are expressed as follows:
[0118]
[0119]
[0120] The longitudinal relative velocities between the main vehicle and the preceding and following vehicles in the lane-changing gap are expressed as follows:
[0121]
[0122]
[0123] Since the lane-changing clearance is determined by the longitudinal movement of the adjacent vehicles in front and behind, the first vehicle in the target lane... The lane change interval status is represented as follows:
[0124]
[0125] set up for The total number of time intervals at any given moment will determine the status of all track change intervals. The set of lane-changing gaps:
[0126]
[0127] S1.5 Interactive Risk Characterization
[0128] This invention employs existing road safety potential field methods for risk quantification, thereby integrating the impact of all risk sources in the environment (such as road boundaries, lane lines, and surrounding vehicles). Let the road conditions be... Combined with the surrounding vehicle information obtained in S1.3 Then at time spatial point The road safety potential field value at a certain location can be expressed as:
[0129]
[0130] center position of the main vehicle The safety potential field value obtained by substituting into the road safety potential field formula is the risk suffered by the main vehicle. The set of lane change clearances derived from S1.4 In the middle, obtain the first one on the target lane The center position of the lane change gap Substituting this into the road safety potential field formula yields the lane change clearance risk. The total number is The risk of lane change gaps constitutes a set .
[0131] Finally, the lane-changing information of the main vehicle obtained from S1.2 is... , The lane change clearance information obtained in S1.4 And the risks to the main vehicle obtained from S1.5 and lane change clearance risk By integrating them, they together constitute the timeline. The continuous lane-changing interaction state.
[0132] S.2 Continuous Lane Changing Decision Model Design
[0133] Based on the continuous lane-changing interaction states obtained from S.1, a meta-reinforcement learning model for continuous lane-changing in autonomous vehicles is constructed. This model uses the SAC reinforcement algorithm as the underlying decision-making basis and the Reptile meta-learning algorithm as the upper-level optimization basis, treating the driver vehicle as an intelligent agent. The intelligent agent, at each time step... The system learns continuous lane-changing decision-making strategies by selecting actions based on the current state and receiving rewards based on changes in traffic conditions after the actions are executed.
[0134] S2.1 Basic Element Design
[0135] Reinforcement learning is based on MDP (Markov Decision Process). This invention designs the state space, action space, and reward function in MDP based on the characteristic of continuous lane changing.
[0136] (1) State space
[0137] Based on the continuous lane-changing interaction status output by S.1, The state space representation of the reinforcement learning model at each time step is:
[0138]
[0139] This invention, through this design, enables the model to retain the task progress information and lane change interval information required for continuous lane changes, and to use the road safety potential field to uniformly express the risk sources, thereby comprehensively depicting the perception of road traffic conditions.
[0140] Considering that the number of time intervals may vary at different times, thus affecting the state space Therefore, due to the dimension of the problem, it is processed uniformly. The number of lane-changing gaps for each target lane is unified to the maximum value. and set validity indicators. ,satisfy:
[0141]
[0142] When the actual number of gaps is less than the preset number At that time, for the index Gap state where there is an actual gap and risk value Use the actual calculated values and mark their validity. =1; for index Gap state with no actual gap and risk value Perform zero-value padding and then mark its validity. The value is 0. This allows the state at different times to be uniformly represented as a fixed dimension.
[0143] (2) Action space
[0144] The main vehicle is The next target lane that can be directly entered at any time is (As defined in S1.2), the set of gap numbers corresponding to the next target lane is:
[0145]
[0146] This invention designs the continuous lane-changing decision model into two parts: longitudinal speed adjustment and lane-changing gap selection. The main vehicle... The action space at a given time is represented as:
[0147]
[0148] in, Indicates that the main vehicle is Longitudinal acceleration at time t; Indicates that the main vehicle is The lane change interval number selected at each moment satisfies , This indicates that the main vehicle will remain in its current lane. This indicates that the main vehicle has chosen to enter the... There must be a lane change gap, and this gap must actually exist.
[0149] (3) Reward function
[0150] To guide the main vehicle to complete continuous lane-changing tasks safely, efficiently, and smoothly, a system based on safety rewards is established. Efficiency rewards Comfort rewards and task completion rewards The weighted reward function consists of:
[0151]
[0152] in, These represent the weight coefficients of the corresponding reward items.
[0153] Safety Rewards Based on the gap risk output in S1.5 A constraint is constructed to restrict the master vehicle from entering a high-risk area; its expression is:
[0154]
[0155] in, This indicates the risk penalty coefficient suffered by the main vehicle; This represents the lane change clearance risk penalty coefficient. This means that the main vehicle did not select any clearance. It is not included in the safety reward.
[0156] Efficiency rewards are used to encourage the lead vehicle to complete consecutive lane changes as quickly as possible while ensuring safety, especially by shortening the following time between adjacent lane changes. If the lead vehicle remains in the middle lane for an extended period after completing the previous lane change, it will affect the timing of subsequent lane changes and increase the risk of mission failure. (Regarding time...) Define the car-following state indicator variable If the main vehicle has completed at least one lane change but has not yet completed all lane changes, and has not yet executed the next lane change, then... ;otherwise The efficiency reward is then expressed as:
[0157]
[0158] in, express The longitudinal speed of the main vehicle at any given time (as defined in S1.2). Indicates the desired speed. express The remaining longitudinal distance between the main vehicle and the end point of the lane change at any given time (as defined in S1.2). This indicates the total longitudinal travel distance during consecutive lane changes; , and These represent the speed bonus coefficient, the remaining distance penalty coefficient, and the following time penalty coefficient, respectively.
[0159] The comfort bonus is used to mitigate ride roughness caused by excessive speed adjustments in the vehicle, thereby ensuring passenger comfort. Its expression is:
[0160]
[0161] in, express The longitudinal acceleration of the main vehicle at any given time (as defined in S1.2); This represents the acceleration penalty coefficient.
[0162] The task completion reward guides the master vehicle to complete consecutive lane-changing tasks according to the future lane sequence. If the master vehicle successfully moves from the current lane to the next target lane according to the future lane sequence, the current stage of lane changing is considered complete, and a stage reward is given; if the master vehicle reaches the final target lane and completes all lane-changing tasks, an achievement reward is given; if the master vehicle collides or fails to complete the task within the maximum number of decision steps, a penalty is given. Its expression is:
[0163]
[0164] in, express The time indicates whether a lane change has been completed; a value of 1 indicates completion, otherwise a value of 0 indicates completion. express The time indicator shows whether the lane change task has been completed; a value of 1 indicates completion, otherwise a value of 0 indicates completion. Indicates whether the current round terminates due to a collision or exceeding the maximum number of decision steps; 1 indicates termination, and 0 indicates otherwise. , and These represent the stage lane change reward coefficient, the final task completion reward coefficient, and the task failure penalty coefficient, respectively.
[0165] S2.2 SAC Network Structure Design
[0166] The SAC model comprises a policy network, two value networks, and two corresponding target networks. The policy network outputs actions based on the current state, the value networks evaluate the value of state-action pairs, and the target networks construct temporal difference target values. This invention focuses on the design of the policy network.
[0167] The policy network in this invention consists of a shared feature extraction layer, a discrete action output branch, and a continuous action output branch. The shared feature extraction layer first extracts common input features from the state space; then, the discrete action branch outputs the corresponding lane-change gap selection probability based on the common features; the continuous action branch further outputs the longitudinal acceleration matching the lane-change gap. Let the policy network be... , can be represented as:
[0168]
[0169] in, This represents a discrete action strategy for selecting lane change intervals. A continuous motion strategy representing longitudinal acceleration.
[0170] S2.3 Reptile Optimization Target Design
[0171] This invention uses the Reptile method to process the network parameters of the SAC policy. Perform cross-task optimization. Assume... For continuous lane-changing task distribution, the adaptation strategy parameters corresponding to different continuous lane-changing tasks are: Then Reptile's meta-learning objective can be expressed as:
[0172]
[0173] in, This represents the optimal initial parameters of the policy network obtained through meta-learning.
[0174] The goal is to make the initial parameters of the policy network generally close to the adaptation parameters corresponding to different tasks, so that when the model faces a new continuous lane-changing task, it can obtain a policy that matches the current scenario with fewer parameter adjustments. Therefore, this model uses SAC to implement specific continuous lane-changing decisions, and Reptile to enhance the cross-task initialization capability of the policy network, ultimately outputting a policy applicable to different continuous lane-changing scenarios.
[0175] S.3 Model Training
[0176] This step takes the continuous lane-changing interaction state obtained from S.1 as input, and clusters the continuous lane-changing scenarios from two aspects: task urgency and risk balance. See the diagram below. Figure 2 The clustering results are used as the basis for model training.
[0177] S3.1 Feature Vector Construction for Continuous Lane Change Scenarios
[0178] According to the next target lane in S1.2 The set of lane change gaps constructed from S1.4 Select the lane change gap located in the next target lane. , forming a set:
[0179]
[0180] The number of lane change intervals on the next target lane is:
[0181]
[0182] Lane change clearance risk derived from S1.5 The average risk of the lane change gap in the next target lane is:
[0183]
[0184] To characterize the spatial urgency of continuous lane-changing tasks, the current driving pressure of the main vehicle, and the lane-changing pressure of the next target lane, this invention selects the remaining longitudinal distance for lane changing. Risks to the main vehicle Average risk of lane change gap with the next target lane As a scene feature vector.
[0185] S3.2 Two-stage Ward clustering for continuous lane change scenarios
[0186] Different vehicles require different times to complete continuous lane-changing tasks, meaning that the number of time frames spanned by each trajectory is inconsistent. Therefore, this invention first normalizes the continuous lane-changing time process and uses an equal-process segmentation method to transform each clustering index to the same dimension to eliminate time scale differences.
[0187] (1) Normalization of lane-changing process
[0188] For each consecutive lane-change trajectory sample, the original time... Mapped to the normalized lane-switching process axis:
[0189]
[0190] in, Indicates the start time of the lane change. This indicates the end time of the lane change. Through the above processing, regardless of whether the actual duration of each track is the same, they can be compared under a unified relative lane change process.
[0191] (2) Normalization process segmentation
[0192] Normalized lane change process range Divided into equal parts The number of sub-intervals, the first The number of subintervals is represented as:
[0193]
[0194] The normalized lane-changing process in each sample falls into the first... The set of time frames for each interval is represented as:
[0195]
[0196] Calculate the average of the three clustering indices for each interval, where the segmented feature of the longitudinal residual distance after lane change is:
[0197]
[0198] The risk segmentation characteristics of the main vehicle are as follows:
[0199]
[0200] The average risk segmentation characteristics of the next target lane candidate gap are as follows:
[0201]
[0202] (3) Clustering implementation
[0203] This invention categorizes continuous lane-changing scenarios into two levels based on the functions of different clustering indicators: a major category based on task urgency and a minor category based on risk balance. The major category on task urgency is determined by the remaining longitudinal distance of the lane change. Based on the clustering criteria, after performing equal-process segmentation, the following results are obtained:
[0204]
[0205] To eliminate different sub-intervals To standardize the impact of differences in value ranges:
[0206]
[0207] in, and Indicates that all samples are in the first... Remaining longitudinal distance for lane changing within each sub-section The mean and standard deviation.
[0208] After standardization, the feature vector of the first-stage clustering is:
[0209]
[0210] And for any two consecutive lane-changing trajectory samples and The characteristic differences are calculated using Euclidean distance:
[0211]
[0212] Based on the longitudinal residual distance feature vector of lane change and corresponding distance The Ward minimum variance clustering method was used to implement the first stage of clustering, which divided all consecutive lane-changing samples into major categories with different task urgency.
[0213] Then, within each major category, based on the risks faced by the main vehicle... Average risk of lane change gap with the next target lane The relative relationships between them are used for the second-stage clustering, namely, dividing them into risk balance subclasses. First, the first... The risk balance indicator within each sub-interval is:
[0214]
[0215] The feature vectors of the second-stage clustering are:
[0216]
[0217] For any two samples within the same major category and The difference in their risk relationships was calculated using Euclidean distance:
[0218]
[0219] Subsequently, for all continuous lane-changing trajectories within the same major category, based on the risk balance feature vector... and corresponding distance We then used the Ward minimum variance clustering method again to further divide the samples into subclasses with different risk balance levels.
[0220] S3.3 Model Task Distribution Construction
[0221] For the first-stage clustering, the remaining longitudinal distance around the lane change is formed. Clusters, namely The major categories of task urgency:
[0222]
[0223] For the The cluster centers of the major task urgency categories are represented as follows:
[0224]
[0225] Will Sort by size from smallest to largest. The smaller the value, the smaller the remaining longitudinal distance for lane changing, and the more urgent the continuous lane changing task. According to... The results are divided based on the number of elements, for example... At this time, lane changes can be divided into urgent lane change types and non-urgent lane change types; When changing lanes, the types can be categorized as urgent, moderate, and non-urgent.
[0226] Within each major category of task urgency, the second-stage clustering further divides the samples based on a risk balance index. The first subclass, for the first The first of the major categories of task urgency The cluster centers of the risk balance subclasses are represented as follows:
[0227]
[0228] The larger the value, the higher the risk to the vehicle in its current lane; The smaller the value, the higher the risk of lane change gap in the next target lane; The closer it is to 0, the more balanced the risk of the current lane and the risk of the lane change gap in the next target lane are. According to The results are divided based on the number of elements, for example... When the pressure is high, it can be classified into current lane pressure type, target lane pressure type, and pressure equal type.
[0229] The final category of continuous lane-changing scenarios is represented as follows:
[0230]
[0231] Based on the different continuous lane-changing scenario categories derived from clustering, the task distribution of the meta-reinforcement learning model is constructed as follows:
[0232]
[0233] In the formula, This indicates the results of two-stage clustering. The m-th consecutive lane-changing task is formed by the corresponding set of consecutive lane-changing samples. This indicates the total number of tasks.
[0234] S3.4 Model Parameter Update
[0235] For the model task distribution derived in S3.3 ,Will Training tasks were divided in a 7:3 ratio. With test tasks The training of the model is in The process is carried out in two parts: intra-task updates and inter-task updates. Intra-task updates use the SAC method to adapt the model to a specific continuous lane-changing task. The Reptile method is used for inter-task updates. The policy network meta-parameters are updated based on the policy adaptation results of multiple tasks, thereby extracting common decision-making knowledge among different consecutive lane-changing tasks.
[0236] (1) In-task update
[0237] SAC aims to maximize cumulative reward and policy entropy. The objective is to use a weighted sum as the target, enabling the model to learn high-reward lane-changing strategies while maintaining a certain level of exploration capability. This allows it to learn more reliable strategies in continuous lane-changing scenarios with varying lane-change intervals. The overall objective function is expressed as:
[0238]
[0239] in, The maximum number of decision steps in a single round. Discount factor; For the agent in state Next action The instant reward received; Representing state The entropy of the policy distribution; This refers to the temperature parameter.
[0240] Guided by the overall objective function, the parameters that SAC needs to update include the policy network parameters. Two value network parameters and and two target network parameters and Among them, value network , Update the policy network parameters using gradient descent based on the target Q-value output by the target network; The target network is updated using gradient descent based on the Q-value output of the value network; , Instead of updating directly through gradient descent, the SAC parameters are obtained through soft updates from the current value network. See the diagram for a schematic of the SAC parameter update. Figure 3 .
[0241] 1) Value network parameter update
[0242] For the current task The agent, based on existing strategies, operates in the current state. Sampling action :
[0243]
[0244] After the main vehicle performs the action, the environment transitions to the next state. And receive instant rewards. Interaction samples are stored in the experience replay pool:
[0245]
[0246] in, This indicates whether the current round should end. If the main vehicle reaches the final target lane and completes all lane-changing tasks, the task is considered complete and the round ends; if the main vehicle collides or reaches the maximum number of decision steps without completing the task, the task is considered failed and the round ends.
[0247] From the experience replay pool Random sampling is performed, with 256 interaction samples drawn each time to form a mini-batch sample set. Within this mini-batch sample set, actions for the next state are sampled.
[0248]
[0249] The objective value of this action is expressed as:
[0250]
[0251] Since the strategy itself is a joint strategy with two branches, it satisfies:
[0252]
[0253] in, Indicates the discount factor; Indicates whether the current round should end; Indicates the temperature coefficient; Indicates the first A target value network.
[0254] The value network is updated by minimizing the Bellman error, and its loss function is:
[0255]
[0256] The corresponding value network parameter update formula is:
[0257]
[0258] in, This represents the learning rate of the value network.
[0259] 2) Policy network parameter update
[0260] After the value network update is completed, the policy network generates gap selection actions and longitudinal control actions based on the current state using discrete action branches and continuous action branches, respectively. The discrete action branch is used to select the target lane change gap at the current time. According to the action space definition in S2.1, the tractor vehicle at time... The selectable gap numbers are:
[0261]
[0262] Discrete Actions The corresponding rating is ,when When that happens, the score for the corresponding action will be set to [value]. This avoids gaps in the policy network selection process that do not actually exist. The output gap selection probability is expressed as:
[0263]
[0264] The continuous action branch outputs the mean and standard deviation of the longitudinal acceleration distribution, and generates the longitudinal acceleration through reparameterization:
[0265]
[0266] in, and These represent the mean and standard deviation of the action distribution of the output of the continuous action branches, respectively. The noise is independent of the network parameters and follows a normal distribution.
[0267] Policy network parameters By maximizing the output of the value network The value is updated while maintaining the policy entropy, and the action output at this time is... Its loss function is:
[0268]
[0269] Similarly, due to the output joint strategy, the following must be satisfied:
[0270]
[0271] Finally, the formula for updating the policy network parameters is:
[0272]
[0273] in, This represents the learning rate of the policy network.
[0274] 3) Target network parameter update
[0275] The target network parameters are obtained from the current value network parameters through a soft update method:
[0276]
[0277] in, This represents the soft update coefficient.
[0278] By continuously updating the policy network and value network, the SAC model can eventually output longitudinal acceleration and lane gap selection results in specific continuous lane changing tasks, i.e., in a certain type of continuous lane changing scenario.
[0279] (2) Inter-task updates
[0280] In each round of meta-training, from Medium sampling One task. For any given task ,implement The above-mentioned task-adapted update yields the policy network parameters after task adaptation. For a batch of parameters updated by SAC for a specific task. The Reptile method is used to process the meta-parameters. Optimize:
[0281]
[0282] in, This indicates the learning step size.
[0283] See the diagram illustrating the Reptile parameter update. Figure 4 Through the above meta-training, The system was gradually optimized to suit various types of continuous lane-changing tasks, ultimately yielding the optimal meta-parameters. During the meta-testing phase, it is used as a test task set. China-Singapore mission The initial parameters of the policy network are determined, and then updated with a small number of SAC network updates to obtain parameters adapted to the characteristics of the task. This allows the model to output corresponding strategies and perform subsequent evaluations. Compared to reinforcement learning models trained only in a single scenario, the meta-reinforcement learning model in this invention can utilize common information between different scenarios, enabling it to adapt to the current interactive environment more quickly when facing new continuous lane-changing scenarios.
[0284] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the program, when executed on the processor, implements the steps in the autonomous driving continuous lane-changing decision-making method based on meta-reinforcement learning as described above.
[0285] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps in the autonomous driving continuous lane-changing decision-making method based on meta-reinforcement learning as described above.
[0286] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding process in the aforementioned method implementation, and will not be repeated here.
[0287] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0288] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.
[0289] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer system (which may be a personal computer, server, or network system, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0290] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for continuous lane-changing decision-making in autonomous driving based on meta-reinforcement learning, characterized in that: The method uses the SAC algorithm as the basic decision-making method to construct a continuous lane-changing decision model, and then realizes the decision-making for continuous lane changing in autonomous driving based on the trained continuous lane-changing decision model; wherein, The continuous lane-changing decision-making model includes a policy network, two value networks, and two corresponding target networks; The continuous lane-changing decision model takes the continuous lane-changing interaction state as input and the lane-changing gap selection and longitudinal acceleration of the main vehicle as action output.
2. The method according to claim 1, characterized in that: The continuous lane-changing interaction state includes the vehicle's motion state from the start to the end of the continuous lane-changing process, the vehicle's lane-changing process, the set of lane-changing gaps, the risks experienced by the vehicle, and the set of lane-changing gap risks. The motion state of the main vehicle at time t from the start to the end of the continuous lane change. Represented as: , In the formula, Let represent the ordinate of the center position of the main vehicle at time t, the abscissa of the center position, the longitudinal velocity, the lateral velocity, the longitudinal acceleration, the lateral acceleration, and the lane number it is in. The main train lane-changing process at time t Represented as: , In the formula, These represent the next target lane number of the main vehicle at time t, the remaining longitudinal distance from the lane change endpoint, and the final target lane number, respectively. Set of lane change gaps at time t Represented as: , In the formula, , This represents the total number of lane-changing gaps at time t. This represents the state of the k-th lane change interval at time t. , These represent the longitudinal position of the center of the k-th lane change gap at time t, the lateral position of the center, and the gap length, respectively. These represent the longitudinal distances between the main vehicle and the vehicle in front and behind at time t during the k-th lane change interval, respectively. Let represent the longitudinal relative velocities between the main vehicle and the preceding and following vehicles at time t during the kth lane change interval, respectively; This represents the lane number where the k-th lane change interval is located at time t; Risks suffered by the main vehicle at time t By determining the center position of the main vehicle at time t Substituting into the road safety potential field formula yields the following: Risk set of lane change gap at time t Represented as: , In the formula, This represents the risk of the k-th lane change gap at time t. By determining the center position of the k-th lane change gap at time t Substituting into the road safety potential field formula, we get... These represent the longitudinal and lateral coordinates of the center position of the k-th lane change gap at time t, respectively.
3. The method according to claim 2, characterized in that: Based on the continuous lane-changing interaction state, the state space of the continuous lane-changing decision model at time t is... Represented as: ; Action space of the continuous lane-changing decision model at time t Represented as: , In the formula, This indicates the lane change interval number selected by the main vehicle at time t; Reward function of the continuous lane-changing decision model at time t Represented as: , In the formula, They are respectively Weighting coefficients; These represent the safety reward, efficiency reward, comfort reward, and task completion reward at time t, respectively. , This indicates the risk penalty coefficient suffered by the main vehicle. This represents the lane change clearance risk penalty coefficient. Represents the time t. Risk of lane change gap; , Indicates the desired speed. This indicates the total longitudinal travel distance for consecutive lane changes. Indicates the start time of the lane change. This indicates whether the main vehicle is in a following mode. , , and These represent the speed bonus coefficient, the remaining distance penalty coefficient, and the following time penalty coefficient, respectively. , Indicates the acceleration penalty coefficient; , express Whether the main vehicle has completed a lane change at any given time. , express Whether the main vehicle has completed all lane-changing tasks at any given time. , This indicates whether the lane-changing task is terminated at time t due to a collision or exceeding the maximum number of decision steps. , , and These represent the stage lane change reward coefficient, the final task completion reward coefficient, and the task failure penalty coefficient, respectively.
4. The method according to claim 3, characterized in that: The policy network consists of a shared feature extraction layer, a discrete action output branch, and a continuous action output branch. The shared feature extraction layer is used to extract common input features in the state space; the discrete action output branch is used to output the corresponding lane change gap selection probability based on the common input features; and the continuous action output branch is used to output the longitudinal acceleration that matches the lane change gap based on the common input features.
5. The method according to claim 4, characterized in that: It also includes, Build task distribution and will The training tasks are divided according to a set ratio. With test task distribution This is to enable the training and testing of the continuous lane-changing decision model; Among them, task distribution The construction steps include: 1) Construct a feature vector for a continuous lane-changing scenario, consisting of the remaining longitudinal distance of the main vehicle from the lane-changing endpoint, the risk experienced by the main vehicle, and the average risk of the lane-changing gap in the next target lane; where the average risk of the lane-changing gap in the next target lane at time t is expressed as: , In the formula, This represents the set of lane-changing gaps in the next target lane at time t. , Indicates the number of lane change gaps on the next target lane; 2) Perform two-stage clustering on continuous lane-changing scenarios, specifically including: by Based on the clustering criteria, the Ward minimum variance clustering method was used to implement the first stage of clustering, dividing all consecutive lane-changing samples into... The task urgency categories are represented as follows: , In the formula, This represents the results of the first stage of clustering. Indicates the first The tasks are categorized by urgency. Within each major category of task urgency, a second-stage clustering process is implemented based on the risk balance index, further dividing the continuous lane-changing samples into... There are several risk balance subcategories; among them, the lane-changing process is... Risk balance index for each sub-interval Represented as: , In the formula, This indicates that the main vehicle is in the lane-changing process at the [number]th stage. The risks within each sub-interval, , For the lane changing process The duration of each sub-interval, This indicates the lane change interval for the next target lane in the lane change process. Average risk within each sub-interval ; The final two-stage clustering results for all consecutive lane-changing samples are represented as follows: , In the formula, This represents the results of two-stage clustering. Indicates the first The first of the major categories of task urgency Each risk balancing subcategory; 3) Based on the two-stage clustering results, a task distribution is constructed. .
6. The method according to claim 5, characterized in that: The training of the continuous lane-changing decision model consists of two parts: intra-task updates and inter-task updates. 1) In-task updates include: Within the training task, the SAC method is adopted. Under the guidance of the overall objective function, the value network is first updated by minimizing the Bellman error, then the policy network is updated by maximizing the Q value of the value network output while maintaining the policy entropy, and finally the target network is updated by soft update according to the corresponding value network. 2) Inter-task updates include: from Several training tasks were sampled, and the Reptile method was used to optimize the policy network parameters obtained by updating within the task.
7. The method according to claim 6, characterized in that: The loss function of the value network is expressed as: , In the formula, , Represents the i-th value network loss function, Represents the i-th value network Q-value estimation for the current state-action pair. This represents the time-difference objective Q-value. , .
8. The method according to claim 7, characterized in that: The loss function of the policy network is expressed as: , In the formula, .
9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can run on the processor, and when the program is executed on the processor, it implements the steps of the method as described in any one of claims 1-8.
10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-8.
Citation Information
Patent Citations
Multi-vehicle cooperative lane changing method based on multi-agent reinforcement learning
CN122116686A
Vehicle lane change depth reinforcement learning decision-making method for lane reduction area
CN122116687A
Lane changing decision-making method for intelligent networked automobiles at signalized intersection in vehicle-road coordination environment
CN122290357A
