Reinforcement learning-based left-turn decision planning method and system for autonomous vehicle

Abstract

This invention discloses a left-turn decision-making and planning method and system for autonomous vehicles based on reinforcement learning, belonging to the field of autonomous driving decision-making and planning technology. The method involves: constructing paired interaction features based on the vehicle's and the environment's vehicle states; building a simulated left-turn environment at an unsignalized intersection using real trajectory data to filter difficult scenarios; training an action-level risk prediction network to suppress high-risk actions in real time; dynamically weighting and aggregating the environment's vehicles through an attention mechanism to generate a context vector, which is then concatenated with the vehicle's state to form an enhanced state representation; and constructing a hierarchical reward function-guided strategy network that balances safety and efficiency. In difficult scenarios, a proximal policy optimization algorithm is used for training, outputting acceleration and front wheel steering angle control commands. Based on this method, a corresponding system is also proposed. This invention achieves dynamic focusing of key vehicles through an attention mechanism, combining risk prediction and hierarchical reward collaborative optimization to improve the safety, traffic efficiency, and human-like smoothness of left-turn decisions.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving decision-making and planning technology, and specifically relates to a method and system for left-turn decision-making and planning of autonomous vehicles based on reinforcement learning. Background Technology

[0002] In autonomous driving, decision-making and planning are crucial, significantly ensuring vehicle safety and traffic efficiency in complex traffic scenarios by combining environmental perception and vehicle motion control commands. Currently, reinforcement learning-based decision-making methods are widely used in high-interaction scenarios such as left turns at unsignalized intersections. However, while ensuring safety, reinforcement learning strategies still suffer from insufficient traffic efficiency, exhibiting overly conservative approaches in some scenarios and difficulty in timely identifying passable gaps. A significant reason for this problem lies in insufficient state representation capabilities. Meanwhile, while supervised learning-based risk assessment methods can predict potential hazards, they struggle to achieve continuous decision-making and long-term benefit optimization, and cannot directly generate smooth, human-like driving actions. Therefore, a single learning framework is insufficient to meet the decision-making requirements for synergistic safety and efficiency in complex intersection scenarios.

[0003] Furthermore, in left-turn scenarios at unsignalized intersections, traditional policy networks typically use average pooling or max pooling to aggregate environmental vehicle features in their state encoding. While this method can extract features from surrounding vehicles, it struggles to differentiate the importance of different vehicles to the current decision, easily weakening key interaction information and affecting the policy's accurate judgment of when to proceed. Existing technologies mainly rely on fixed-structure feature encoding and manually designed reward functions. These methods usually assume that environmental vehicles are of equal importance, making it difficult to adaptively focus on key interaction targets. In scenarios with dense multi-vehicle traffic and uncertain behavior, they are prone to problems such as decision lag and low traffic efficiency.

[0004] Therefore, how to achieve dynamic perception of key interactive vehicles, improve state representation capabilities, and improve left-turn efficiency while ensuring safety remains a significant challenge for current autonomous driving decision-making and planning technologies. Summary of the Invention

[0005] To address the aforementioned technical challenges, this invention proposes a left-turn decision-making and planning method and system for autonomous vehicles based on reinforcement learning. By dynamically focusing on key interacting vehicles through an attention mechanism, combining a risk prediction network for real-time suppression of high-risk actions, and employing a hierarchical reward function-guided strategy that balances safety and efficiency, this method effectively solves the problems of difficulty in identifying key vehicles and low traffic efficiency caused by conservative or aggressive decision-making in left-turn scenarios at unsignalized intersections. It significantly improves the safety, efficiency, and human-like smoothness of left-turn decision-making in autonomous driving.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, this invention proposes a left-turn decision-making and planning method for autonomous vehicles based on reinforcement learning, comprising the following steps: Based on the acquired vehicle state information and environmental vehicle state information, a pairwise interaction feature between the vehicle and each environmental vehicle is constructed; based on the acquired real vehicle historical trajectory information, a left-turn reinforcement learning simulation environment for unsignaled intersections is constructed, and a state space and action space are defined in the simulation environment, as well as difficult scenarios with difficulty scores greater than a preset threshold are identified from the simulation environment. A supervised learning approach is adopted, using state-action pairs in the state space as input and the future collision probability under the state-action pair condition as the prediction target to train an action-level risk prediction network for real-time suppression of high-risk actions. The environmental vehicle state information is encoded into an environmental vehicle embedding vector, and the vehicle state information is transformed into a query vector through linear projection. The environmental vehicle embedding vector and the query vector are concatenated to form multiple joint feature vectors. Based on each joint feature vector, a multilayer perceptron is used for scoring and mapping. The scoring results are used to weight and aggregate the environmental vehicle embedding vector to generate a context vector. The context vector is concatenated with the vehicle state information to form an enhanced state representation. A hierarchical reward function is constructed to guide the policy network to balance safety and traffic efficiency in a left-turn scenario at an unsignalized intersection. In this challenging scenario, the enhanced state representation is used as the input to both the policy network and the value network, and the hierarchical reward function is used as the optimization objective. The policy network is trained using a proximal policy optimization algorithm. After training, the policy network outputs a left-turn control command containing the desired acceleration and the desired front wheel angle based on the enhanced state representation at the current moment.

[0007] Furthermore, based on the acquired vehicle state information and environmental vehicle state information, pairwise interaction features between the vehicle and each environmental vehicle are constructed, specifically: The vehicle status information Including vehicle location Bicycle speed and vehicle heading angle ; The environmental vehicle status information Including the Location of environmental vehicles ,speed and heading angle ; The paired interaction features include the relative distance between the vehicle and each ambient vehicle. Relative velocity and relative heading angle ; ; ; ; in, , This indicates the total number of vehicles in the environment at the current moment.

[0008] Furthermore, the difficulty score is calculated based on the number of vehicles in the current scene, the absolute value of the average relative speed between the vehicle and all vehicles in the scene, and the length of the time gap during which the vehicle can safely pass.

[0009] Furthermore, a supervised learning approach is adopted, using state-action pairs in the state space as input and the future collision probability under the given state-action pair conditions as the prediction target, to train an action-level risk prediction network for real-time suppression of high-risk actions; specifically: Multiple candidate states are collected from the simulation environment, and each candidate state is input into the policy network to obtain the corresponding candidate action output by the policy network under the candidate state; each candidate state is combined with the corresponding candidate action to construct multiple candidate state-action pairs. Based on the real vehicle historical trajectory information, each candidate state-action pair is labeled with a corresponding future collision probability label; using the candidate state-action pair as input features and the future collision probability label as the prediction target, a supervised learning method is used to train the action-level risk prediction network. After training, the current state and the current action output by the policy network are used to form a current state-action pair, which is then input into the action-level risk prediction network. When the output collision probability exceeds a preset safety threshold, the current action is suppressed.

[0010] Furthermore, the environmental vehicle state information is encoded into an environmental vehicle embedding vector, and the vehicle state information is transformed using a linear projection to generate a query vector; the environmental vehicle embedding vector and the query vector are concatenated to form multiple joint feature vectors; specifically: The state information of each environmental vehicle is encoded into an environmental vehicle embedding vector. Vehicle status information Generate query vector through linear projection transformation ; the query vector With each environment vehicle embedding vector The features are concatenated to form a joint feature vector. ; ; ; in, Represents a linear projection matrix; This indicates the bias term.

[0011] Furthermore, based on the joint feature vectors, a multilayer perceptron is used for scoring and mapping. The scoring results are then used to weight and aggregate the environmental vehicle embedding vectors to generate a context vector; specifically: The joint feature vector Input a scoring function consisting of two layers of sensing mechanisms, and calculate the first... Original importance score of each environmental vehicle ; ; The original importance scores of all valid environmental vehicles are normalized to obtain the [number]th [value]. Attention weight of vehicles in the environment Wherein, the effective environmental vehicle is an environmental vehicle that actually exists at the current moment and is not masked; ; The attention weights of each environmental vehicle are used to weight the corresponding environmental vehicle embedding vectors, and all weighted environmental vehicle embedding vectors are summed to generate a context vector. ; ; in, and These represent the linear mapping parameter matrices of the first and second layers of the two-layer perceptron, respectively. and These represent the corresponding bias terms; This represents the set of vehicles in the current effective environment. Indicates the first The original importance scores of each environmental vehicle; Indicates the first Attention weights for each vehicle in the environment.

[0012] Furthermore, the hierarchical reward function includes sparse result-driven rewards and continuous behavior-guided rewards; The result-driven reward is assigned based on the collision detection results fed back by the simulation environment and whether the left-turn task was successfully completed; The behavioral guidance rewards include a heading angle change reward calculated based on the heading angle change, a position-related reward calculated based on the distance between the vehicle's current position and the centerline of the target lane, and a reference trajectory guidance reward calculated based on the deviation between the vehicle's actual trajectory and a preset reference trajectory.

[0013] Furthermore, the results drive rewards Represented as: ; in, This indicates that the reward is driven by the outcome. Indicates time Collision type marker; Indicates no collision; Indicates a lane collision; Indicates a car-to-car collision; Indicates time The left turn task is completed; This indicates that the left turn has been completed; This represents the reward value corresponding to a lane collision. This represents the reward value corresponding to a vehicle-to-vehicle collision; This represents the reward value for successfully completing the left turn task; , and All are constants and satisfy the following conditions: .

[0014] Furthermore, the behavior-guided reward Represented as:

[0015] in, This indicates a reward based on changes in heading angle; This indicates location-related rewards; This indicates the reference trajectory guiding the reward items; Indicates at time state; Indicates at time action.

[0016] Secondly, this invention also proposes a left-turn decision-making and planning system for autonomous vehicles based on reinforcement learning, comprising: The environment interaction module is used to construct paired interaction features between the self-vehicle and each environmental vehicle based on the acquired self-vehicle state information and environmental vehicle state information; to construct a reinforcement learning simulation environment for left turns at unsignalized intersections based on the acquired real vehicle historical trajectory information; to define the state space and action space in the simulation environment; and to identify difficult scenarios with difficulty scores greater than a preset threshold from the simulation environment. The risk prediction module is used to train an action-level risk prediction network using supervised learning, taking state-action pairs in the state space as input and the future collision probability under the state-action pair conditions as the prediction target, in order to achieve real-time suppression of high-risk actions. The encoding and concatenation module is used to encode environmental vehicle state information into environmental vehicle embedding vectors, and generate query vectors from the vehicle state information through linear projection transformation; the environmental vehicle embedding vectors and query vectors are concatenated to form multiple joint feature vectors; based on each joint feature vector, a multilayer perceptron is used for scoring and mapping, and the scoring results are used to perform weighted aggregation on the environmental vehicle embedding vectors to generate context vectors; the context vectors are concatenated with the vehicle state information to form an enhanced state representation; The decision output module is used to construct a hierarchical reward function to guide the policy network in a left-turn scenario at an unsignalized intersection, balancing safety and traffic efficiency. In this challenging scenario, the enhanced state representation is used as the input to the policy network and the value network, and the hierarchical reward function is used as the optimization objective. The policy network is trained using a proximal policy optimization algorithm. After training, the policy network outputs a left-turn control command containing the desired acceleration and desired front wheel angle based on the enhanced state representation at the current moment.

[0017] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: (i) Improve the recognition accuracy and state representation capability of key interactive vehicles. This invention introduces attention and masking mechanisms into a reinforcement learning state encoding network. The vehicle's state is linearly projected to generate a query vector, which is then concatenated with the embedding vectors of each environmental vehicle before being input into a scoring network. This achieves dynamic evaluation and differentiated weighting of the importance of environmental vehicles. Compared to traditional methods that use average pooling or max pooling for feature aggregation, this invention effectively distinguishes the differences in importance of different environmental vehicles to the current left-turn decision, preventing key interaction information from being weakened or buried. The level of attention weights directly reflects the degree of influence of each environmental vehicle on the vehicle's decision, enabling the policy network to adaptively focus on the target vehicle with the highest conflict risk or the greatest impact on the timing of passage, thereby significantly improving the accuracy of state representation and the reliability of decision-making.

[0018] (ii) Achieving synergistic optimization of safety and traffic efficiency This invention addresses the technical challenge of balancing safety and efficiency in existing technologies by jointly designing an action-level risk prediction network and a hierarchical reward function. On one hand, the action-level risk prediction network, trained using supervised learning, models risk as the probability of future collisions under state-action conditions. This enables real-time safety assessment of candidate actions output by the policy network. When the collision probability exceeds a preset safety threshold, high-risk actions are preemptively suppressed, avoiding potential collision risks at the decision-making level. On the other hand, the hierarchical reward function clearly distinguishes between acceptable and unacceptable behaviors through sparse result-driven rewards. Continuous behavior-guided rewards provide stable local guidance signals in the early stages of training, allowing the policy network to obtain both clear task objective guidance and fine-grained behavior optimization guidance during the learning process. The synergistic effect of these two mechanisms enables the policy to promptly identify passable gaps while ensuring safety constraints, significantly improving left-turn efficiency.

[0019] (iii) Generate smooth human-like driving trajectories This invention uses an attention mechanism to generate a context vector, which is then concatenated with the vehicle's state information to form an enhanced state representation. This enables the policy network to more accurately understand the semantic information of the current interaction scenario, thereby outputting a left-turn control command that is more consistent with the decision-making logic of a human driver. Simultaneously, the reference trajectory guidance reward introduced in the behavior guidance reward constrains the deviation between the vehicle's actual trajectory and the preset reference trajectory. A position-related reward guides the vehicle towards the center area of ​​the target lane, and a heading angle change reward encourages smooth steering operations. The combined effect of these three elements results in better continuity and smoothness in the final generated control command, avoiding the jitter, sharp turns, or hesitations common in traditional reinforcement learning methods, thus improving passenger comfort and traffic efficiency.

[0020] (iv) Enhancing decision-making robustness in challenging scenarios This invention selects challenging scenarios with difficulty scores exceeding a preset threshold from a simulation environment and specifically trains the policy network using reinforcement learning in these high-risk, high-interaction-density scenarios, ensuring the policy network receives sufficient coverage of challenging samples during the training phase. The difficulty score comprehensively considers multiple dimensions, including the number of vehicles in the environment, the absolute value of the average relative speed between the vehicle and all other vehicles, and the length of the safe passage interval for the vehicle, ensuring that the selected scenarios have high decision-making difficulty and risk levels. Through targeted training in these challenging scenarios, the policy network learns more robust decision boundaries, maintaining stable decision-making performance even in complex left-turn scenarios with dense multi-vehicle traffic and uncertain behavior, significantly reducing the risk of decision-making errors in extreme scenarios.

[0021] (v) Construct a complete technology closed loop to achieve end-to-end decision-making and planning. This invention forms a complete technical closed loop, encompassing state acquisition, interaction feature construction, attention encoding, risk prediction, hierarchical reward guidance, policy training, and control command output. The modules collaborate through clear data flows and interface definitions: the enhanced state representation output by the attention encoding module serves as input to the policy network, value network, and risk prediction network; the hierarchical reward function provides a unified optimization objective for the policy network; and the risk prediction network performs security verification on the policy network's output. This integrated design gives the entire decision-making and planning system excellent scalability and deployability, enabling direct application in real-world autonomous driving left-turn scenarios and demonstrating clear engineering practical value. Attached Figure Description

[0022] Figure 1 This is a flowchart of the left-turn decision-making and planning method for autonomous vehicles based on reinforcement learning proposed in Embodiment 1 of the present invention; Figure 2 This is a diagram illustrating the overall framework of the reinforcement learning-based left-turn decision-making and planning method for autonomous vehicles proposed in Embodiment 1 of the present invention. Figure 3 This is a general framework diagram of the attention interaction perception state encoding module proposed in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the left-turn decision-making and planning system for autonomous vehicles based on reinforcement learning proposed in Embodiment 2 of the present invention. Detailed Implementation

[0023] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0024] Example 1 Embodiment 1 of this invention proposes a left-turn decision-making and planning method for autonomous vehicles based on reinforcement learning, which is used to solve the problems of difficulty in balancing safety and efficiency and difficulty in identifying key vehicles in the left-turn scenario at unsignalized intersections in the prior art.

[0025] Figure 1 This is a flowchart of the left-turn decision-making and planning method for autonomous vehicles based on reinforcement learning proposed in Embodiment 1 of the present invention; In step S1, based on the acquired vehicle state information and environmental vehicle state information, a pairwise interaction feature between the vehicle and each environmental vehicle is constructed; based on the acquired real vehicle historical trajectory information, a left-turn reinforcement learning simulation environment for unsignaled intersections is constructed, and a state space and action space are defined in the simulation environment, as well as difficult scenarios with difficulty scores greater than a preset threshold are identified from the simulation environment. Vehicle status information Including vehicle location Bicycle speed and vehicle heading angle ; Environmental vehicle status information Including the Location of environmental vehicles ,speed and heading angle ; Therefore, pairwise interaction features include the relative distance between the vehicle and each surrounding vehicle. Relative velocity and relative heading angle ; ; ; ; in, , This indicates the total number of vehicles in the environment at the current moment; The actual location of a vehicle is represented as follows: The vehicle speed is expressed as The heading angle is expressed as .

[0026] In this application, a reinforcement learning simulation environment for left turns at unsignalized intersections is constructed based on acquired real vehicle historical trajectory information. A state space and action space are defined within this simulation environment, and difficult scenarios with difficulty scores exceeding a preset threshold are identified from the simulation environment. The detailed process includes: This invention constructs a reinforcement learning simulation environment for left turns at unsignalized intersections based on real vehicle historical trajectory information from real driving datasets. Specifically, it extracts the position, speed, and heading angle sequences of real vehicles during left turns at unsignalized intersections from real driving datasets (such as nuScenes, Waymo Open Dataset, etc.). Based on the position sequences in the real vehicle historical trajectory information, it extracts the lane topology of the unsignalized intersection, including the geometric connection relationship between each approach lane and exit lane, lane centerline coordinates, and width information. It also extracts conflict area boundaries, which are the intersection areas of the left-turn trajectory of the vehicle with the trajectories of oncoming straight-ahead vehicles and lateral conflicting vehicles. Finally, it extracts traffic priority rules, including yield rules and right-of-way allocation logic in the unsignalized intersection scenario. Based on the extracted information, it reconstructs the static road scene of the unsignalized intersection in the simulation environment and generates the initial positions and motion states of vehicles in the environment that conform to the real traffic flow distribution. This ensures that the distribution of the interactive scene in the simulation environment is consistent with the distribution of real data, thereby ensuring that the trained strategy has real-world transferability.

[0027] The state space includes the vehicle kinematic state variables composed of the vehicle's state information and the environmental vehicle kinematic state variables composed of the environmental vehicle's state information.

[0028] The motion space includes longitudinal acceleration control and front wheel steering angle control, wherein the value range of the longitudinal acceleration control is: The range of the front wheel steering angle control value is: .in, This represents the preset absolute value of the maximum acceleration; This represents the preset maximum absolute value of the front wheel steering angle.

[0029] The difficulty score is calculated based on the number of vehicles in the current scene, the absolute value of the average relative speed between the vehicle and all vehicles in the scene, and the length of the time gap that the vehicle can safely pass.

[0030] Multiple left-turn scenarios are randomly sampled from the simulation environment. For each sampled scenario, the total number of vehicles in the environment at the current time is... Based on the average absolute value of the relative speed between the vehicle and all surrounding vehicles. Determine the relative speed index, among which The time interval length at which the vehicle can safely pass is calculated based on the time it takes for the vehicle to reach the conflict zone along the planned path from its current position and the time it takes for other vehicles in the environment to reach the same conflict zone. The difficulty score for this scenario is calculated using the following formula. :

[0031] in, , and These represent the preset positive weight coefficients. The calculated difficulty score... With preset threshold Compare and filter those that meet the requirements. The scenarios are designated as difficult scenarios. A higher difficulty score indicates a greater number of vehicles, higher relative speeds, and shorter gaps between vehicles in the scenario, resulting in higher decision-making difficulty and risk levels. This ensures that the selected difficult scenarios have targeted training value. Training the policy network specifically under these difficult scenarios allows the network to obtain sufficient coverage of difficult samples during the training phase, learning more robust decision boundaries.

[0032] In step S2, a supervised learning approach is adopted, using state-action pairs in the state space as input and the future collision probability under the state-action pair condition as the prediction target to train an action-level risk prediction network to achieve real-time suppression of high-risk actions. This invention employs supervised learning to train an action-level risk prediction network for safety assessment of candidate actions output by a policy network. Specifically, a large number of candidate states covering various traffic scenarios are collected from the simulation environment. These candidate states include the vehicle's kinematic state and the kinematic states of surrounding vehicles. Each candidate state is input into the current policy network to obtain the corresponding candidate action output by the policy network under that state. Each candidate state is combined with its corresponding candidate action to construct multiple candidate state-action pairs. .

[0033] Simultaneously, based on the real vehicle historical trajectory information, a corresponding future collision probability label is assigned to each candidate state-action pair. The future collision probability label is generated as follows: for each candidate state... Next action The frequency of collisions occurring within a preset time window is used as a monitoring label for the future collision probability of the state-action pair. Candidate state-action pairs... Using the input features and the future collision probability labels as the prediction targets, an action-level risk prediction network is trained using supervised learning. This ensures that its output accurately reflects the state. Next action The network determines the collision risk. It employs a deep neural network structure and optimizes network parameters by minimizing the cross-entropy loss or mean squared error loss between the predicted collision probability and the label.

[0034] After training, the action-level risk prediction network runs in real time during the reinforcement learning decision-making process. Specifically, at each step of the policy network's decision-making, the current state is... The current candidate action output by the policy network Constructing the current state-action pair Inputting the action-level risk prediction network yields the predicted collision probability value corresponding to the state-action pair. The predicted collision probability value is compared with a preset safety threshold. Comparison: When When a candidate action is deemed high-risk, it is suppressed in real time, preventing it from being sent to the vehicle's underlying actuators and triggering the policy network to regenerate an alternative action or select a default safe alternative action (such as deceleration and stopping). If the current action is deemed safe, it is allowed to proceed normally.

[0035] Through the aforementioned mechanism, this invention achieves real-time pre-emptive suppression of high-risk actions. Without affecting the exploration efficiency of the policy network, it effectively avoids potential collision risks at the decision-making level, significantly improving the safety of left-turn decisions. This risk prediction network operates in parallel with the policy network, with inference latency controlled at the millisecond level, meeting the real-time requirements of autonomous driving systems.

[0036] In step S3, the environmental vehicle state information is encoded into an environmental vehicle embedding vector, and the vehicle state information is transformed by linear projection to generate a query vector. The environmental vehicle embedding vector and the query vector are concatenated to form multiple joint feature vectors. Based on each joint feature vector, a multilayer perceptron is used for scoring and mapping. The scoring results are used to weight and aggregate the environmental vehicle embedding vectors to generate a context vector. The context vector is concatenated with the vehicle state information to form an enhanced state representation. Figure 3 This is a general framework diagram of the attention interaction perception state encoding module proposed in Embodiment 1 of the present invention; (I) Vehicle Feature Projection and Query Vector Generation The input to the attention-based interactive perception state encoding module includes the vehicle's state information and the environment's vehicle state information. The vehicle's state information first passes through a vehicle feature projection layer, where a linear projection transformation maps the vehicle's state vector to the same feature space as the environment's vehicle embedding vector, generating a query vector. This projection layer includes a linear transformation and a ReLU activation function, encoding the vehicle's kinematic features (position, velocity, heading angle) into a fixed-dimensional query vector through a single-layer or multi-layer fully connected network. This allows the query vector to be interactively computed with the subsequent environment's vehicle embedding vector in the same feature space.

[0037] (II) Environmental Vehicle Input Representation and Feature Encoding Environmental vehicle state information is processed in parallel through an environmental vehicle feature encoding layer. The kinematic state (position, velocity, heading angle) of each environmental vehicle is mapped to an environmental vehicle embedding vector through a linear layer, forming the environmental vehicle input representation. These embedding vectors preserve the independent feature information of each environmental vehicle, providing a foundation for subsequent attention weight calculation. The encoded environmental vehicle embedding vectors and the expanded query vector are concatenated along the channel dimension to form a joint feature vector, enabling a one-to-one association between the vehicle's information and the information of each environmental vehicle.

[0038] (III) Score Mapping between Score MLP and Masking Mechanism The concatenated joint feature vector is input into a scoring multilayer perceptron, which consists of hidden layers containing ReLU activation functions and a linear output layer. The scoring multilayer perceptron maps each joint feature to a scalar score, i.e., the original importance score of each environmental vehicle. Based on this, this invention introduces a masking mechanism to block invalid environmental vehicles that are outside a preset perception distance or within the vehicle's blind spot, setting their original importance scores to extremely low values ​​or removing them directly, thereby avoiding interference from invalid vehicles on the attention weight distribution.

[0039] (iv) Softmax normalization and attention weight generation After masking, the environmental vehicle scores are normalized using the Softmax function to generate an attention weight for each environmental vehicle. The Softmax operation ensures that the sum of the attention weights of all valid environmental vehicles is 1, so that the weight values ​​can be directly interpreted as the proportion of importance of each environmental vehicle in the current decision. Environmental vehicles with higher weights indicate that they have a greater impact on the vehicle's left-turn decision and should be given priority; vehicles with weights close to zero are judged by the model to have low relevance to the vehicle's interaction at the current moment, and their impact on the decision can be ignored.

[0040] (v) Weighted aggregation and context vector generation The corresponding environmental vehicle embedding vectors are weighted using the attention weights of each environmental vehicle, and then aggregated by weighted summation to generate a context vector. This process can be viewed as selective attention reading of environmental vehicle information, with the attention weights acting as gating signals to control the contribution of each environmental vehicle's information to the context vector. The context vector comprehensively reflects the information of the most important interactive objects for the vehicle's decision-making in the current scene.

[0041] (vi) Formation of the final state representation The generated context vector is concatenated with the original vehicle state information along the channel dimension to form the final state representation (i.e., the enhanced state representation). This final state representation simultaneously includes the vehicle's own kinematic state and environmental interaction information filtered and aggregated by the attention mechanism. It is then input into the policy network and value network to support subsequent action selection and state evaluation. Through this framework, this invention achieves dynamic perception and adaptive focusing of key interactive vehicles, enabling the policy network to more accurately determine the timing of passage and significantly improving decision-making quality and traffic efficiency in left-turn scenarios at unsignalized intersections.

[0042] Specifically, the state information of each environmental vehicle is encoded into an environmental vehicle embedding vector. Vehicle status information Generate query vector through linear projection transformation ; the query vector With each environment vehicle embedding vector The features are concatenated to form a joint feature vector. ; ; ; in, Represents a linear projection matrix; This indicates the bias term.

[0043] The joint feature vector Input a scoring function consisting of two layers of sensing mechanisms, and calculate the first... Original importance score of each environmental vehicle ; ; The original importance scores of all valid environmental vehicles are normalized to obtain the [number]th [value]. Attention weight of vehicles in the environment Wherein, the effective environmental vehicle is an environmental vehicle that actually exists at the current moment and is not masked; ; The attention weights of each environmental vehicle are used to weight the corresponding environmental vehicle embedding vectors, and all weighted environmental vehicle embedding vectors are summed to generate a context vector. ; ; in, and These represent the linear mapping parameter matrices of the first and second layers of the two-layer perceptron, respectively. and These represent the corresponding bias terms; This represents the set of vehicles in the current effective environment. Indicates the first Attention weighting for each vehicle in the environment; Indicates the first The initial importance score of each vehicle in the environment; the higher the score, the greater the influence of the vehicle on the driver's decision-making. Through nonlinear mapping of a two-layer perceptron, the scoring network can learn complex interaction patterns, including the combined influence of paired interaction features such as relative position, relative speed, and relative heading angle.

[0044] This context vector comprehensively reflects the information of the most important interactive objects for the vehicle's decision-making in the current scenario, and can be further input into the policy network or value network to support subsequent action selection and state evaluation.

[0045] In step S4, a hierarchical reward function is constructed to guide the policy network to balance safety and traffic efficiency in a left-turn scenario at an unsignalized intersection. In the difficult scenario, the enhanced state representation is used as the input to the policy network and the value network, and the hierarchical reward function is used as the optimization objective. The policy network is trained using a proximal policy optimization algorithm. After training, the policy network outputs a left-turn control command containing the desired acceleration and the desired front wheel angle based on the enhanced state representation at the current moment.

[0046] The hierarchical reward function in this invention includes sparse result-driven rewards and continuous behavior-guided rewards; The result-driven reward is assigned based on the collision detection results fed back from the simulation environment and whether the left turn task was successfully completed; the behavior guidance reward includes a heading angle change reward calculated based on the heading angle change, a position-related reward calculated based on the distance between the vehicle's current position and the center line of the target lane, and a reference trajectory guidance reward calculated based on the deviation between the vehicle's actual trajectory and the preset reference trajectory.

[0047] This invention employs a hierarchical reward design that combines outcome-driven sparse rewards with behavior-driven continuous rewards. It integrates sparse but semantically clear key events with continuous and smooth local guidance, thereby improving policy learning efficiency while ensuring training stability. Let the state at time step t be... Actions The single-step reward function is expressed as shown in the formula:

[0048] in, It mainly describes the success or failure of the task level and belongs to sparse rewards, which we call left turn result rewards here. This is used to provide continuous guiding signals during training, helping the policy to converge quickly to the appropriate behavioral region in the early stages of learning; here we call it behavioral guidance reward.

[0049] Among them, the reward for turning left Constructed directly based on the termination state of the environment and collision detection results, its core purpose is to clearly distinguish between acceptable and unacceptable behaviors during the policy optimization process. Results drive rewards. Represented as: ; in, This indicates that the reward is driven by the outcome. Indicates time The collision type markings indicate that lane collisions typically reflect significant control failures or trajectory deviations. Indicates no collision; Indicates a lane collision; Indicates a car-to-car collision; Indicates time The left turn task is completed; This indicates that the left turn has been completed; This represents the reward value corresponding to a lane collision. This represents the reward value corresponding to a vehicle-to-vehicle collision; This represents the reward value for successfully completing the left turn task; , and All are constants and satisfy the following conditions: This means that the penalty for a collision between vehicles is greater than that for a collision between lanes, and the reward for a successful completion is positive and has the largest absolute value.

[0050] Behavioral guidance and reward By introducing continuous reward signals, the agent's specific actions at each step are meticulously guided, thereby significantly improving training efficiency. Behavior-guided rewards. Represented as:

[0051] in, This indicates a reward based on changes in heading angle, used to guide the vehicle to travel in the expected turning direction; This indicates location-related rewards used to guide vehicles toward the center of the target lane. This represents the reference trajectory-guided reward item, used to constrain trajectory smoothness; Indicates at time state; Indicates at time action.

[0052] In the challenging scenario of the simulation environment, the augmented state representation is used as the input to the policy network and value network, and the hierarchical reward function is used as the optimization objective. The proximal policy optimization (PPO) algorithm is employed to train the policy network. The PPO algorithm limits the policy update magnitude by truncating the objective function, avoiding training instability caused by excessively large single-step updates. Specifically, the policy network receives the augmented state representation. As input, the output is an action probability distribution, from which actions are sampled. The value network receives the same augmented state representation as input and outputs a value estimate of the current state. The advantage function is used to calculate the advantage function and guide the direction of policy gradient updates. During training, the agent continuously explores the simulation environment, receives reward feedback based on the hierarchical reward function, and iteratively updates the parameters of the policy network and value network through the PPO algorithm, so that the policy network gradually converges to the optimal policy that can efficiently complete the left turn task while ensuring safety.

[0053] After training, the policy network outputs a left-turn control command containing the desired acceleration and desired front wheel steering angle based on the augmented state representation at the current moment. Specifically, the augmented state representation at the current moment is input into the trained policy network, which outputs an action probability distribution. The desired acceleration is then sampled or averaged from this distribution. and expected front wheel steering angle The left-turn control command. Wherein, the desired acceleration... The range of values ​​is The desired front wheel steering angle The range of values ​​is ; The preset maximum absolute value of acceleration; The maximum absolute value of the front wheel steering angle is preset. This left-turn control command is then sent to the vehicle's underlying actuators (including the throttle, braking, and steering systems) to achieve autonomous left-turn decision-making and planning. Through the synergistic effect of the aforementioned hierarchical reward guidance and PPO optimization, the control command output by the policy network not only has clear safety constraints but also possesses high traffic efficiency and smoothness, enabling the generation of high-quality human-like left-turn decision trajectories.

[0054] This invention constructs a reinforcement learning decision framework incorporating attention state encoding. Based on the OpenAI Gym interface standard, this framework realizes end-to-end left-turn decision planning from environmental perception to control command output on the basis of vehicle dynamics model. Figure 2 This invention illustrates a reinforcement learning training framework incorporating attention mechanism state encoding. The following detailed description of each component within this framework is provided in conjunction with the invention. Vehicle dynamics model and environment interaction The core of the simulation environment is the vehicle dynamics model, which uses a kinematic bicycle model to describe the vehicle's planar motion.

[0055] Vehicle status includes vehicle location. Bicycle speed and vehicle heading angle Environmental vehicle status information Including the Location of environmental vehicles ,speed and heading angle Location of the actual vehicle status speed and heading angle This is used for trajectory generation and behavior reference of background vehicles in the simulation environment, making the simulated traffic flow distribution closer to the real data distribution. The environment provides raw kinematic observation information of the vehicle and surrounding vehicles through the above vehicle dynamics model as input to the state space. Figure 4 Solid arrows represent the relative positional relationship between the vehicle and the vehicles in the environment, while dashed arrows represent the guiding effect of the real vehicle's state on the trajectory of the background vehicles in the simulation environment.

[0056] Attention State Encoding Network The attention-based state encoding network takes the vehicle's state and the states of surrounding vehicles as input and outputs an encoded enhanced state representation. The network first encodes the state information of each surrounding vehicle into an environmental vehicle embedding vector, and generates a query vector from the vehicle's state information through a linear projection transformation. The query vector is then concatenated with each environmental vehicle embedding vector to form a joint feature vector. The original importance score for each environmental vehicle is calculated using a multilayer perceptron scoring function and normalized to obtain attention weights. These attention weights are then used to weight and aggregate the environmental vehicle embedding vectors to generate a context vector. Finally, the context vector is concatenated with the vehicle's state information to form the enhanced state representation. This encoding network achieves adaptive weighted aggregation of environmental vehicles through an attention mechanism, ensuring that information from key conflicting vehicles dominates the state representation, thus solving the problem of weakened key interaction information in traditional average pooling or max pooling methods. This enhanced state representation also serves as input to both the actor network and the critic network.

[0057] Actor-Critic Network Structure This invention employs a proximal policy optimization algorithm as the core optimization algorithm of the reinforcement learning framework. The actor-critic network comprises an actor network, a legacy actor network, and a value network. The legacy actor network is used to calculate the importance sampling ratio to constrain the policy update magnitude. The actor network takes the augmented state representation as input and outputs the action probability distribution, specifically including a front wheel angle decoder and an acceleration decoder, which output the expected front wheel angle and expected acceleration, respectively. The value network takes the same augmented state representation as input and outputs the value estimate of the current state. The actor network and the value network are co-optimized during training. The state value estimate output by the value network is used to calculate the advantage function, guiding the update direction of the actor network.

[0058] Motion decoding and control command generation The decoding network decouples the action probability distribution output by the policy network into a front wheel angle decoding head and an acceleration decoding head, which respectively generate the desired front wheel angle. and expected front wheel steering angle Expected acceleration The range of values ​​is The desired front wheel steering angle The range of values ​​is This decoding network decomposes the continuous action space output by the policy network into independent steering control and speed control channels, making it easier for the policy network to learn independent features in different control dimensions.

[0059] (v) Feedback Mechanism of Hierarchical Reward Function The reward module receives the action at the current moment and outputs the result of a scalar reward function. The reward function employs a hierarchical design, comprising sparse result-driven rewards and continuous behavior-guided rewards. Result-driven rewards are assigned values ​​based on the collision detection results from the simulation environment and whether the left-turn task was successfully completed, providing a clear task completion signal. Behavior-guided rewards include heading angle change rewards, position-related rewards, and reference trajectory guidance rewards, providing continuous behavior guidance signals during training. This reward signal serves as the optimization objective and is fed back to the policy network and value network via the PPO algorithm, driving network parameter updates.

[0060] (vi) Overall Workflow of the Framework In summary, the overall workflow of this invention's framework is as follows: the environment module provides the vehicle's and the surrounding vehicle's states; the attention state encoding network enhances the encoding of the states, generating enhanced state representations; the actor-critic network takes the enhanced state representations as input and outputs action probability distributions; the decoding network decouples the action probability distributions into specific acceleration and front wheel steering angle control commands; the control commands are applied to the vehicle dynamics model, updating the environmental states; and the reward module calculates reward values ​​based on the action execution results and updates network parameters using the PPO algorithm. This framework achieves end-to-end optimization across the entire link from state perception to control command generation. Each component works collaboratively through standardized data interfaces, exhibiting good scalability and engineering deployment value.

[0061] This invention introduces an attention mechanism to optimize the reinforcement learning state encoding process. It effectively solves the problems of difficult key vehicle identification, conservative or aggressive decision-making, and the difficulty in balancing safety and efficiency in left-turn scenarios at unsignalized intersections. By combining a spatial attention-based interactive perception module with an action-level risk prediction module, it ensures precise focusing of interactive features and pre-emptive suppression of dangerous actions, while guaranteeing a synergistic balance between safety constraints and efficiency optimization. In the left-turn decision-making process, this method effectively handles multi-vehicle game theory and gap judgment through dynamic interactive perception, greatly improving the safety, efficiency, and human-like characteristics of autonomous driving decision-making.

[0062] Example 2 Embodiment 2 of this invention also proposes a left-turn decision-making and planning system for autonomous vehicles based on reinforcement learning. Figure 4 This is a schematic diagram of the left-turn decision-making and planning system for autonomous vehicles based on reinforcement learning, as proposed in Embodiment 2 of the present invention. The system includes: The environment interaction module is used to construct paired interaction features between the self-vehicle and each environmental vehicle based on the acquired self-vehicle state information and environmental vehicle state information; to construct a reinforcement learning simulation environment for left turns at unsignalized intersections based on the acquired real vehicle historical trajectory information; to define the state space and action space in the simulation environment; and to identify difficult scenarios with difficulty scores greater than a preset threshold from the simulation environment. The risk prediction module is used to train an action-level risk prediction network using supervised learning, taking state-action pairs in the state space as input and the future collision probability under the state-action pair conditions as the prediction target, in order to achieve real-time suppression of high-risk actions. The encoding and concatenation module is used to encode environmental vehicle state information into environmental vehicle embedding vectors, and generate query vectors from the vehicle state information through linear projection transformation; the environmental vehicle embedding vectors and query vectors are concatenated to form multiple joint feature vectors; based on each joint feature vector, a multilayer perceptron is used for scoring and mapping, and the scoring results are used to perform weighted aggregation on the environmental vehicle embedding vectors to generate context vectors; the context vectors are concatenated with the vehicle state information to form an enhanced state representation; The decision output module is used to construct a hierarchical reward function to guide the policy network in a left-turn scenario at an unsignalized intersection, balancing safety and traffic efficiency. In this challenging scenario, the augmented state representation is used as input to both the policy network and the value network, and the hierarchical reward function is used as the optimization objective. The policy network is trained using a proximal policy optimization algorithm. After training, the policy network outputs a left-turn control command containing the desired acceleration and desired front wheel angle based on the augmented state representation at the current moment. The modules in Embodiment 2 of the present invention modularize the working process of the left-turn decision planning method for autonomous vehicles based on reinforcement learning disclosed in Embodiment 1 of the present invention, and can achieve the same technical effect as Embodiment 1 of the present invention.

[0063] The description of the relevant parts of the reinforcement learning-based autonomous vehicle left-turn decision-making and planning system provided in Embodiment 2 of this application can be found in the detailed description of the corresponding parts of the reinforcement learning-based autonomous vehicle left-turn decision-making and planning method provided in Embodiment 1 of this application, and will not be repeated here.

[0064] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0065] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A left turn decision planning method for a reinforcement learning based autonomous vehicle, characterized in that, Includes the following steps: Based on the acquired vehicle state information and environmental vehicle state information, a pairwise interaction feature between the vehicle and each environmental vehicle is constructed; based on the acquired real vehicle historical trajectory information, a left-turn reinforcement learning simulation environment for unsignaled intersections is constructed, and a state space and action space are defined in the simulation environment, as well as difficult scenarios with difficulty scores greater than a preset threshold are identified from the simulation environment. A supervised learning approach is adopted, using state-action pairs in the state space as input and the future collision probability under the state-action pair condition as the prediction target to train an action-level risk prediction network for real-time suppression of high-risk actions. The environmental vehicle state information is encoded into an environmental vehicle embedding vector, and the vehicle state information is transformed into a query vector through linear projection. The environmental vehicle embedding vector and the query vector are concatenated to form multiple joint feature vectors. Based on each joint feature vector, a multilayer perceptron is used for scoring and mapping. The scoring results are used to weight and aggregate the environmental vehicle embedding vector to generate a context vector. The context vector is concatenated with the vehicle state information to form an enhanced state representation. A hierarchical reward function is constructed to guide the policy network to balance safety and traffic efficiency in the scenario of left turns at unsignalized intersections. In the difficult scenario, the enhanced state representation is used as the input to the policy network and the value network, the hierarchical reward function is used as the optimization objective, and the policy network is trained using the proximal policy optimization algorithm. After training, the policy network outputs a left turn control command containing the desired acceleration and desired front wheel steering angle based on the enhanced state representation at the current moment.

2. The method of claim 1, wherein, Based on the acquired vehicle state information and environmental vehicle state information, pairwise interaction features between the vehicle and each environmental vehicle are constructed, specifically: The self-vehicle state information includes a self-vehicle position , a self-vehicle speed , and a self-vehicle heading angle ; The environmental vehicle state information includes a first vehicle's position , speed and heading angle ​ The pair of interaction features include a relative distance between the ego vehicle and each of the environment vehicles , a relative speed , and a relative heading angle ​ ； ； ； in, , This indicates the total number of vehicles in the environment at the current moment.

3. The method according to claim 1, characterized in that, The difficulty score is calculated based on the number of vehicles in the current scene, the absolute value of the average relative speed between the vehicle and all vehicles in the scene, and the length of the time gap during which the vehicle can safely pass.

4. The method according to claim 1, characterized in that, A supervised learning approach is adopted, using state-action pairs in the state space as input and the future collision probability under the given state-action pair conditions as the prediction target, to train an action-level risk prediction network for real-time suppression of high-risk actions; specifically: Multiple candidate states are collected from the simulation environment, and each candidate state is input into the policy network to obtain the corresponding candidate action output by the policy network under the candidate state; each candidate state is combined with the corresponding candidate action to construct multiple candidate state-action pairs. Based on the real vehicle historical trajectory information, each candidate state-action pair is labeled with a corresponding future collision probability label; using the candidate state-action pair as input features and the future collision probability label as the prediction target, a supervised learning method is used to train the action-level risk prediction network. After training, the current state and the current action output by the policy network are used to form a current state-action pair, which is then input into the action-level risk prediction network. When the output collision probability exceeds a preset safety threshold, the current action is suppressed.

5. The method according to claim 1, characterized in that, Environmental vehicle state information is encoded into an environmental vehicle embedding vector, and the vehicle state information is transformed using a linear projection to generate a query vector. The environmental vehicle embedding vector and the query vector are concatenated to form multiple joint feature vectors. Specifically: The state information of each environmental vehicle is encoded into an environmental vehicle embedding vector. Vehicle status information Generate query vector through linear projection transformation ; the query vector With each environment vehicle embedding vector The features are concatenated to form a joint feature vector. ; ； ； in, Represents a linear projection matrix; This indicates the bias term.

6. The method according to claim 5, characterized in that, Based on the joint feature vectors, a multilayer perceptron is used for scoring and mapping. The scoring results are then used to weighted aggregate the environmental vehicle embedding vectors to generate a context vector. Specifically: The joint feature vector Input a scoring function consisting of two layers of sensing mechanisms, and calculate the first... Original importance score of each environmental vehicle ; ； The original importance scores of all valid environmental vehicles are normalized to obtain the [number]th [value]. Attention weight of vehicles in the environment Wherein, the effective environmental vehicle is an environmental vehicle that actually exists at the current moment and is not masked; ； The attention weights of each environmental vehicle are used to weight the corresponding environmental vehicle embedding vectors, and all weighted environmental vehicle embedding vectors are summed to generate a context vector. ; ； in, and These represent the linear mapping parameter matrices of the first and second layers of the two-layer perceptron, respectively. and These represent the corresponding bias terms; This represents the set of vehicles in the current environment. Indicates the first The original importance scores of each environmental vehicle; Indicates the first Attention weights for each vehicle in the environment.

7. The method according to claim 1, characterized in that, The hierarchical reward function includes sparse result-driven rewards and continuous behavior-guided rewards; The result-driven reward is assigned based on the collision detection results fed back by the simulation environment and whether the left turn task was successfully completed; The behavioral guidance rewards include a heading angle change reward calculated based on the heading angle change, a position-related reward calculated based on the distance between the vehicle's current position and the centerline of the target lane, and a reference trajectory guidance reward calculated based on the deviation between the vehicle's actual trajectory and a preset reference trajectory.

8. The method according to claim 7, characterized in that, The result-driven reward Represented as: ； in, This indicates that the reward is driven by the outcome. Indicates time Collision type marker; Indicates no collision; Indicates a lane collision; Indicates a car-to-car collision; Indicates time The left turn task is completed; This indicates that the left turn has been completed; This represents the reward value corresponding to a lane collision. This represents the reward value corresponding to a vehicle-to-vehicle collision; This represents the reward value for successfully completing the left turn task; , and All are constants and satisfy the following conditions: .

9. The method according to claim 7, characterized in that, The behavior guidance reward Represented as: in, This indicates a reward based on changes in heading angle; This indicates location-related rewards; This indicates the reference trajectory guiding the reward items; Indicates at time state; Indicates at time action.

10. A left-turn decision-making and planning system for autonomous vehicles based on reinforcement learning, characterized in that, include: The environment interaction module is used to construct paired interaction features between the self-vehicle and each environmental vehicle based on the acquired self-vehicle state information and environmental vehicle state information; to construct a reinforcement learning simulation environment for left turns at unsignalized intersections based on the acquired real vehicle historical trajectory information; to define the state space and action space in the simulation environment; and to identify difficult scenarios with difficulty scores greater than a preset threshold from the simulation environment. The risk prediction module is used to train an action-level risk prediction network using supervised learning, taking state-action pairs in the state space as input and the future collision probability under the state-action pair conditions as the prediction target, in order to achieve real-time suppression of high-risk actions. The encoding and concatenation module is used to encode environmental vehicle state information into environmental vehicle embedding vectors, and generate query vectors from the vehicle state information through linear projection transformation; the environmental vehicle embedding vectors and query vectors are concatenated to form multiple joint feature vectors; based on each joint feature vector, a multilayer perceptron is used for scoring and mapping, and the scoring results are used to perform weighted aggregation on the environmental vehicle embedding vectors to generate context vectors; the context vectors are concatenated with the vehicle state information to form an enhanced state representation; The decision output module is used to construct a hierarchical reward function to guide the policy network to balance safety and traffic efficiency in the scenario of left turns at unsignalized intersections. In the difficult scenario, the enhanced state representation is used as the input to the policy network and the value network, the hierarchical reward function is used as the optimization objective, and the policy network is trained using a proximal policy optimization algorithm. After training, the policy network outputs a left turn control command containing the desired acceleration and desired front wheel steering angle based on the enhanced state representation at the current moment.

Images