Non-signalized intersection traffic decision-making method based on large model enabling reinforcement learning decision-making framework

Abstract

The invention relates to a non-signalized intersection traffic decision-making method based on a large model enabling reinforcement learning decision-making framework, and belongs to the technical field of automatic driving. The method comprises the following steps: firstly, acquiring an environment map and a vehicle state, providing coarse-grained reference actions based on a navigation reference model of classical path planning and trajectory tracking, and then generating a reward function through a large model by adopting a chain reasoning structure based on designer-suggestor-optimizer; based on the reference action and the reward function, the vehicle decision-making agent learns a driving strategy based on a reinforcement learning framework; and finally, establishing a rule-based interaction field, and finely adjusting the driving strategy output by the reinforcement learning framework to obtain a final control strategy. The reward design of the invention gets rid of excessive dependence on expert knowledge, and the design of the reference action can significantly reduce the exploration dimension and accelerate strategy learning.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology and relates to a method for decision-making at unsignalized intersections based on a large model-enabled reinforcement learning decision-making framework. Background Technology

[0002] Achieving safe and intelligent decision-making in complex scenarios (such as unsignalized intersections) is a major challenge in autonomous driving research. The dynamic interactions of multiple traffic participants in these scenarios significantly increase the complexity of decision-making. Reinforcement learning (RL)-based methods have received widespread attention in recent years because RL can autonomously learn strategies through the interaction between agents and the environment, thereby acquiring adaptability and environmental perception capabilities, and reducing reliance on the manual design of a large number of rules.

[0003] However, RL-based methods still face several key limitations: 1) low learning efficiency and long training time; 2) the design of reward functions is highly dependent on human experience, the process is complex and prone to sparse reward problems; 3) the training environment is usually limited to simulation environment and is difficult to transfer directly to real scene.

[0004] Traditional decision-making methods play a crucial role in the field of autonomous driving, primarily including logical rule-based methods, optimization methods, and game theory methods. Logical rule-based methods rely on expert system structures, using knowledge bases and inference engines combined with Time-to-Collision (TTC) metrics for safety assessment. Decision trees were adopted early on due to their structural interpretability, but they are prone to overfitting and cannot handle variable interactions. Static traffic rule systems are effective in some scenarios, but they depend on perception accuracy and struggle to cope with complex situations. Optimization methods (such as robust convex optimization and Markov decision processes) can handle dynamic uncertainty. Adaptive MPC is also used in integrated decision control. Due to its high computational cost, researchers have proposed more flexible models that consider interactive effects, but these suffer from drawbacks such as strong subjectivity.

[0005] Current rule-based methods, due to their heavy reliance on developers' prior knowledge, struggle to adapt to dynamic and complex driving scenarios. Existing research has shown that deep reinforcement learning (DRL) can autonomously learn optimal policies through interaction with the environment, demonstrating superior capabilities across various driving tasks. In many applications of autonomous driving, reinforcement learning is not directly used for end-to-end decision training, but rather for scene representation learning and optimization.

[0006] To enhance decision-making security, researchers have proposed integrating risk assessment into DRL (Reinforcement Learning Link) to enable agents to learn policies that are both safe and risk-sensitive, while leveraging lightweight networks to improve computational efficiency. Other works combine finite state machines (FSMs) with reinforcement learning agents to enhance policy stability and have proposed an efficient event-triggered communication model, ETDQN.

[0007] Observational adversarial reinforcement learning (ARL) has also been used to mitigate the effects of policy constraints and perceptual uncertainty, improving policy robustness, but its training process is complex and parameter setting is difficult. Subsequently, hierarchical reinforcement learning (HRL) and multi-agent reinforcement learning (MARL) have emerged. Researchers have proposed a safe-enhanced HRL with learnable modules to reduce reliance on prior models. Multi-agent reinforcement learning (MARL) can model multi-vehicle behavior as a partially observable stochastic game, enabling altruistic policies in mixed traffic.

[0008] Furthermore, combining MPC with Safety Reinforcement Learning (SRL) has been shown to improve the robustness and safety of control in highway scenarios, effectively addressing system uncertainties and dynamic constraints. Some mainstream methods employ distributed reinforcement learning to improve value estimation accuracy and utilize hierarchical optimization structures to decouple path and velocity planning, thereby achieving more reliable decision-making performance under heterogeneous information fusion and collision risk constraints.

[0009] Despite the significant progress made by the above methods, reinforcement learning-based schemes still generally face two core challenges: 1) the reward function design is complex and highly dependent on human experience; 2) the training iteration time is too long and the learning efficiency is low.

[0010] Large language models (LLMs), trained on massive amounts of network data, have shown the potential to simulate human reasoning processes and have gradually gained attention in autonomous driving decision-making research. Techniques such as prompt engineering, contextual learning, few-shot learning, and reinforcement learning based on human feedback (RLHF) are often used to guide large models to adapt to specific driving tasks.

[0011] A range of methods based on cue word engineering, such as LanguageMPC, obtain driving strategies and explanations from natural language input. However, these methods heavily rely on accurate perception data and have high computational costs. Contextual learning and few-shot learning leverage the knowledge transfer and reasoning capabilities of LLMs to drive large models to learn driving operations through examples. Examples include GPT-Driver and DILU, both of which demonstrate good driving agent capabilities. However, their performance is limited by specific training data and cue structures, and they are prone to hallucinations or erroneous decisions.

[0012] The RLHF method leverages human experience in training to improve performance by designing "coach-like" agents to reinforce vehicle strategies using expert knowledge. However, such methods are costly and rely heavily on expert involvement, making them difficult to scale.

[0013] In summary, traditional rule-based methods are easily limited in complex scenarios due to their strong subjectivity and insufficient generalization; reinforcement learning methods have adaptive capabilities but are constrained by the bottleneck of reward design; large models have strong reasoning capabilities but are easily affected by inaccurate perception and illusions. Summary of the Invention

[0014] In view of this, the purpose of this invention is to provide a method for decision-making at signalless intersections based on a large model-enabled reinforcement learning decision framework.

[0015] To achieve the above objectives, the present invention provides the following technical solution: A method for decision-making at signalless intersections based on a large model-enabled reinforcement learning decision framework, comprising the following steps: S1. Obtain the environmental map and vehicle status, and provide coarse-grained reference actions based on the navigation reference model of classic path planning and trajectory tracking. The navigation reference model includes at least three stages: global path planning, local control analysis, and reinforcement learning input construction. S2. A chain-like reasoning structure based on designer-advisor-optimizer is adopted. A reward function is generated through a large model. The designer generates initial reward function candidates based on task prompts. The advisor comprehensively evaluates the generated reward function and proposes improvement suggestions. The optimizer selects the final optimal reward function based on multiple rounds of iteration. S3. Based on reference actions and reward functions, the vehicle decision-making agent learns driving strategies using a reinforcement learning framework. S4. Establish a rule-based interaction field to fine-tune the driving strategy output by the reinforcement learning framework and obtain the final control strategy.

[0016] Furthermore, the global path planning phase in step S1 includes the following process: First, the original road map is transformed into a topological representation, including the environment map. Each road is represented by its entrance waypoint and exit waypoint. Therefore, the topological set of all road information is represented as:

[0017] In the formula, Indicates the coordinates of the entrance waypoint. Indicates the coordinates of the exit waypoint. Number of roads; Subsequently, a directed graph is constructed based on the topology set:

[0018] Among them, the node set Represents the set of all unique waypoints and the set of edges. This represents the road connectivity, where the cost of each edge is given by the road length or the number of discrete sampling points; where, each It is unique in the set; It can be represented as A subset of, for The first in a road, if and Then there exists an edge. ,Right now ; Starting from the vehicle's initial state The initial condition and the target state are: As a termination condition, the A* algorithm is used in the graph A heuristic search is performed to obtain an optimal sequence of nodes:

[0019] in, These represent the vehicle's state parameters under initial conditions, namely the horizontal coordinate, vertical coordinate, and heading angle; These represent the vehicle's state parameters under termination conditions, namely the horizontal coordinate, vertical coordinate, and heading angle. Indicates the first One node; Expanding the node sequence yields a continuous sequence of global reference waypoints:

[0020] in, Indicates position coordinates, Indicates the corresponding heading angle. This indicates the number of global waypoints.

[0021] Furthermore, in the local control parsing phase of step S1, the nearest waypoint in the global waypoint sequence to the current position is selected. Each waypoint serves as a local reference trajectory:

[0022] Subsequently, a PID controller is introduced to analyze the reference trajectory, obtaining the vehicle's reference steering angle signal. The control law is defined as follows:

[0023] in, The reference turning angle output by the navigation model. This represents the lateral error between the vehicle's current position and the local reference trajectory. This is a heuristic estimate of the cost from the current node to the target node during the A* algorithm search process. , , These are proportional, integral, and differential gains, respectively. It is the feedforward gain, used to inject global planning information into local control.

[0024] Furthermore, in the reinforcement learning input construction stage of step S1, the reference turning angle output by the navigation model... Target state vector and environmental map information Inputs that are combined into a reinforcement learning strategy:

[0025] In the subsequent training process of the reinforcement learning framework, the reinforcement learning strategy directly optimizes decisions within a limited range under the constraints of the navigation reference actions.

[0026] Furthermore, in step S2, a system prompt word is used. Contextual clues Output structure constraints and large models As input, an executable reward function is obtained, in which the system prompt word is used. Define the overall goals, constraints, and safety principles of the reward design; contextual clues Includes detailed task description information; output structure constraints. Used to specify the interface form and executability requirements of the reward function; large model As a unified generation and inference engine, the resulting reward function is expressed as:

[0027] in, and These represent the environment state space and the action space, respectively.

[0028] Furthermore, the designer's reward function candidate generation process is as follows: During the initialization phase, the system integrates character settings and task context into a unified prompt:

[0029] And initialize the history context:

[0030] Designers based on system prompts User prompts and the current context Generate a set of candidate reward functions:

[0031] Each candidate reward function includes multiple sub-items, namely:

[0032] in, It is a target distance reward. It is a smoothness of movement or a penalty item. It is a safety constraint. These are supplementary items related to the task. .

[0033] Furthermore, the process of evaluating and improving the suggestion's reward function is as follows: For each candidate reward function A systematic evaluation will be conducted, and structured recommendations will be generated.

[0034] The evaluation dimensions for the recommender include logical consistency, task alignment, numerical stability, and executability. Logical consistency refers to whether there are any conflicts between reward items; task alignment refers to whether they accurately reflect the task objectives; numerical stability refers to whether the reward scale is reasonable; and executability refers to whether the interface and input constraints are met. Generated improvement suggestions Added to the context:

[0035] Subsequently, the designers generated a revised reward function based on the suggestions:

[0036] The process of generating reward function candidates by the advisors and designers involves multiple iterations, resulting in a gradually converging set of reward candidates.

[0037] Furthermore, before determining the final reward function, the optimizer performs consistency checks and candidate selection, specifically including output consistency constraints and executable constraints. For output consistency constraints, a set of predefined test inputs... Next, assess the volatility of reward output:

[0038] like If the reward function is unstable, it will be discarded. For executability constraints, check whether the reward function is satisfied. If the condition is not met, it will be eliminated; Only retain the set of candidate functions that simultaneously satisfy both consistency and executability:

[0039] After incorporating all candidate reward functions and their evaluation information into the context, the optimizer outputs the final reward function by integrating historical information:

[0040] This process selects the most consistent, robust, and task-goal-aligned reward function from the candidate space as the final reward design for the reinforcement learning training phase.

[0041] Furthermore, in step S3, a deep deterministic policy gradient is used as the base model to transform the decision-making process of the autonomous driving agent into a Markov decision process:

[0042] Among them, at time The intelligent agent receives the environmental state. Based on strategy Execute actions Then the environment returns a reward. and transferred to The optimization objective is to maximize the expected cumulative discount return:

[0043] The agent's state includes the position, size, and current control information of the vehicle and surrounding vehicles:

[0044] in, Including bicycles and The position of the surrounding vehicles , , The vehicle's 3D bounding box; The value function update objective is as follows:

[0045] Policy gradient:

[0046] The intelligent agent ultimately outputs vehicle control information, including throttle and steering wheel angle.

[0047] Furthermore, in step S4, firstly, the interaction field is defined, constructing a time-varying elliptical safety field centered on the vehicle:

[0048] in, For the geometric center of the vehicle, and For the major and minor semi-axles, The rotation angle is... and For angular velocity and phase; The elliptical field shape changes as the vehicle enters the intersection, encompassing two forms: a vertically elongated TTS and a flattened, wide TWS. Before entering the intersection, it is a vertically elongated TTS; during its passage through the intersection, it becomes a flattened, wide TWS; and after leaving the intersection, it reverts to a vertically elongated TTS. This shape change is adjusted based on variations in the rotation angle and the major and minor semi-axles. Rotation angle , It is the vehicle's heading angle; Long half shaft ,in It is the geometric length of the vehicle. It is the minimum longitudinal static safety distance. It is a speed safety correlation coefficient. It's the vehicle speed; short half shaft ,in For the geometric width of the vehicle, For fixed width redundancy, It is the compensation coefficient. It is yaw compensation; Vehicles at different stages at the intersection are indicated by symbols. This indicates that TTS corresponds to TWS corresponds ; The process of elliptical field morphological change is as follows:

[0049]

[0050] Instantaneous trajectory prediction is introduced to determine whether a collision occurs. The next prediction step is performed based on the current vehicle state, and the prediction point is represented as follows:

[0051] The coordinates of the predicted points are obtained by predicting the self-vehicle and other vehicles. If the line connecting the self-vehicle's predicted points to the predicted points of the self-vehicle and the line connecting other vehicles' predicted points to the predicted points of other vehicles intersect, it indicates that there is a conflict area, and then the next step of traffic strategy adjustment is triggered. If a conflict occurs, the traffic strategy is adjusted according to the current elliptical field shape. Specifically, if the self-vehicle has not yet entered the intersection, a deceleration strategy is adopted; if the self-vehicle has already entered the intersection, an acceleration strategy is adopted.

[0052] The beneficial effects of this invention are as follows: This invention proposes a chain-based reasoning method that explicitly decomposes the reward design process into three stages: generation, evaluation, and decision-making, by simulating the hierarchical and iterative reasoning process of human experts. The DAO structure includes three roles: Designer, Advisor, and Optimizer, along with a consistency check mechanism. The Designer generates initial reward function candidates based on task prompts; the Advisor comprehensively evaluates the generated reward functions and proposes improvements; and the Optimizer selects the final optimal reward function based on multiple iterations. The consistency check mechanism ensures the stability and reliability of the generated reward functions through preset standards, effectively reducing over-reliance on expert knowledge and improving the automation and generalization capabilities of reward design. This innovative method not only simplifies the complexity of reward design but also significantly improves the training efficiency and policy quality of reinforcement learning agents.

[0053] This invention constructs a navigation-based guidance mechanism. This mechanism provides coarse-grained action references to the reinforcement learning agent through a modern navigation system, significantly compressing the explorable action space of the policy. Specifically, the navigation model takes the environment map and target state as input, uses the A* algorithm for global path planning, and analyzes local control variables through a PID controller to output a reference turning angle signal. This reference signal, together with the environment state, constitutes the input to the reinforcement learning policy, enabling the agent to perform rapid and centralized learning within the constrained action range. This navigation guidance mechanism significantly reduces the exploration dimension, accelerates the policy learning process, and improves the agent's adaptability and decision-making efficiency in complex scenarios.

[0054] This invention also proposes a rule-based interaction field model. This model constructs a time-varying elliptical safety field centered on the vehicle to determine whether a vehicle is in a critical interaction area and incorporates the interaction impact into the policy output. Specifically, the interaction field morphology changes with the vehicle's different stages at the intersection, including a vertical elongated shape before entering, a flattened and wide shape during crossing, and a renewed vertical elongated shape after leaving. By introducing instantaneous trajectory prediction, the model can predict path conflicts and trigger corresponding protective actions, such as braking, deceleration, or acceleration. These rule-based adjustments, as an important component of the policy, effectively improve the safety and stability of the agent in complex interaction scenarios. This interaction field model not only enhances the environmental perception ability of reinforcement learning agents but also significantly improves their reliability and robustness in practical applications.

[0055] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall architecture of the signalless intersection passage decision-making method based on a large model-enabled reinforcement learning decision-making framework according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the reward-designed chain reasoning structure DAO according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an example of a reward-designed chain-based reasoning structure (DAO) according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the unsignalized intersection traffic decision system based on a large model-enabled reinforcement learning decision framework, according to an embodiment of the present invention. Detailed Implementation

[0057] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0058] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0059] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0060] Please see Figures 1-4 This is a method for decision-making at unsignalized intersections based on a large model-enabled reinforcement learning decision framework.

[0061] Example 1 This embodiment first describes in detail a method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision-making framework, such as... Figure 1 As shown, it includes the following steps: S1. Obtain environmental map and vehicle status, and provide coarse-grained reference actions based on the navigation reference model of classic path planning and trajectory tracking; S2. Employs a chain-based reasoning structure based on designer-advisor-optimizer to generate a reward function through a large model; S3. Based on reference actions and reward functions, the vehicle decision-making agent learns driving strategies using a reinforcement learning framework. S4. Establish a rule-based interaction field to fine-tune the driving strategy output by the reinforcement learning framework and obtain the final control strategy.

[0062] In step S1 of this embodiment, in order to provide coarse-grained but reasonable action priors to the reinforcement learning policy learning process, this embodiment constructs a navigation reference model based on classical path planning and trajectory tracking. This model takes the environment map and target state as input, outputs an interpretable reference turning angle signal (coarse value), and together with the environment state, constitutes the input to the reinforcement learning policy, thereby significantly compressing the policy's explorable action space.

[0063] The navigation model can be divided into three stages: global path planning, local control analysis, and reinforcement learning input construction.

[0064] (1) Phase 1: Global Path Planning In the global planning phase, the original road map is first transformed into a topological representation. Specifically, this involves creating an environmental map. Each road in the network is represented by its entry waypoint and exit waypoint, and all road information is organized into a topological set: ,in, Indicates the coordinates of the entrance waypoint. Indicates the coordinates of the exit waypoint. Number of roads; Subsequently, a directed graph is constructed based on the above topological relationships:

[0065] Among them, the node set Represents the set of all unique waypoints and edges. This represents the road connectivity. The cost of each edge is given by the road length or the number of discrete sampling points, where... each It is unique in the set; It can be represented as A subset of, for The first in a road, if and Then there exists an edge. ,Right now .

[0066] Based on this, the vehicle's initial state and target state As start and end conditions. The A* algorithm is used in the graph. A heuristic search is performed to obtain an optimal sequence of nodes:

[0067] in, These represent the vehicle's state parameters under initial conditions, namely the horizontal coordinate, vertical coordinate, and heading angle; These represent the vehicle's state parameters under termination conditions, namely the horizontal coordinate, vertical coordinate, and heading angle. Indicates the first Each node.

[0068] Further expansion of the node sequence yields a continuous sequence of global reference waypoints:

[0069] in, Indicates position coordinates, Indicates the corresponding heading angle. This indicates the number of global waypoints.

[0070] (2) Phase Two: Local Control Parsing Considering that reinforcement learning decisions are usually based on local perception range, this embodiment only selects the nearest waypoint from the global waypoint sequence to the current position. Each waypoint serves as a local reference trajectory:

[0071] Subsequently, a PID controller is introduced to analyze the reference trajectory, obtaining the vehicle's reference steering angle signal. The control law is defined as follows:

[0072] in, The reference turning angle output by the navigation model. This represents the lateral error between the vehicle's current position and the local reference trajectory. This provides a heuristic estimate of the cost from the current node to the target node in A* search. , , These are the proportional, integral, and differential gains, respectively. It is the feedforward gain, used to inject global planning information into local control.

[0073] The control output not only relies on local geometric errors, but also incorporates global target orientation information, thereby improving navigation consistency and stability.

[0074] (3) Phase 3: Preparation for Reinforcement Learning Finally, the reference turning angle output by the navigation model Target state vector and environmental map information Inputs that are combined into a reinforcement learning strategy:

[0075] During training, the reinforcement learning strategy does not search directly in the complete action space, but optimizes decisions within a limited interval under the constraint of the navigation reference action, thereby significantly reducing the difficulty of exploration and accelerating convergence.

[0076] This structure effectively compresses the learnable action range of the policy while ensuring basic navigation capabilities, enabling direction-guided navigation. Due to the presence of reference actions, the learnable action range of RL is significantly compressed, thereby reducing the policy search space and improving exploration efficiency, thus significantly accelerating trial-and-error learning.

[0077] In step S2 of this embodiment, to improve the intelligence level of the reinforcement learning agent, this embodiment proposes an innovative reward design framework that fully utilizes the rich driving and general knowledge of the large model. This framework constructs an "optimization bridge" between the environment and the policy, fundamentally alleviating the limitations of manual reward design.

[0078] Existing research has shown that large models can be directly used to construct RL reward functions, but simple prompts are insufficient to fully leverage the reasoning capabilities of large models. In contrast, chain-of-thought (CoT) can guide large models to reason step by step and reflect continuously, thereby obtaining higher quality output.

[0079] Based on this, this embodiment proposes a reward design chain reasoning structure called DAO (Designer–Advisor–Optimizer), which reflects the hierarchy and iterative nature of human reasoning. DAO includes three roles: 1) Designer: generates initial reward function candidates based on task prompts; 2) Advisor: comprehensively evaluates the generated reward functions and proposes improvement suggestions; 3) Optimizer: selects the final optimal reward function based on multiple rounds of iteration.

[0080] To ensure the stability of the large model output and reduce the risk of illusion, this embodiment introduces a consistency check mechanism. Multiple candidate results are generated through multi-round interactions between the designer and the advisor, and the optimizer selects the most consistent and reliable function from the result pool. The consistency check includes two criteria: 1) detecting the magnitude of output changes using predefined inputs and eliminating candidates with large fluctuations; 2) checking the executability of the reward function, the process of which is as follows: Figure 2 As shown.

[0081] This framework transforms the general knowledge and generative capabilities of large models into structured, executable, and stable reinforcement learning reward functions through multi-role collaboration and multi-round inference. This differs from directly generating reward functions using single-round prompts. Figure 3 As illustrated in one reasoning example, DAO explicitly breaks down the reward design process into three stages: generation, evaluation, and decision-making. This simulates the hierarchical and iterative reasoning process of human experts in complex system design, thereby alleviating the problems of strong subjectivity and insufficient generalization ability in manual reward design.

[0082] The DAO framework takes the following information as input: 1) System prompt words 1) Define the overall goals, constraints, and safety principles of the reward design; 2) Contextual clues : Includes specific task descriptions (such as intersection passage, obstacle avoidance, interaction rules, etc.); 3) Output structure constraints 4) Large Model : As a unified generation and reasoning engine.

[0083] The final output is an executable reward function:

[0084] in and These represent the environment state space and the action space, respectively.

[0085] (1) Designer: Reward function candidate generation During the initialization phase, the system integrates character settings and task context into a unified prompt:

[0086] And initialize the history context:

[0087] The designer's responsibility is to respond to system prompts. User prompts and the current context Generate a set of candidate reward functions:

[0088] Each candidate reward function typically consists of multiple sub-items, for example:

[0089] in: It is a target distance reward. It is a smoothness of movement or a penalty item. These are safety constraints (such as distance from other vehicles). These are supplementary items related to the task. .

[0090] (2) Advisor: Evaluation and improvement suggestions for the reward function For each candidate reward function The Advisor then conducts a systematic evaluation and generates structured recommendations:

[0091] The evaluation dimensions mainly include: 1) Logical consistency, whether there are any conflicts between reward items; 2) Task alignment, whether it accurately reflects the task objectives; 3) Numerical stability, whether the reward scale is reasonable; 4) Feasibility, whether it meets the interface and input constraints.

[0092] Generated improvement suggestions Added to the context:

[0093] Subsequently, the Designer generates a revised reward function based on this suggestion:

[0094] This process can be iterated in multiple rounds to form a gradually converging set of reward candidates.

[0095] (3) Consistency check and candidate screening To mitigate the risks of instability and illusions in large model generation, DAO introduces a consistency check mechanism in the candidate pool. This includes the following two types of constraints: 1) Output consistency constraints In a set of predefined test inputs Next, assess the volatility of reward output:

[0096] like If the reward function is unstable, it will be discarded. 2) Enforceability Constraints Check if the reward function is satisfied:

[0097] No illegal operation or undefined variable Only retain the set of candidate functions that simultaneously satisfy both consistency and executability:

[0098] (4) Optimizer: Decision-maker for the final reward function After incorporating all candidate reward functions and their evaluation information into the context, the Optimizer makes a final decision based on historical information:

[0099] The decision-making process essentially involves selecting the most consistent, robust, and task-goal-aligned reward function from the candidate space as the final reward design for the reinforcement learning training phase.

[0100] In step S3 of this embodiment, the reinforcement learning agent is the core module of this decision framework, which directly learns the driving strategy based on the navigation reference and the reward function generated by the LM.

[0101] Autonomous driving agents follow Markov decision processes (MDPs):

[0102] At time t, the agent receives the environmental state. Based on strategy Execute actions Then the environment returns a reward. and transferred to .

[0103] The objective is to maximize the expected cumulative discount return:

[0104] The agent's state includes the position, size, and current control information of the vehicle and surrounding vehicles:

[0105] in, Including bicycles and The position of the surrounding vehicles , , The vehicle's 3D bounding box (coordinates of 8 points).

[0106] This embodiment uses Deep Deterministic Policy Gradient (DDPG) as the main model, while also referencing algorithms such as SAC, TD3, and DSACT, all of which have advantages in handling continuous action spaces and overestimation problems.

[0107] The value function update objective is as follows:

[0108] Policy gradient:

[0109] The intelligent agent ultimately outputs vehicle control information, including throttle and steering wheel angle.

[0110] In step S4 of this embodiment, to further enhance decision security, this embodiment proposes a rule-based interaction field model for security fine-tuning of the RL output.

[0111] First, the interaction field is defined, and an elliptical safety field centered on the vehicle and varying over time is constructed:

[0112] in, For the geometric center of the vehicle, and For the major and minor semi-axles, The rotation angle is... and For angular velocity and phase.

[0113] The elliptical field shape changes depending on the stage of the vehicle at the intersection: ① Before entering: vertical elongated shape (TTS), with a longer warning range; ② During crossing: flat and wide shape (TWS), with enhanced lateral attention; ③ After leaving: returns to TTS, focusing on forward safety.

[0114] The morphological changes are adjusted according to the rotation angle and the changes in the major and minor semi-axis. Rotation angle , It is the vehicle's heading angle; Long half shaft ,in It is the geometric length of the vehicle. It is the minimum longitudinal static safety distance. It is a speed safety correlation coefficient. It's the vehicle speed; short half shaft ,in For the geometric width of the vehicle, For fixed width redundancy, It is the compensation coefficient. It is yaw compensation; Vehicles at different stages at the intersection are indicated by symbols. This indicates that TTS corresponds to TWS corresponds ; The process of elliptical field morphological change is as follows:

[0115]

[0116] Instantaneous trajectory prediction is introduced to determine whether a collision occurs. The next prediction step is performed based on the current vehicle state, and the prediction point is represented as follows:

[0117] The coordinates of the predicted points are obtained by predicting the self-vehicle and other vehicles. If the line connecting the self-vehicle's predicted points to the self-vehicle's predicted points and the line connecting other vehicles' predicted points to other vehicles intersect, it indicates that there is a conflict area, and then the next step of traffic strategy adjustment is triggered.

[0118] After a conflict occurs, if the vehicle has not yet entered the intersection, a yielding strategy (deceleration) is adopted; if the vehicle has already entered the intersection, a rapid passage strategy (acceleration) is adopted. Both situations involve adjustments based on the existing throttle control. After fine-tuning, the final output is still the control amount (throttle + steering wheel angle), where acceleration / deceleration is scaled up or down based on the existing throttle, for example, increasing the throttle by 1.1 times or decreasing it by 1.1 times.

[0119] In summary, this invention constructs a navigation-based reference method to improve learning efficiency and simplify the training structure, providing feasible directional action references for RL agents, thereby significantly reducing the exploration space. Addressing the difficulties in designing manual rewards and the problem of sparse rewards, a general knowledge-driven chain reasoning method (DAO) for large models is proposed. This method includes a reasoning chain of "reward designer—advisor—verifier," along with a consistency check mechanism, to generate a reliable reward function. Furthermore, to enhance the security of the training policy in dynamic interactions, a rule-based interaction field is proposed for policy-level fine-tuning of the RL output.

[0120] Example 2 This embodiment provides a system for executing the signalless intersection passage decision-making method based on a large model-empowered reinforcement learning decision-making framework in Embodiment 1, such as... Figure 4 As shown, it includes a navigation reference layer, a large model reward layer, a reinforcement learning decision layer, and an interaction rule constraint layer. The navigation reference layer provides effective reference for the reinforcement learning agent, enabling it to learn quickly and centrally within the constrained action range, significantly reducing the exploration dimension and accelerating policy learning. The large model reward layer executes an innovative chain-based reasoning method, DAO, which includes a reward designer, advisor, and verifier structure with a consistency check mechanism, thereby constructing a large model-enabled reward generation module that frees reward design from excessive reliance on expert knowledge. The reinforcement learning decision layer outputs the learned travel policy based on the reference action and reward function, and fine-tunes and optimizes it through the interaction rule constraint layer. The interaction rule constraint layer determines whether the vehicle is in an important interaction area based on an elliptical interaction area and incorporates the interaction influence into the policy output.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework, characterized in that: The method includes the following steps: S1. Obtain the environmental map and vehicle status, and provide coarse-grained reference actions based on the navigation reference model of classic path planning and trajectory tracking. The navigation reference model includes at least three stages: global path planning, local control analysis, and reinforcement learning input construction. S2. A chain-like reasoning structure based on designer-advisor-optimizer is adopted. A reward function is generated through a large model. The designer generates initial reward function candidates based on task prompts. The advisor comprehensively evaluates the generated reward function and proposes improvement suggestions. The optimizer selects the final optimal reward function based on multiple rounds of iteration. S3. Based on reference actions and reward functions, the vehicle decision-making agent learns driving strategies using a reinforcement learning framework. S4. Establish a rule-based interaction field to fine-tune the driving strategy output by the reinforcement learning framework and obtain the final control strategy.

2. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework as described in claim 1, characterized in that: The global path planning phase in step S1 includes the following process: First, the original road map is transformed into a topological representation, including the environment map. Each road is represented by its entrance waypoint and exit waypoint. Therefore, the topological set of all road information is represented as: In the formula, Indicates the coordinates of the entrance waypoint. Indicates the coordinates of the exit waypoint. Number of roads; Subsequently, a directed graph is constructed based on the topology set: Among them, the node set Represents the set of all unique waypoints and the set of edges. This represents the road connectivity, where the cost of each edge is given by the road length or the number of discrete sampling points; where, each It is unique in the set; It can be represented as A subset of, for The first in a road, if and Then there exists an edge. ,Right now ; Starting from the vehicle's initial state The initial condition and the target state are: As a termination condition, the A* algorithm is used in the graph A heuristic search is performed to obtain an optimal sequence of nodes: in, These represent the vehicle's state parameters under initial conditions, namely the horizontal coordinate, vertical coordinate, and heading angle; These represent the vehicle's state parameters under termination conditions, namely the horizontal coordinate, vertical coordinate, and heading angle. Indicates the first One node; Expanding the node sequence yields a continuous sequence of global reference waypoints: in, Indicates position coordinates, Indicates the corresponding heading angle. This indicates the number of global waypoints.

3. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework according to claim 2, characterized in that: In the local control resolution phase of step S1, the nearest waypoint in the global waypoint sequence to the current position is selected. Each waypoint serves as a local reference trajectory: Subsequently, a PID controller is introduced to analyze the reference trajectory, obtaining the vehicle's reference steering angle signal. The control law is defined as follows: in, The reference turning angle output by the navigation model. This represents the lateral error between the vehicle's current position and the local reference trajectory. This is a heuristic estimate of the cost from the current node to the target node during the A* algorithm search process. , , These are proportional, integral, and differential gains, respectively. It is the feedforward gain, used to inject global planning information into local control.

4. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework as described in claim 3, characterized in that: In the reinforcement learning input construction phase of step S1, the reference turning angle output by the navigation model Target state vector and environmental map information Inputs that are combined into a reinforcement learning strategy: In the subsequent training process of the reinforcement learning framework, the reinforcement learning strategy directly optimizes decisions within a limited range under the constraints of the navigation reference actions.

5. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework according to claim 4, characterized in that: In step S2, the system prompt word is used. Contextual clues Output structure constraints and large models As input, an executable reward function is obtained, in which the system prompt word is used. Define the overall goals, constraints, and safety principles of the reward design; contextual clues Includes detailed task description information; output structure constraints. Used to specify the interface form and executability requirements of the reward function; large model As a unified generation and inference engine, the resulting reward function is expressed as: in, and These represent the environment state space and the action space, respectively.

6. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework according to claim 5, characterized in that: The designer's reward function candidate generation process is as follows: During the initialization phase, the system integrates character settings and task context into a unified prompt: And initialize the history context: Designers based on system prompts User prompts and the current context Generate a set of candidate reward functions: Each candidate reward function includes multiple sub-items, namely: in, It is a target distance reward. It is a smoothness of movement or a penalty item. It is a safety constraint. These are supplementary items related to the task. .

7. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework according to claim 6, characterized in that: The process of evaluating and improving the suggestion's reward function is as follows: For each candidate reward function A systematic evaluation will be conducted, and structured recommendations will be generated. The evaluation dimensions for the recommender include logical consistency, task alignment, numerical stability, and executability. Logical consistency refers to whether there are any conflicts between reward items; task alignment refers to whether they accurately reflect the task objectives; numerical stability refers to whether the reward scale is reasonable; and executability refers to whether the interface and input constraints are met. Generated improvement suggestions Added to the context: Subsequently, the designers generated a revised reward function based on the suggestions: The process of generating reward function candidates by the advisors and designers involves multiple iterations, resulting in a gradually converging set of reward candidates.

8. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework according to claim 7, characterized in that: Before determining the final reward function, the optimizer performs consistency checks and candidate selection, specifically including output consistency constraints and executable constraints. For output consistency constraints, a set of predefined test inputs is used. Next, assess the volatility of reward output: like If the reward function is unstable, it will be discarded. For executability constraints, check whether the reward function is satisfied. If the condition is not met, it will be eliminated; Only retain the set of candidate functions that simultaneously satisfy both consistency and executability: After incorporating all candidate reward functions and their evaluation information into the context, the optimizer outputs the final reward function by integrating historical information: This process selects the most consistent, robust, and task-goal-aligned reward function from the candidate space as the final reward design for the reinforcement learning training phase.

9. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework according to claim 5, characterized in that: In step S3, a deep deterministic policy gradient is used as the base model to transform the decision-making process of the autonomous driving agent into a Markov decision process: Among them, at time The intelligent agent receives the environmental state. Based on strategy Execute actions Then the environment returns a reward. and transferred to The optimization objective is to maximize the expected cumulative discount return: The agent's state includes the position, size, and current control information of the vehicle and surrounding vehicles: in, Including bicycles and The position of the surrounding vehicles , , The vehicle's 3D bounding box; The value function update objective is as follows: Policy gradient: The intelligent agent ultimately outputs vehicle control information, including throttle and steering wheel angle.

10. The method for decision-making at signalless intersections based on a large model-empowered reinforcement learning decision framework according to claim 9, characterized in that: In step S4, firstly, the interaction field is defined, constructing a time-varying elliptical safety field centered on the vehicle: in, For the geometric center of the vehicle, and For the major and minor semi-axles, The rotation angle is... and For angular velocity and phase; The elliptical field shape changes as the vehicle enters the intersection, encompassing two forms: a vertically elongated TTS and a flattened, wide TWS. Before entering the intersection, it is a vertically elongated TTS; during its passage through the intersection, it becomes a flattened, wide TWS; and after leaving the intersection, it reverts to a vertically elongated TTS. This shape change is adjusted based on variations in the rotation angle and the major and minor semi-axles. Rotation angle , It is the vehicle's heading angle; Long half shaft ,in It is the geometric length of the vehicle. It is the minimum longitudinal static safety distance. It is a speed safety correlation coefficient. It's the vehicle speed; short half shaft ,in For the geometric width of the vehicle, For fixed width redundancy, It is the compensation coefficient. It is yaw compensation; Vehicles at different stages at the intersection are indicated by symbols. This indicates that TTS corresponds to TWS corresponds ; The process of elliptical field morphological change is as follows: Instantaneous trajectory prediction is introduced to determine whether a collision occurs. The next prediction step is performed based on the current vehicle state, and the prediction point is represented as follows: The coordinates of the predicted points are obtained by predicting the self-vehicle and other vehicles. If the line connecting the self-vehicle's predicted points to the predicted points of the self-vehicle and the line connecting other vehicles' predicted points to the predicted points of other vehicles intersect, it indicates that there is a conflict area, and then the next step of traffic strategy adjustment is triggered. If a conflict occurs, the traffic strategy is adjusted according to the current elliptical field shape. Specifically, if the self-vehicle has not yet entered the intersection, a deceleration strategy is adopted; if the self-vehicle has already entered the intersection, an acceleration strategy is adopted.

Images