A method, system, device and medium for controlling left turns of vehicles at an unsignalized intersection

Abstract

This invention relates to the field of intelligent transportation technology and discloses a method, system, device, and medium for controlling left turns at unsignalized intersections. The method includes: acquiring a bird's-eye view semantic map of the unsignalized intersection; inputting the bird's-eye view semantic map into a left-turn decision model trained based on a reinforcement learning algorithm to obtain the projection of the relative speeds of the target vehicle making a left turn at the unsignalized intersection and the straight-ahead vehicles onto their relative distance, as well as the collision avoidance relative accelerations of the target vehicle and the straight-ahead vehicles, to determine the collision risk between the target vehicle and the straight-ahead vehicles; and obtaining a control strategy for the target vehicle making a left turn at the unsignalized intersection using road and vehicle information of the unsignalized intersection and the collision risk between the target vehicle and the straight-ahead vehicles, to control the target vehicle; wherein, during model training, experience is generated by an agent through human expert intervention in the reinforcement learning algorithm based on the collision risk, and this experience is stored in an experience buffer along with the agent-generated experience.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a method, system, device and medium for controlling left turns of vehicles at unsignalized intersections. Background Technology

[0002] Currently, high construction and maintenance costs mean that T-junctions often lack traffic lights, leading to frequent interactions between vehicles from different directions within the intersection. Consequently, the proportion of left-turn accidents at unsignalized intersections is relatively high. Left turns for autonomous vehicles at unsignalized T-junctions are a typical high-risk driving scenario. When making a left turn at an unsignalized intersection, autonomous vehicles need to avoid and respond to surrounding vehicles with uncertain driving intentions, while also aiming to pass through the intersection as quickly as possible.

[0003] As a commonly used method for controlling left turns at unsignalized intersections, reinforcement learning (RL) enables autonomous vehicles to adjust their driving strategies in real time to cope with dynamically changing road conditions and sudden uncertainties through interactive learning between the vehicle and its environment. However, in typical high-risk scenarios such as unsignalized intersections, especially in environments where sparse rewards or collision risks are not easily perceived, traditional RL agents often struggle to learn effective strategies from key experiences, resulting in a high failure rate for left turns, a high risk of collision, and a long training time. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, device, and medium for controlling left turns of vehicles at unsignalized intersections, which can solve the problems of high failure rate, high collision risk, and long training time of traditional RL methods in unsignalized intersection scenarios.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for controlling left turns of vehicles at unsignalized intersections, comprising the following steps: Obtain a bird's-eye view semantic map of an unsignalized intersection, wherein the bird's-eye view semantic map includes road information and vehicle information of the unsignalized intersection; The bird's-eye view semantic map is input into a left-turn decision model trained based on a reinforcement learning algorithm. The left-turn decision model obtains the projection of the relative speed of the target vehicle making a left turn at an unsignalized intersection and the straight-going vehicle onto their relative distance, as well as the collision avoidance relative acceleration of the target vehicle and the straight-going vehicle, based on road and vehicle information, in order to determine the collision risk between the target vehicle and the straight-going vehicle. The left-turn decision model is trained based on a reinforcement learning algorithm. By using road and vehicle information at unsignalized intersections and the collision risk between the target vehicle and straight-going vehicles, the control strategy for the target vehicle when making a left turn at an unsignalized intersection is obtained. During training, if the collision risk is determined to be greater than a preset threshold, the left-turn decision model uses human experts to take over the reinforcement learning algorithm to generate experience. This experience is stored in an experience buffer along with the experience generated by the agent, so that experience can be sampled from the experience buffer for training. Based on the acquired control strategy, the target vehicle is controlled to make a left turn at an unsignalized intersection.

[0006] Optionally, during training, if the collision risk is less than or equal to a preset threshold, the left-turn decision model will only generate experience through the agent and store it in the experience buffer, so as to sample experience from the experience buffer for training.

[0007] Optionally, if the experience buffer contains both human expert and agent-generated experiences, then the human expert and agent-generated experiences are assigned sampling priorities according to the collision risk, so that experience is sampled from the experience buffer for training based on the priority of the experience.

[0008] Optionally, the reward function of the left-turn decision model includes: a task completion reward function that rewards the target vehicle after it successfully turns left; a collision penalty function that penalizes the target vehicle when it is involved in an accident; an efficiency reward function that rewards the target vehicle based on its successful speed; a risk penalty function that penalizes the target vehicle based on the collision risk with oncoming vehicles; an idling penalty function that penalizes the target vehicle for its low-speed behavior or waiting behavior; and a human intervention penalty function that penalizes the human expert when the human expert begins to generate experience.

[0009] Optionally, the control strategy includes the longitudinal pedal opening and lateral steering angle of the target vehicle.

[0010] Embodiments of the present invention also provide a vehicle left-turn control system for unsignalized intersections, comprising: The data acquisition module is used to acquire a bird's-eye view semantic map of an unsignalized intersection, which includes road information and vehicle information of the unsignalized intersection. The risk detection module is used to input the bird's-eye view semantic map into a left-turn decision model trained based on a reinforcement learning algorithm. The left-turn decision model obtains the projection of the relative speed of the target vehicle making a left turn at an unsignalized intersection and the straight-going vehicle onto their relative distance, as well as the collision avoidance relative acceleration of the target vehicle and the straight-going vehicle, based on road and vehicle information, in order to determine the collision risk between the target vehicle and the straight-going vehicle. The left-turn decision model is trained based on a reinforcement learning algorithm. The strategy acquisition module is used to acquire the control strategy of the target vehicle when making a left turn at an unsignalized intersection by using road and vehicle information at the unsignalized intersection and the collision risk between the target vehicle and the straight-going vehicle; wherein, when the left turn decision model is trained, if it is determined that the collision risk is greater than a preset threshold, the experience generated by the agent of the reinforcement learning algorithm is taken over by a human expert and stored in the experience buffer along with the experience generated by the agent, so as to sample the experience from the experience buffer for training. The vehicle control module is used to control the target vehicle to make a left turn at an unsignalized intersection based on the acquired control strategy.

[0011] Embodiments of the present invention also provide a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described unsignaled intersection vehicle left-turn control method.

[0012] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for controlling left turns of vehicles at unsignalized intersections.

[0013] The method for controlling left turns at unsignalized intersections provided by this invention has at least the following beneficial effects: This invention improves upon existing reinforcement learning algorithms. Faced with road and vehicle information at unsignalized intersections, it obtains the control strategy for a target vehicle making a left turn at an unsignalized intersection through a left-turn decision model trained using reinforcement learning. During training, the left-turn decision model first determines the collision risk between the target vehicle and oncoming vehicles based on road and vehicle information. If the collision risk exceeds a preset threshold (high-risk scenario), existing reinforcement learning algorithms may be unable to handle it. In this case, the strategy of human experts is worth learning. Therefore, in the data placement area (experience buffer) autonomously explored by the reinforcement learning algorithm, both the experience gained by the agent through the reinforcement learning algorithm and the experience gained by human experts are simultaneously generated. Instead of simply updating the strategy based on the agent's experience, a safer, more comfortable, and more efficient left-turn strategy at unsignalized intersections can be obtained.

[0014] The collision risk is determined by combining the projection of the relative speed of the target vehicle turning left at an unsignalized intersection and the relative distance between the two vehicles, as well as the collision avoidance relative acceleration of the target vehicle and the straight-going vehicle. The direction factor is introduced into the projection component of the relative distance between the relative speeds of the two vehicles, which is more consistent with the scenario of a vehicle turning left. The collision avoidance relative acceleration of the two vehicles can also characterize the collision severity during the vehicle's left turn. Therefore, the collision risk determined by combining these two aspects is more accurate, providing a solid foundation for the subsequent left-turn strategy search process. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0016] Figure 1 A flowchart illustrating a method for controlling left turns at unsignalized intersections provided by the present invention; Figure 2 A schematic diagram of an autonomous vehicle making a left turn at a signalless T-junction provided by the present invention; Figure 3 A schematic diagram of a traditional RL framework for autonomous driving provided by the present invention; Figure 4 A schematic diagram of a REPH-RL algorithm framework provided by the present invention; Figure 5 This invention provides two schematic diagrams of left-turn conflicts at an unsignalized T-junction; wherein, (a) is a schematic diagram of side-scratching conflicts in opposite directions, and (b) is a schematic diagram of side-scratching conflicts in the same direction; Figure 6 This invention provides a schematic diagram of a risk enhancement priority experience replay mechanism for humans in a human-environmental learning (RL) loop. Figure 7 A schematic diagram of an intervention time step provided by the present invention; Figure 8 This is a schematic diagram of an experimental simulation environment CARLA provided by the present invention; wherein, (a) is a schematic diagram of the town01 experimental scene in CARLA, (b) is a bird's-eye view of the scene of an autonomous vehicle turning left at an unsignaled T-junction in CARLA, and (c) and (d) are schematic diagrams of the training process in the CARLA experimental scene respectively. Figure 9 This invention provides a bird's-eye view semantic diagram constituting a state space; Figure 10 The diagram illustrates the training process of different algorithms provided by the present invention; wherein, (a) is a schematic diagram of the training reward curve, and (b) is a box curve diagram of the average risk coefficient value; Figure 11 A schematic diagram of the frequency distribution of the average value of the longitudinal acceleration rate of change provided by the present invention; wherein, (a) is a schematic diagram of the training scenario and (b) is a schematic diagram of the test scenario; Figure 12 A box curve diagram of an average risk coefficient value provided by the present invention; wherein, (a) is a schematic diagram of a training scenario and (b) is a schematic diagram of a test scenario; Figure 13 This invention provides a box plot of the average risk coefficient value of an algorithm under different human driving levels of intervention; wherein, (a) is a training scenario diagram and (b) is a test scenario diagram; Figure 14 This invention provides a schematic diagram of the learning efficiency evaluation of a priority experience replay mechanism with different priority indicators in a training scenario; wherein, (a) is a schematic diagram of the training reward curve, and (b) is a schematic diagram of the box curve of the average risk coefficient value. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0018] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] One embodiment of the present invention relates to a method for controlling left turns of vehicles at unsignalized intersections. The specific process of the method for controlling left turns of vehicles at unsignalized intersections in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Obtain the bird's-eye view semantic map of the unsignalized intersection. The bird's-eye view semantic map includes road information and vehicle information of the unsignalized intersection.

[0020] Step 102: Input the bird's-eye view semantic map into the left-turn decision model trained based on the reinforcement learning algorithm. The left-turn decision model obtains the projection of the relative speed of the target vehicle making a left turn at an unsignalized intersection and the straight-going vehicle on the relative distance between them, as well as the collision avoidance relative acceleration of the target vehicle and the straight-going vehicle, based on road information and vehicle information, so as to determine the collision risk between the target vehicle and the straight-going vehicle. The left-turn decision model is trained based on the reinforcement learning algorithm.

[0021] Step 103: Obtain the control strategy for the target vehicle when making a left turn at an unsignalized intersection by using the road and vehicle information of the unsignalized intersection and the collision risk between the target vehicle and the straight-going vehicle; wherein, when the left-turn decision model is trained, if it is determined that the collision risk is greater than a preset threshold, the experience generated by the agent of the reinforcement learning algorithm is taken over by human experts and stored in the experience buffer along with the experience generated by the agent, so as to sample the experience from the experience buffer for training.

[0022] Step 104: Based on the acquired control strategy, control the target vehicle to make a left turn at the unsignalized intersection.

[0023] The following is a detailed description of the implementation details of the vehicle left-turn control method at an unsignalized intersection in this embodiment. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0024] This embodiment models the autonomous driving left-turn decision-making task for controlling left turns of vehicles at unsignalized intersections as a reinforcement learning problem in a continuous action space, and uses a Markov Decision Process (MDP) to define the interaction between the environment and the agent.

[0025] A Markov decision process is represented by a quadruple (S, A, R, P), where: S is the state space, consisting of continuous state variables, denoted as s∈S; A is the action space, consisting of continuous action variables, denoted as a∈A; the reward function R(·|s,a): S×A→R is used to map the state-action pair (s,a) to a deterministic immediate reward value r; the state transition probability function P(·|s,a): S×A→P(S) represents the conditional probability distribution of the agent transitioning to the next state s′ after performing the action a∈A in the current state s∈S.

[0026] When faced with a scenario where an autonomous vehicle enters an unsignalized intersection to make a left turn, follow these steps: State input: A bird's-eye view semantic map is generated by fusing onboard sensors and used as the state space input. The image encodes the vehicle's position, the position and speed of surrounding vehicles traveling straight, and road boundary and lane information.

[0027] Risk Calculation: The risk perception module extracts relative distance and velocity vectors based on the bird's-eye view semantic map, calculates the improved TTC and rDRAC in real time, and outputs the risk coefficient λ(t).

[0028] Takeover and Demonstration: If λ(t) exceeds the preset threshold, the system triggers the takeover mechanism, allowing human experts to gain full control of the vehicle; their operations are recorded as demonstration data and stored in the experience playback buffer.

[0029] Automatic control: If no intervention is triggered, the reinforcement learning model outputs actions based on the current state, including longitudinal pedal opening and lateral steering angle. Longitudinal actions control vehicle acceleration or braking, while lateral actions are tracked via a PI controller.

[0030] Reward Calculation: After an action is performed, the system generates a reward value, which includes task completion reward, collision penalty, timeout penalty, idling penalty, risk penalty, and human intervention penalty.

[0031] Iterative update: The state, action, reward and next state form a transition tuple and are stored in the experience buffer; the buffer adopts a composite priority sampling mechanism based on TD error and risk coefficient to make high-risk demonstration data easier to sample; then the Actor and Critic networks are updated until convergence is completed.

[0032] In the above process, each state observation, risk calculation, takeover judgment, action execution, and reward feedback of the vehicle constitutes a training or control loop, thereby ensuring that the algorithm is not only an abstract improvement framework, but can be directly mapped to the actual left-turn task of the autonomous vehicle.

[0033] At each discrete time step t, the reinforcement learning agent responds to the environment state. Perform observations and select and execute actions within the action space. Then receive instant rewards from the environment. and the corresponding next state Feedback. The agent's behavior is controlled by the policy function π(a|s):S→P(A), which defines the conditional probability distribution of each possible action in a given state. The state-action distribution caused by policy π can be expressed as... .

[0034] The optimization objective of reinforcement learning is to obtain the optimal control policy for vehicle steering, maximizing the cumulative expected reward in the dynamic environment V. To this end, a Bellman value function (critic function) is constructed to estimate the state value V(s); the commonly used action value function is denoted as Q(s,a), which characterizes the expected reward value under the state-action pair (s,a). Its mathematical definition under any policy π is as follows:

[0035] Where γ∈(0,1) is the discount factor. Then we can obtain the policy function that considers maximizing the Q-value, expressed as follows:

[0036] A neural network is used as a function approximator to define the actor and critic, and then the target is achieved through loss functions. Specifically, the loss function ℒQ for the critic and the loss function ℒπ for the actor can be expressed as:

[0037] Wherein, Q(·; () represents the parameterized critic function. These represent the parameters of the critic network. This represents a parameterized actor function. The parameters represent the actor network.

[0038] This example uses two human guidance behaviors used in RL training: intervention and demonstration.

[0039] 1) Intervention: During the training of the RL agent, human experts determine whether intervention is necessary based on prior knowledge and reasoning ability. If a human expert decides to intervene, they gain full control over the agent. Human experts typically intervene when the agent performs a disastrous action or gets trapped in a local optimum. Therefore, by reflecting human intervention factors in the reward function of the RL model, the agent can learn human strategies to avoid recurring adverse situations requiring intervention.

[0040] 2) Demonstration: When a human expert intervenes, the resulting action influences the RL agent's generation of corresponding reward signals and next state. The resulting transition tuple can be viewed as demonstration data, caused by the human expert's policy rather than the agent's behavior. The agent can learn human behavioral policies from this demonstration data.

[0041] Based on the two types of human-guided behaviors described above, this embodiment establishes an actor-critic framework adapted to human intervention, simultaneously applying intervention learning and demonstration learning. First, in the interaction between the RL agent and the environment, the agent's behavioral policy outputs actions to explore the environment. For an off-policy actor-critic framework, the above process can be described as follows:

[0042] At this point, the action output by the agent is modified as follows: In the formula, This represents actions taken under the strategy of human experts. It is a demonstration mask. When a human demonstration occurs, It is a verification matrix, which is a zero matrix in non-demonstration steps.

[0043] In the human-guided actor-critic framework, the experience replay mechanism establishes an experience buffer to store process data at each interaction between the agent and the environment. The agent can retrieve data generated by previous policies from the buffer to evaluate and improve the policy. At any given time step t, the interaction between the agent and the environment generates a transition tuple, which is stored in the experience replay buffer and represented as:

[0044] When actions resulting from human intervention are performed in the environment, the interactive transition tuple... Actions generated under both human and RL policies will be recorded and stored in the experience replay buffer. Specifically, actions generated under both human and RL policies will be stored in the same buffer. In this case, new transition tuples... The following definition is used to distinguish between human demonstration data and RL agent output data:

[0045] Existing research often uses uniform random sampling to sample from the experience replay buffer, but the PER mechanism is a more efficient method. The PER mechanism originates from the concept of "importance sampling," and its core idea is to accelerate learning by dynamically adjusting sample priorities. The PER mechanism makes the data samples follow a non-uniform distribution. Its probability distribution function Represented as:

[0046] in, [0,1] is the scaling factor. Represent each tuple i The priority, and the tuple is determined by the TD error. Decision. Priority. Defined as;

[0047] Suppose there is a batch of transition tuples of size N, which may contain data from the agent's policy. and data from human strategies The critic network based on the optimal value function can learn from both strategies simultaneously. The loss function is defined as:

[0048] For human demonstration data, the actor network not only needs to maximize the critic's value estimate but should also learn from this demonstration data. Therefore, a loss function that integrates behavioral clones was designed for the actor network. Behavioral cloning is typically achieved by minimizing the output of the policy network. With expert action The policy network is trained using the mean squared error between the two sides. Therefore, its loss function is defined as:

[0049] S3 This invention achieves real-time risk perception by introducing two types of indicators: rTTC and rDRAC. Figure 2 In the schematic diagram of the medium-risk perception module, the circular area centered on the vehicle represents the interaction range between left-turning vehicles and straight-going vehicles on intersecting roads, as defined in this embodiment. The interaction time between the two vehicles begins when both enter the circular area and ends when one vehicle passes the assumed conflict point. During the interaction, the collision risk of the turning vehicle depends on both its own motion and that of the straight-going vehicle. This embodiment incorporates the vehicle's steering angle into the improvement to TTC (Traffic Tracing) to construct an evaluation index for measuring potential collision risk. Figure 5 As shown in the left figure, assuming a vehicle turns left... n and straight-going vehicles n The potential collision point of -1 is D( x , y The position of vehicle n is ( x n , y n ), the motion state is ( v n , a n );vehicle n -1 is the position ( x n-1 , y n-1 The motion state is ( v n-1 , a n-1 The real-time relative distance between the two vehicles is then calculated. L n,n-1 The formula is shown in equation (13):

[0050] When vehicles traveling straight come from another approach lane, such as Figure 5As shown in the right figure, since the autonomous vehicle only performs risk perception modeling for the closest environmental vehicles, the real-time relative distance between the two vehicles is still calculated using the definition of equation (13). According to equation (13), the relative distance between the two vehicles can be calculated based on their current positions relative to the potential collision point. The formula for calculating the angle between the driving direction of the left-turning vehicle and the straight-going vehicle is obtained from the cosine theorem:

[0051] vehicle n The component of the velocity in the direction of relative distance is: This embodiment defines the potential collision risk between a left-turning vehicle and a straight-going vehicle as equal to the projection of their relative speeds over their relative distance, while the severity of the potential collision risk is equal to the maximum relative acceleration required for collision avoidance. A comprehensive steering risk index is constructed based on these two aspects. First, collision risk is evaluated by improving the Time-to-Cross (TTC) method. A directional factor is introduced into the projection components of the two vehicles' speeds over their relative distance, making the TTC applicable to vehicle steering scenarios. The improved index is defined as follows:

[0052] In addition, drawing inspiration from the concept of the collision avoidance acceleration (DRAC) index, a collision avoidance relative acceleration (rDRAC) that takes into account the relative motion of the two vehicles is proposed to characterize the severity of the collision during the vehicle's left turn.

[0053] Based on the rTTC and rDRAC indicators defined above, the risk assessment index for left-turning vehicles is constructed as shown in the following formula.

[0054] Finally, the comprehensive risk index is converted into a risk coefficient. : λ(t) is a dynamic weighting index used for risk perception, which will be used to strengthen the selection of experience memory in experience-first sampling. The larger the risk coefficient, the greater the potential collision risk of the vehicle.

[0055] A reinforcement learning paradigm that integrates risk-enhanced PER mechanisms is proposed, such as... Figure 4 and Figure 6 As shown. Its priority calculation does not simply use TD error, but also uses the real-time turning risk output by the risk perception unit. Compared to traditional reinforcement learning paradigms (such as... Figure 3As shown in the figure, the reinforcement learning paradigm proposed in this embodiment adds a risk perception module, a human demonstration module under human intervention, a reward correction method for adjusting the reward value of the RL algorithm under human intervention, and a REn-PER mechanism that incorporates the risk coefficient.

[0056] Specifically, the risk perception module is a lightweight module used to calculate the dynamic risk of the vehicle during steering in real time and quantify it into a risk coefficient. If the output of this module is a low-risk value, and human experts believe that no intervention is needed, the vehicle will act according to the strategy output by the RL algorithm. If the output of this module is a high-risk value, and human experts believe that intervention is needed to avoid potential collision risks, the model will accept human expert intervention, giving the human expert full control, and using the human demonstration module to collect expert demonstration data. Borrowing the strategy of reshaping the reward function during human intervention, the reward function of the RL algorithm is modified to distinguish between human intervention and non-intervention. This human intervention mechanism not only improves the training efficiency of RL but also reduces the workload of expert demonstration to some extent. Furthermore, the risk coefficient is integrated into the PER mechanism, i.e., REn-PER. The fused mechanism can adjust the sampling priority of human demonstration data according to the risk coefficient, ensuring that human demonstration data in high-risk situations can be preferentially used for agent training. This complete RL strategy is called the REPH strategy.

[0057] This embodiment selects the TD3 algorithm as the basic algorithm for vehicle steering action decision-making, which has been proven to be an effective method in unsignalized intersection scenarios. TD3 is a DRL algorithm specifically designed for continuous motion control. It learns the most effective strategy by optimizing the action value function through a dual-Q network and a delayed update strategy. This embodiment will apply the REPH strategy to improve the TD3 algorithm, and the improved algorithm will be named REPH-TD3.

[0058] In TD3, the target network, i.e., the parameters are... criticism and parameters are actor This is used for algorithm updates. Considering that human experts sometimes intervene in the RL training process, at a certain time step... t The actions of the agent will be represented as:

[0059] in, The noise coefficient vector depends on the training process, and c is the boundary of the action space exploration. It is a Gaussian distribution The covariance matrix.

[0060] A new transition tuple is obtained through the above interactive steps. And store it in the experience playback buffer: The experience replay buffer, which stores tuples of human experience data, will be sampled for use in... and Training. A transition tuple with human demonstration is defined as... When human intervention occurs, the buffer zone The empirical data in the data follow a new non-uniform distribution. Then the probability distribution function in equation (9) is redefined as :

[0061] in, The new sampling priority for the experience buffer is calculated as follows: in, The risk priority weight has a value of 0.5. Equation (25) represents the risk priority enhancement term REn. During RL training, as the level of scenario risk changes and the agent gradually learns human expert experience, the priority level of the human demonstration data tuples will also change accordingly. This reflects a dynamic prioritization mechanism. This mechanism is called the REnTD mechanism because it combines two indicators as a measure of the quality of human role models.

[0062] The prioritization of experience alters the distribution of experience within the buffer, introducing a bias in the expected estimate of the value function. While this bias generally has a small impact on the asymptotic performance of RL, it can affect the stability and robustness of the policy in certain situations. This estimation bias can be gradually eliminated by introducing importance sampling weights into the loss function of the value network. (Transitional tuples) i The importance sampling weights are calculated as follows:

[0063] in, It is a coefficient. If, Then completely uneven sampling will occur if, This will result in perfectly uniform sampling. As the training process progresses, the weights gradually decrease to zero. The loss function of a value network that includes importance sampling weights is defined as:

[0064] To reduce the workload of human involvement in training RL agents, human experts are only allowed to intervene in the training process when the agent's behavior presents a potential risk or a suboptimal tendency. In this case, the intervention event can be seen as a negative signal, and the RL should avoid the corresponding state. This negative feedback can be achieved by modifying the reward.

[0065] First, define the intervention event. The demonstration mask defined in equation (6) Used to record actions Whether the action is demonstrated by a human. In the temporal Markov decision training process of RL, the initial time step controlled by human demonstration is set as the intervention time step, using... and Indicates, such as Figure 7 As shown. It's important to note that penalties are only applied at the intervention time step, not in subsequent steps, because humans demonstrate in later steps, and the agent needs to learn from the demonstration data. For example, in... Figure 7 In this scenario, S2 is penalized, while S3 and S4 are not.

[0066] Experimental scenarios were configured using the high-fidelity autonomous driving simulation software CARLA, and the proposed method was trained and evaluated. In CARLA simulator version 0.9.12, the Town01 map was selected to configure an unsignalized intersection scenario, such as... Figure 8 As shown in the diagram. In this invention, a scenario of an unsignalized intersection involving multiple traffic participants is designed. The vehicles in the red boxes are the resident vehicles, and the vehicles in the blue boxes are background vehicles. The resident vehicle attempts to turn left and merge into the main road. The initial speed of the background vehicles is randomly generated within the range of [4m / s, 8m / s]. The types of background vehicles are also randomly generated, including small cars, medium-sized cars, and large cars. The number of background vehicles is randomly generated within the range of [3, 6], and all are controlled by the IDM model to drive straight along the lane. All background vehicles are set to an aggressive driving style, meaning they will not yield to the resident vehicles. The state update cycle for all vehicles is 0.1s.

[0067] The specific steps for MDP modeling in this embodiment are as follows: 1) State: Use Figure 9 The bird's-eye view semantic map shown serves as the state information of the RL agent. This form encodes a wealth of information about road structure and interaction, and can handle any number of objects, making it suitable for highly interactive urban driving scenarios. In the map, the red rectangles turning left represent the driver's vehicle, and the red rectangles going straight represent background vehicles. The size of the color blocks distinguishes vehicle types. The dark purple area represents the road. Two consecutive frames of the bird's-eye view semantic map are used to assign values ​​to state variables to achieve temporal awareness. Furthermore, to reduce computational burden, the bird's-eye view semantic map is scaled to a smaller size. The final state variables can be represented as:

[0068] in, It is a pixel matrix whose elements are normalized.

[0069] 2) Action: To facilitate comparative testing with the REPH and PHIL strategies proposed in this embodiment, this invention uses independent lateral and longitudinal control to control the self-steering vehicle. The accelerator / brake pedal opening is selected as the action variable of the RL agent for longitudinal control, which can be expressed as:

[0070] in, It represents the continuous pedal opening, where negative values ​​indicate braking commands and positive values ​​correspond to acceleration commands. When the agent controls the pedal opening, a proportional-integral (PI) controller is used for lateral control.

[0071] 3) Reward: The vehicle's goal is to safely, comfortably, and efficiently complete a left turn at an unsignalized intersection and reach the target exit lane. Therefore, the reward function for RL is defined as:

[0072] in, , and These are the rewards for successfully navigating the intersection and reaching the destination, and the penalties for collisions and timeouts, with a timeout standard of 400 time steps. They are represented as follows:

[0073] This is an idling penalty set to encourage vehicles to pass through intersections quickly, and the calculation method is as follows: in The target speed is 0.3. v target It is 5 m / s. It has a implicit impact on improving vehicle ride stability. It is a real-time driving risk value that encourages vehicles to perform low-risk actions and penalizes high-risk situations. It is defined as follows:

[0074] Furthermore, at the intervention time step, an additional penalty term is used to modify the original reward function. The modified reward function is shown below:

[0075] in It is the revised reward value, and It is a scalar measure that weighs intervention and punishment; its value is related to... same.

[0076] The following algorithm is used to compare the performance of the REPHIL-TD3 algorithm proposed in this embodiment.

[0077] 1) Vanilla TD3: In the comparative test, the basic PER mechanism was combined with the original TD3, but no human demonstration mechanism was set up.

[0078] 2) Vanilla SAC: Soft-Actor-Critic (SAC) is an advanced offline policy reinforcement learning algorithm that derives the optimal policy by jointly optimizing future cumulative rewards and Shannon entropy. In the comparative experiment, the basic PER mechanism was combined with the original SAC, but no human demonstration mechanism was set up.

[0079] 3) HI-TD3: Human intervention reinforcement learning (HIRL) is a combination of discrete action reinforcement learning algorithms and human demonstrations. By replacing the discrete action basic algorithm (DQN) used in this method with the TD3 algorithm, an improved HIRL algorithm is obtained. Furthermore, a PER mechanism is added to this algorithm for fair comparison.

[0080] 4) IA-TD3: Intervention-Assisted Reinforcement Learning (IARL) is a combination of continuous action RL algorithms and human demonstrations. By replacing the basic continuous action algorithm (PPO) used in this method with the TD3 algorithm, an improved IARL algorithm is obtained. Furthermore, a PER mechanism is added to this algorithm for fair comparison.

[0081] 5) PHIL-TD3: Compared with the proposed REPH-TD3 method, this method does not consider scenario risks, ignores the potential collision risks during left turns at unsignalized intersections, and has insufficient safety requirements for the turning process.

[0082] The experimental hardware included a driving simulator and a high-performance workstation. The driving simulator was used to perform human demonstration driving when human intervention was required. The workstation was used to run the RL algorithm. Detailed configuration of the experimental platform is provided in Table 2.

[0083] The algorithm's training process consists of 400 episodes, with a preset maximum number of steps per episode of 402. During training, an episode ends when the vehicle reaches the target position, collides with another vehicle, or reaches the preset time step limit. Detailed parameter settings for the algorithm are shown in Table 3.

[0084] Table 2. Configuration of the experimental platform Table 3 Hyperparameter settings for the REPH-TD3 algorithm from Figure 10 As shown in (a), compared to the other four human-guided algorithms, the baseline TD3 (vanillaTD3) has the lowest reward value, despite its faster convergence speed. This may be because Vanilla TD3's learning strategy is poor and unable to quickly optimize the current decision, thus keeping the reward value at a low level. The four human-guided algorithms achieved higher reward values ​​in shorter training periods, indicating the effectiveness of human guidance. Among these four algorithms, although HI-TD3 has the fastest convergence speed, its asymptotic performance is the worst compared to the other three human-guided algorithms, as can be seen from its reward value. IA-TD3 has certain advantages over HI-TD3 in both asymptotic performance and convergence speed, but it still has shortcomings compared to PHIL-TD3 and REPH-TD3. PHIL-TD3 is inferior to REPH-TD3 in both asymptotic performance and convergence speed. This may be because PHIL-TD3 is insufficient in learning human decision-making in complex, high-risk scenarios, causing the algorithm to repeatedly fail when facing high-risk scenarios. As the best-performing algorithm, REPH-TD3 can quickly learn from demonstrations made after human intervention in high-risk scenarios and obtain a relatively optimal strategy. Notably, REPH-TD3 achieves the best asymptotic average reward over approximately 120 episodes, demonstrating a learning efficiency more than 33% higher than PHIL-TD3. REPH-TD3's average reward is 25.9% higher than HIRL-TD3 and more than 30% higher than vanilla TD3.

[0085] In addition, the performance of human-guided algorithms was evaluated using the mean episodic risk coefficient value. A lower mean episodic risk coefficient value indicates better learning performance. Figure 10(b) shows the average risk values ​​of different algorithms on the training set, where Van-TD3 refers to Vanilla-TD3 and Van-SAC refers to Vanilla-SAC. Compared with the TD3 algorithm, the REPHIL-TD3 algorithm reduces the average risk coefficient by 45.8%, and compared with the best algorithm PHIL-TD3, the average risk coefficient reduces by 29.2%, indicating that REPHIL-TD3 can significantly reduce the potential collision risk and improve safety. Compared with the five baseline algorithms, the REPHIL-TD3 algorithm can complete the training at a low risk level.

[0086] Table 4 reports the CPU time consumed during the training process of each algorithm. HI-TD3 and the original TD3 algorithm consumed the least amount of time. This is because, compared to REPH-TD3 and PHIL-TD3, they lack certain modules such as the risk-aware PER mechanism and the switching design between RL and human actions, resulting in relatively lower computation time. Although the REPH-TD3 algorithm is more complex, its training time is basically the same as PHIL-TD3. This is mainly attributed to the fact that the lightweight dynamic risk-aware module of the REPH-TD3 algorithm has lower requirements for computing resources during operation, thus avoiding additional computational overhead.

[0087] It is worth noting that, by Figure 10 As shown in (a), compared to the Vanilla-SAC algorithm, Vanilla-TD3 is superior in both convergence speed and asymptotic performance during training. Meanwhile, Table 4 reveals that the SAC algorithm takes significantly longer than the TD3 algorithm, while the difference is not significant compared to other improved TD3 algorithms. This indicates that the SAC algorithm requires more computational resources, and as a baseline algorithm, it lacks the advantage over the TD3 algorithm. Although... Figure 10 As can be seen from (b) in the figure, SAC has some advantages in terms of average risk coefficient value, but it is significantly inferior to TD3 in terms of learning performance and computational performance during the training process. This is basically consistent with the research conclusions of Xu et al., so it is not selected as the benchmark algorithm for improvement.

[0088] Overall, the REPH-TD3 method demonstrates significant advantages in improving training efficiency and performance, while significantly reducing the potential collision risk, and without significantly increasing computational resource consumption.

[0089] Table 4 CPU clock time for average step size of the algorithm Besides the learning performance during model training, post-training performance is also an important indicator for evaluating the advantages of different methods. In the testing phase, the trained RL agent is deployed to an experimental environment, and the performance of the REPH-TD3 algorithm and the comparison algorithm is evaluated in terms of generalization, safety, and comfort. No human guidance is required in this phase.

[0090] To verify the algorithm's generalization performance, the number of background vehicles in the experimental scenarios was expanded to a range of [4, 8]. It should be noted that this range was determined through multiple tests. When the number of background vehicles exceeded 8, the cross-crossing interval at the intersection was too short, causing the driver vehicle to always be in a yielding state during the trials, resulting in a lack of valuable test data. As the number of background vehicles increased, their vehicle types also became more diverse. To further enhance the scenario complexity, conservative and aggressive vehicles were configured in a 1:3 ratio. The added conservative vehicles meant they would stop when encountering turning vehicles. In each scenario, 50 trials were conducted on each trained algorithm, using the same random seed sequence to ensure reproducible results.

[0091] Table 5 reports the generalization performance test results of the four algorithms. The training scenarios in the table refer to those defined in IV.A, while the more complex scenarios defined in this section are called test scenarios. Success rate is defined as the ratio of the number of successfully completed trials to the total number of attempts in the same scenario. It can be observed that REPH-TD3 achieved the highest success rate. Observing the training scenarios, REPH-TD3 and PHIL-TD3 both showed relatively high success rates, but HIRL-TD3 and baseline TD3 performed poorly, especially baseline TD3. In the test scenarios, the success rates of all algorithms decreased to varying degrees. PHIL-TD3, HIRL-TD3, and TD3 showed significant decreases in success rate. However, the decrease in success rate for REPH-TD3 was relatively smaller. These results indicate that REPH-TD3 exhibits good robustness in the new test environment.

[0092] Table 5. Average episode time consumption and success rate of the human-centered driving strategy. This is a component of the RL reward, used to optimize the traffic efficiency of autonomous vehicles making left turns. This embodiment uses the average running time of the algorithm, i.e., the time from start to finish of the simulation, to evaluate the traffic efficiency of left-turning vehicles. Table 5 shows the running times of the four algorithms in training and testing scenarios. It can be seen that the REPH-TD3 algorithm reduces the running time in training and testing scenarios by 11.7% and 10.01% respectively compared to the PHIL-TD3 algorithm. Furthermore, given the smaller standard deviation of REPH-TD3's running time, its stability can be considered higher than that of the PHIL-TD3 algorithm. One possible explanation for this result is that PHIL-TD3 did not adequately learn human guidance in high-risk scenarios during training, thus failing to effectively cope with high-risk scenarios during the testing phase. It is worth noting that although the TD3 and HI-TD3 algorithms have the shortest average running times, this is mainly due to their low success rates. When a collision occurs while a vehicle is crossing an intersection, the simulation stops, resulting in the shorter average running times of TD3 and HI-TD3.

[0093] Figure 11 This paper presents the distribution of the average longitudinal acceleration change rate for four algorithms in training and testing scenarios. The longitudinal acceleration change rate reflects the vehicle's stability during operation and determines driving and riding comfort; a lower acceleration change rate indicates better comfort. Figure 11 As can be seen, all five algorithms appear frequently in the low acceleration rate of change region (8-12 m / s³), indicating that these algorithms can stably control the vehicle in most cases. However, in the high acceleration rate of change region (12-24 m / s³), the REPH-TD3 algorithm appears far less frequently than the other four algorithms, meaning that the vehicle controlled by this algorithm has the highest stability.

[0094] Figure 12 The real-time risk values ​​reflect the safety of vehicles controlled by the five algorithms in both training and testing scenarios. Higher box-ratio values ​​indicate a higher potential collision risk. It can be observed that REPH-TD3 exhibits the highest safety in both scenarios. Notably, in the testing scenarios, the safety of all algorithms decreases to varying degrees. The PHIL algorithm shows the largest decrease in safety at 45.5%, followed by the TD3 algorithm at 36.4%. REPH-TD3 and HI-TD3 show smaller safety decreases, but HI-TD3 has the lowest safety in both scenarios. Therefore, REPH not only performs best in both scenarios but also demonstrates strong robustness in more complex testing scenarios.

[0095] The skill level of human demonstrators can influence the learning effectiveness of RL agents; therefore, the behavior of demonstrators with lower driving skills may not be worth learning. This embodiment designs an experiment to test the impact of human driver proficiency on four human-guided RL algorithms. Two types of drivers were selected for the experiment: practiced drivers (PD) and unpracticed drivers (UD). Practiced drivers had five years of driving experience and at least 30 hours of simulator experience. Unpracticed drivers had less than one year of driving experience and less than one hour of simulator driving experience. Both drivers performed 50 tests each in training and testing scenarios. The test results are as follows: Figure 13 As shown in the figure, the vertical axis represents the average risk coefficient; a lower coefficient value indicates better algorithm performance.

[0096] from Figure 13 As can be seen, in both training and testing scenarios, the average risk of the REPH-TD3 algorithm under the demonstration of skilled drivers is lower than that under the demonstration of unskilled drivers, with average risk reductions of 60.2% and 29.6%, respectively. However, the performance of the HI-TD3 algorithm is exactly the opposite. This result may be because the HI-TD3 algorithm only replaces part of the RL exploration with human guidance, and is less affected by human guidance. Compared to the HI-TD3 algorithm, the REPH-TD3, PHIL-TD3, and IA-TD3 algorithms learn from human guidance by using behavioral cloning targets, thus intuitively showing that poor guidance significantly reduces the algorithm's performance. In addition, the risk coefficient of the REPH-TD3 algorithm differs significantly between the two different levels of driving guidance, while the difference for the PHIL-TD3 algorithm is relatively small. This indicates that driving level has a greater impact on the REPH-TD3 algorithm than on the PHIL-TD3 algorithm. This difference may be attributed to the fact that REPH-TD3 can distinguish the quality levels of different human demonstration strategies, thus making it easier to learn the optimal human demonstration strategy of skilled drivers, resulting in higher learning efficiency than PHIL-TD3.

[0097] Human demonstration data can improve the performance of RL agents. This embodiment uses the priority enhancement term REn defined in Equation (25) to evaluate the quality of human demonstration data and influences the agent's learning of sampled data based on the evaluation results. This embodiment sets up experiments to compare the impact of different priority schemes on performance. The "REn" scheme indicates that only the overall risk coefficient value is considered. Calculate REn priority. The “TD” scheme is inherited from the original PER method, but the TD weight for human demonstrations is doubled to highlight human guidance in the replay buffer. The “REn+TD” scheme is the REn priority calculation method used by the REPH-TD3 algorithm. Each algorithm performs five training attempts using the same random seed.

[0098] The learning curve of the algorithm using the 3-priority scheme is as follows: Figure 14 As shown in (a), the algorithms are labeled "TD", "REn", and "REn+TD". It can be seen that "TD" has a 19.8% faster learning speed than "REn", but its asymptotic performance is significantly lower than "REn". "REn+TD" combines the advantages of both priority schemes, achieving a faster learning speed than "TD" and a 12.8% higher asymptotic performance than "TD" and a 7.5% higher performance than "REn". Furthermore, as... Figure 14 As shown in (b), the risk value of "TD" is 16.5% higher than that of "REn". The risk value of "REn+TD" is 28.3% lower than that of "TD".

[0099] "TD" and "REn" use different metrics to score human demonstration data. The results above show that using TD metrics (such as the standard PER) does indeed help to quickly improve algorithm performance. However, in the later stages of training, algorithms using REn metrics outperform those using TD metrics. Unlike directly using TD, REn enables the agent to cope with complex, high-risk scenarios and maintain good asymptotic performance when learning the optimal policy. The proposed REPH-TD3 algorithm cleverly combines these two metrics, achieving optimal performance in both cases.

[0100] Overall, the core advantage of the "REn+TD" scheme lies in its ability to dynamically evaluate the quality level of human demonstrations, enabling refined screening of guidance samples. This allows for the precise transmission of positively instructive human demonstration experiences to the RL agent, ensuring that it continuously focuses on policy transfer from high-value demonstration data during the learning process, rather than passively accepting mixed experiences containing negative interference. This risk-enhanced evaluation-oriented priority optimization mechanism significantly improves the sample quality and utilization efficiency of human experience guidance, enabling the algorithm to more accurately integrate prior human knowledge with autonomous exploration results in complex decision-making scenarios, resulting in a substantial improvement in policy learning efficiency and performance.

[0101] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0102] Another embodiment of the present invention relates to a vehicle left-turn control system at an unsignalized intersection. The implementation details of this unsignalized intersection vehicle left-turn control system are described below. The following details are provided for ease of understanding and are not essential for implementing this solution. The unsignalized intersection vehicle left-turn control system of this embodiment includes: The data acquisition module is used to acquire a bird's-eye view semantic map of an unsignalized intersection, which includes road information and vehicle information of the unsignalized intersection. The risk detection module is used to input the bird's-eye view semantic map into a left-turn decision model trained based on a reinforcement learning algorithm. The left-turn decision model, based on road and vehicle information, obtains the projection of the relative speed of the target vehicle making a left turn at an unsignalized intersection onto the relative distance between the two vehicles, as well as the collision avoidance relative acceleration of the target vehicle and the straight-going vehicle, to determine the collision risk between the target vehicle and the straight-going vehicle. The left-turn decision model is trained based on a reinforcement learning algorithm. The strategy acquisition module is used to acquire the control strategy of the target vehicle when making a left turn at an unsignalized intersection by using road and vehicle information at the unsignalized intersection and the collision risk between the target vehicle and the straight-going vehicle. Among them, when the left turn decision model is trained, if it is determined that the collision risk is greater than a preset threshold, the human expert takes over the reinforcement learning algorithm to generate experience. This experience is stored in the experience buffer along with the experience generated by the intelligent agent, so that the experience can be sampled from the experience buffer for training. The vehicle control module is used to control the target vehicle to make a left turn at an unsignalized intersection based on the acquired control strategy.

[0103] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0104] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0105] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the unsignaled intersection vehicle left-turn control method of the above embodiments.

[0106] In this embodiment, the memory and processor are connected via a bus, which can include any number of interconnected buses and bridges, connecting various circuits of one or more processors and the memory together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be further described in this embodiment. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0107] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0108] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0109] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for controlling left turns of vehicles at an unsignalized intersection, characterized in that, The method includes: Obtain a bird's-eye view semantic map of an unsignalized intersection, wherein the bird's-eye view semantic map includes road information and vehicle information of the unsignalized intersection; The bird's-eye view semantic map is input into the left-turn decision model trained based on reinforcement learning algorithm. The left-turn decision model obtains the projection of the relative speed of the target vehicle and the straight-going vehicle at the unsignalized intersection on the relative distance between them, as well as the collision avoidance relative acceleration of the target vehicle and the straight-going vehicle, based on road information and vehicle information, so as to determine the collision risk between the target vehicle and the straight-going vehicle. By using road and vehicle information at unsignalized intersections and the collision risk between the target vehicle and oncoming vehicles, a control strategy for a target vehicle making a left turn at an unsignalized intersection is obtained. Specifically, during training, if the collision risk is determined to be greater than a preset threshold, the experience generated by the agent through a human expert taking over the reinforcement learning algorithm is stored in an experience buffer along with the agent-generated experience. If the collision risk is less than or equal to the preset threshold, only the agent-generated experience is stored in the experience buffer, and training is performed by sampling experience from the experience buffer. If the experience buffer contains both human expert and agent-generated experience, sampling priorities are assigned to the human expert and agent-generated experience based on the collision risk, and training is performed by sampling experience from the experience buffer according to these priorities. Based on the acquired control strategy, the target vehicle is controlled to make a left turn at an unsignalized intersection.

2. The method for controlling left turns of vehicles at unsignalized intersections according to claim 1, characterized in that, The reward functions of the left-turn decision model include: a task completion reward function that rewards the target vehicle after it successfully makes a left turn; a collision penalty function that penalizes the target vehicle when it is involved in an accident; an efficiency reward function that rewards the target vehicle based on its successful and rapid speed; a risk penalty function that penalizes the target vehicle based on the collision risk with oncoming vehicles; an idling penalty function that penalizes the target vehicle for slow-speed behavior or waiting behavior; and a human intervention penalty function that penalizes the human expert when the human expert begins to generate experience.

3. The method for controlling left turns of vehicles at unsignalized intersections according to claim 1 or 2, characterized in that, The control strategy includes the longitudinal pedal opening and lateral steering angle of the target vehicle.

4. A vehicle left-turn control system for unsignalized intersections, characterized in that, The system includes: The data acquisition module is used to acquire a bird's-eye view semantic map of an unsignalized intersection, which includes road information and vehicle information of the unsignalized intersection. The risk detection module inputs the bird's-eye view semantic map into a left-turn decision model trained based on a reinforcement learning algorithm. The left-turn decision model, based on road and vehicle information, obtains the projection of the relative speed of the target vehicle making a left turn at an unsignalized intersection onto the relative distance between the target vehicle and a vehicle going straight, as well as the relative acceleration of the target vehicle and the vehicle going straight to avoid collision, thus determining the collision risk between the target vehicle and the vehicle going straight. The strategy acquisition module obtains the control strategy for the target vehicle making a left turn at an unsignalized intersection based on road and vehicle information and the collision risk between the target vehicle and the vehicle going straight. During training, if the collision risk is determined to be greater than a preset threshold, the left-turn decision model generates experience through an agent that takes over the reinforcement learning algorithm from a human expert. This experience is stored in an experience buffer along with the agent-generated experience. If the collision risk is less than or equal to the preset threshold, only the agent-generated experience is stored in the experience buffer, and training is performed by sampling experience from the experience buffer. If the experience buffer contains both human expert and agent-generated experience, sampling priorities are assigned to the human expert and agent-generated experience based on the collision risk, and training is performed by sampling experience from the experience buffer according to these priorities. The vehicle control module is used to control the target vehicle to make a left turn at an unsignalized intersection based on the acquired control strategy.

5. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the unsignaled intersection vehicle left-turn control method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for controlling left turns of vehicles at unsignalized intersections as described in any one of claims 1 to 3.

Images