Implicit state representation method based on signal-free intersection automatic driving system

Abstract

The invention relates to an implicit state representation method based on a signal-free intersection automatic driving system, and aims to simplify state space representation and improve system training efficiency and robustness. According to the method, the lane keeping sign and the target speed are introduced as implicit state variables to represent the environment information irrelevant to the vehicle, so that the state space dimension is reduced, and the calculation complexity brought by considering too many surrounding traffic participants and complex environment information is avoided. The target speed is set according to traffic rules and road conditions in the absence of surrounding traffic participants, and the target speed is dynamically adjusted according to the relative position and speed of surrounding vehicles in the presence of surrounding traffic participants. The lane keeping sign judges whether lane changing or lane keeping is needed or not according to the current position of the vehicle and the road information. According to the method, the state space is effectively simplified, the reinforcement learning training efficiency is improved, and automatic driving tasks in complex environments such as non-signal intersections can be better coped with.

Description

Technical Field

[0001] The present invention proposes an implicit state representation method based on an autonomous driving system at a signal-free intersection, and relates to the technical field of simplified strategy network input for autonomous driving vehicles based on deep reinforcement learning. Background Art

[0002] Autonomous driving technology faces many challenges when dealing with complex traffic environments such as unsignalized intersections, and requires efficient decision-making to ensure driving safety and stability. Traditional methods usually rely on vehicle dynamic information and surrounding traffic conditions to make decisions. Although many factors are taken into account, as the complexity of the environment increases, the state space dimension continues to expand, resulting in large consumption of computing resources, which in turn affects training efficiency and response speed. To solve this problem, the present invention proposes an implicit state representation method based on an unsignalized intersection autonomous driving system. By introducing discrete lane keeping signs and continuous target speeds, the state information of surrounding traffic participants and road environment is implicit, reducing direct dependence on the state of complex traffic participants. Compared with traditional methods, the present invention significantly reduces the state space dimension while ensuring the decision-making effect, simplifies the policy network input, and improves computing efficiency and training speed. This method is particularly suitable for complex traffic scenarios such as unsignalized intersections, and is of great significance for promoting the application and development of autonomous driving technology. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes an automatic driving system method based on implicit state representation, which reduces the dimension of the state space by converting the vehicle's state information and the complex information of the surrounding environment into a simplified implicit state representation, thereby simplifying the input of the policy network and improving the calculation and training efficiency. Especially in a changeable and uncertain traffic environment, the implicit state representation method demonstrates its high robustness and superiority.

[0004] According to the present invention, an implicit state representation method based on an automatic driving system at an unsignalized intersection includes the following specific related steps:

[0005] Step S1: define the implicit state space;

[0006] Step S2: Calculate two variable parameters of implicit state representation: lane keeping sign and target speed;

[0007] Step S3: Optimize the driving strategy based on the implicit state representation and design the reward function.

[0008] Step S1 specifically includes the following contents:

[0009] Step S11: A series of state variables are defined to describe the state of the autonomous vehicle itself, the road environment, and the information of surrounding traffic participants. The state variables are divided into three different state spaces: the vehicle state space , environment state space and the implicit state representation space ;

[0010] Step S12: Represents the state space of the vehicle, which is used to describe the state information related to the vehicle itself, including speed, position and attitude parameters, that is, the longitudinal speed of the vehicle and lateral speed , the vehicle's yaw angle , the slip rate of the four tires in front, behind, left and right ;

[0011] Step S13: Represents the environment state space, which is used to describe the parameters contained in the surrounding traffic participants and road environment state space, including lane markings Identifier indicating lane information, lateral offset Indicates the vertical distance from the vehicle to the lane centerline in the horizontal direction and the lateral offset change rate Indicates the speed at which the vehicle deviates from the lane;

[0012] Step S14: Represents the implicit state representation space, including 2 implicit state variable parameters, lane keeping sign and target speed ,in Used to implicitly indicate whether the vehicle should maintain the current lane or perform lane change during autonomous driving. Used to implicitly represent environmental information related to the vehicle's position, surrounding traffic participants, and road environment;

[0013] Step S15: Represents the implicit state space, which is used to calculate the implicit state variables: and , describes the relative position and speed of the vehicle and the surrounding traffic participants, including: the longitudinal distances between the vehicle and the front and rear vehicles in the same lane and the left and right lanes are and the corresponding speeds are , the distance between the vehicle and the nearest traffic participant in the unsignalized intersection and the corresponding speed , the distance between the entrance lane and the pedestrian crossing of an unsignalized intersection , longitudinal safety distance , lane width , the lateral distance from the vehicle to the center line of the target lane .

[0014] Step S2 specifically includes the following contents:

[0015] Step S21: Implicit state representation by and To determine the behavior of the vehicle, is a discrete value, which takes the value of 0 or 1, representing the following two situations: When , it means the vehicle stays in the current lane. When , it means the vehicle is changing lanes. It is a continuous value, which is set based on different scenarios and conditions of the vehicle in the unsignalized intersection autonomous driving task;

[0016] Step S22: Scene without surrounding traffic participants, The changes are as follows: When the vehicle's lateral displacement , in meters, yaw angle , and the lateral distance from the vehicle to the center line of the target lane is greater than half the lane width, that is, ,at this time , indicating that the vehicle is changing lanes. When the vehicle crosses the boundary of the current lane, the lane change is completed. Until the next lane change task begins, when in an unsignaled intersection, the vehicle needs to follow the desired path generated by the Bezier curve to enter the left lane. , indicating that the vehicle follows the target path and does not change lanes;

[0017] Step S23: Scene without surrounding traffic participants, It is mainly determined by factors such as road environment, lane position and location of unsignaled intersections. The calculation rules are as follows: On the straight lane, the vehicle in the outermost lane 10m / s, vehicles in other lanes 15m / s. In order to ensure safety at non-signaled intersections, vehicles It will drop to 10m / s when the vehicle is far from the intersection without signal. , in meters, at this time is a linear function, that is ;

[0018] Step S24: Scene with surrounding traffic participants, The changes of are as follows: When the longitudinal relative distance between the vehicle and the vehicle in front of the same lane is greater than the longitudinal safety distance, that is, , and the vehicle's current speed is less than the target speed, that is ,at this time , indicating that the vehicle is changing lanes. When the vehicle is in the left turn lane or at an intersection without a signal, , indicating that the vehicle must stay in the current lane and drive along the predetermined route;

[0019] Step S25: Scene with surrounding traffic participants, It is determined by the relative distance between the vehicle and surrounding traffic participants, speed, and traffic rules. The calculation rule of is as follows: When the longitudinal relative distance between the vehicle and the vehicle in front of it in the same lane is greater than the safety distance, , ,in , when the longitudinal relative distance between the vehicle and the vehicle behind in the left lane is less than the safety distance, that is , , when the vehicle is in the left turn lane, , when the vehicle is in an intersection without a signal, .

[0020] Step S3 specifically includes the following contents:

[0021] Step S31: Design a state-dependent reward function , the reward function consists of five parts: speed reward , Lane Keeping Reward , Lane Change Reward , Yaw Angle Reward and turn reward ;

[0022] Step S32: is with and The relevant penalties, By measuring and The gap between the two encourages the vehicle to stay within the desired speed range. The expression is as follows:

[0023]

[0024] in, is the current longitudinal velocity of the vehicle, is the absolute difference between the two, when The closer , the larger the reward value, the closer the reward is to 0, indicating no punishment. Leave The farther away, the smaller the reward value, the larger the absolute difference, and the greater the penalty. The system guides the vehicle to adjust its speed through penalties, thereby reducing speed deviation;

[0025] Step S33: Used to assess whether the vehicle is staying in the center of the lane and prevent the vehicle from deviating from the lane. The expression is as follows:

[0026]

[0027] in, is the lateral offset of the vehicle from the centerline of the lane, is the rate of change of lateral offset, is the lane width, when When , the maximum reward is given , indicating that the vehicle is in the center of the lane. hour, , indicating that the vehicle is reducing the lateral deviation and returning to the center of the lane. When larger, Gradually decreases, and when the reward is negative, the vehicle is encouraged to take immediate measures to reduce its lateral deviation from the lane centerline to correct the lane departure;

[0028] Step S34: Used to encourage vehicles to perform lane change maneuvers when appropriate. The expression is as follows:

[0029]

[0030] when When , the maximum reward is given , indicating that the vehicle is changing lanes. When it is larger, according to Changes according to the changes in the lane, encouraging vehicles to complete lane changes quickly and accurately;

[0031] Step S35: Used to evaluate the vehicle's steering stability, The expression is as follows:

[0032]

[0033] in, is the vehicle’s yaw angle, which indicates the vehicle’s steering angle;

[0034] Step S36: Turn reward for the straight lane Turn rewards at intersections without signals The sum of In a straight lane, in order to ensure driving comfort and in line with actual driving habits, the vehicle should maintain a small steering wheel angle. , at an unsignaled intersection, executing a left turn requires the vehicle to reach a yaw angle of To complete the turn smoothly, and The expression is as follows:

[0035]

[0036]

[0037] The total state-dependent reward function All sub-rewards , , , , The sum of is expressed as follows:

[0038]

[0039] Among them, the coefficient Represents the weight coefficient, which is used to determine the relative weight of each sub-reward item in the total reward calculation;

[0040] Step S37: Initialize a policy network Actor and a value network Critic. The policy network is a deep neural network that can determine the best action based on the current implicit state representation. Through continuous training and learning, it improves and optimizes its decision-making ability in different states, so that the policy network can select actions that can bring higher cumulative rewards. The value network is used to evaluate the long-term expected return of the current strategy. The value network helps adjust the policy network by selecting actions that may lead to higher cumulative rewards.

[0041] Step S38: Execute the actions of the policy network in the simulation environment or in the actual vehicle, and collect the implicit state representation and corresponding rewards , , , and , use these data to train and optimize the policy and value networks, use the Advantage Actor-Critic algorithm to train the policy network, and the A2C algorithm uses the advantage function to speed up the convergence of the network so that it can select the optimal action based on the implicit state representation. A2C is a parallel synchronization method based on the advantage function, which is used to evaluate the output of the network and reduce the estimated variance under the policy gradient. The expression of its advantage function is shown in the following formula:

[0042]

[0043] in, It is in strategy Next, status Take action The action value function and state value function Indicated in strategy Next, from the state The expected cumulative return from departure, Indicates from the state The expected cumulative return from departure, is in state Take action After receiving the instant reward, is the discount factor;

[0044] The expression of policy gradient is as follows:

[0045] J ( ?? ?? ) = E ?? ~ ?? ?? [ ?? t = 0 T A ? ( s t , a t ) ] ? ?? log ?? ?? ( a t | s t )

[0046] in, According to the strategy Sampled Trajectory expectations, represents the length of the trajectory, Indicates that in a given state Take action The probability of using the parameter To express the strategy, Indicates in status Take action Strategy The log probability of The gradient of the strategy with respect to the parameter The rate of change, Indicates in status Take action The advantage function depends on the parameters , adjusts the policy by evaluating the larger cumulative reward of the action in the current state.

[0047] The beneficial effects of the present invention are:

[0048] 1. An implicit state representation method based on an unsignalized intersection autonomous driving system is proposed, specifically for scenes with complex state spaces at unsignalized intersections. This method converts the vehicle's position, road environment, and state information of surrounding traffic participants into two variable parameters of the implicit state: a discrete lane keeping sign and a continuous target speed. Such a conversion not only simplifies the dimension of the state space, but also enhances the robustness of the system in a dynamic traffic environment by reducing the direct dependence on the states of complex traffic participants. In addition, the simplified state representation also reduces the computational complexity, thereby improving the training efficiency of the autonomous driving system when dealing with complex traffic scenarios such as unsignalized intersections.

[0049] 2. A state-dependent reward function is proposed, which can guide the vehicle to optimize its actions according to the vehicle's motion state. This reward mechanism not only improves the stability and safety of the autonomous driving system, but also enhances driving efficiency, enabling autonomous driving vehicles to make more accurate and efficient decisions when facing complex traffic challenges. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is the scenario of implicit state representation in the absence of surrounding traffic participants;

[0051] Figure 2 The implicit state representation scenario for a straight lane with surrounding traffic participants;

[0052] Figure 3 This is a scenario of implicit state representation in an unsignalized intersection with surrounding traffic participants;

[0053] Figure 4 is the vehicle's yaw angle when making a steering decision and steering wheel angle Schematic diagram of the relationship between;

[0054] Figure 5 This is a framework diagram of the autonomous driving strategy at unsignalized intersections. DETAILED DESCRIPTION

[0055] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments. It should be understood that the embodiments are merely for explaining the present invention but not for limiting it.

[0056] The specific implementation steps are as follows:

[0057] Step S1: define the implicit state space;

[0058] Step S2: Calculate two variable parameters of implicit state representation: lane keeping sign and target speed;

[0059] Step S3: Optimize the driving strategy based on the implicit state representation and design the reward function.

[0060] In step S1, in order to define the implicit state space, the following steps need to be performed:

[0061] Step S11: A series of state variables are defined to describe the state of the autonomous vehicle itself, the road environment, and the information of surrounding traffic participants. The state variables are divided into three different state spaces: the vehicle state space , environment state space and the implicit state representation space ;

[0062] Step S12: Represents the state space of the vehicle, which is used to describe the state information related to the vehicle itself, including speed, position and attitude parameters, that is, the longitudinal speed of the vehicle and lateral speed , the vehicle's yaw angle , the slip rate of the four tires in front, behind, left and right ;

[0063] Step S13: Represents the environment state space, which is used to describe the parameters contained in the surrounding traffic participants and road environment state space, including lane markings Identifier indicating lane information, lateral offset Indicates the vertical distance from the vehicle to the lane centerline in the horizontal direction and the lateral offset change rate Indicates the speed at which the vehicle deviates from the lane;

[0064] Step S14: Represents the implicit state representation space, including 2 implicit state variable parameters, lane keeping sign and target speed ,in Used to implicitly indicate whether the vehicle should maintain the current lane or perform lane change during autonomous driving. Used to implicitly represent environmental information related to the vehicle's position, surrounding traffic participants, and road environment;

[0065] Step S15: Represents the implicit state space, which is used to calculate the implicit state variables: and , describes the relative position and speed of the vehicle and the surrounding traffic participants, including: the longitudinal distances between the vehicle and the front and rear vehicles in the same lane and the left and right lanes are and the corresponding speed is , the distance between the vehicle and the nearest traffic participant in the unsignalized intersection and the corresponding speed is , the distance between the entrance lane and the pedestrian crossing of an unsignalized intersection , longitudinal safety distance , lane width , the lateral distance from the vehicle to the center line of the target lane .

[0066] In step S2, the lane keeping sign and the target speed are calculated, which specifically includes the following steps:

[0067] Step S21: Implicit state representation by and To determine the behavior of the vehicle, is a discrete value, which takes the value of 0 or 1, representing the following two situations: When , it means the vehicle stays in the current lane. When , it means the vehicle is changing lanes. It is a continuous value, which is set based on different scenarios and conditions of the vehicle in the unsignalized intersection autonomous driving task;

[0068] Step S22: Scene without surrounding traffic participants, The changes are as follows: When the vehicle's lateral displacement , in meters, yaw angle , and the lateral distance from the vehicle to the center line of the target lane is greater than half the lane width, that is, ,at this time , indicating that the vehicle is changing lanes. When the vehicle crosses the boundary of the current lane, the lane change is completed. , until the next lane change task begins. When in an unsignaled intersection, the vehicle needs to follow the expected path generated by the Bezier curve to enter the left lane. , indicating that the vehicle follows the target path and does not change lanes;

[0069] Step S23: Scene without surrounding traffic participants, It is mainly determined by factors such as road environment, lane position and location of unsignaled intersections. The calculation rules are as follows: On the straight lane, the vehicle in the outermost lane 10m / s, vehicles in other lanes 15m / s. In order to ensure safety at non-signaled intersections, vehicles It will drop to 10m / s when the vehicle is far from the intersection without signal. , in meters, such as Figure 1 As shown, at this time is a linear function, that is ;

[0070] Step S24: Scene with surrounding traffic participants, The changes of are as follows: When the longitudinal relative distance between the vehicle and the vehicle in front of the same lane is greater than the longitudinal safety distance, that is, , and the vehicle's current speed is less than the target speed, that is ,at this time , indicating that the vehicle is changing lanes. When the vehicle is in the left turn lane or at an intersection without a signal, , indicating that the vehicle must stay in the current lane and drive along the predetermined route;

[0071] Step S25: Scene with surrounding traffic participants, It is determined by the relative distance between the vehicle and surrounding traffic participants, speed, and traffic rules. The calculation rule of is as follows: When the longitudinal relative distance between the vehicle and the vehicle in front of it in the same lane is greater than the safety distance, , ,in , when the longitudinal relative distance between the vehicle and the vehicle behind in the left lane is less than the safety distance, that is , , when the vehicle is in the left turn lane, ,like Figure 2 As shown, in the straight section, Figure 2 The number 1 in the figure indicates the vehicle in front of the vehicle in the same lane as the vehicle, the number 2 indicates the vehicle in front of the vehicle in the left lane, the number 3 indicates the vehicle in the left lane behind the vehicle, the number 4 indicates the vehicle in front of the vehicle in the right lane, and the number 5 indicates the vehicle in the right lane behind the vehicle. When the vehicle is in an intersection without a signal, ,like Figure 3 As shown, Figure 3 The number 6 in the figure represents the traffic participant closest to the vehicle in the unsignalized intersection.

[0072] In step S3, a state-dependent reward function is designed, which specifically includes the following steps:

[0073] Step S31: Design a state-dependent reward function , the reward function consists of five parts: speed reward , Lane Keeping Reward , Lane Change Reward , Yaw Angle Reward and turn reward ;

[0074] Step S32: is with and The relevant penalties, By measuring and The gap between the two encourages the vehicle to stay within the desired speed range. The expression is as follows:

[0075]

[0076] in, is the current longitudinal velocity of the vehicle, is the absolute difference between the two, when The closer , the larger the reward value, the closer the reward is to 0, indicating no punishment. Leave The farther away, the smaller the reward value, the larger the absolute difference, and the greater the penalty. The system guides the vehicle to adjust its speed through penalties, thereby reducing speed deviation;

[0077] Step S33: Used to assess whether the vehicle is staying in the center of the lane and prevent the vehicle from deviating from the lane. The expression is as follows:

[0078]

[0079] in, is the lateral offset of the vehicle from the centerline of the lane, is the rate of change of lateral offset, is the lane width, when When , the maximum reward is given , indicating that the vehicle is in the center of the lane. hour, , indicating that the vehicle is reducing the lateral deviation and returning to the center of the lane. When larger, Gradually decreases, and when the reward is negative, the vehicle is encouraged to take immediate measures to reduce its lateral deviation from the lane centerline to correct the lane departure;

[0080] Step S34: Used to encourage vehicles to perform lane change maneuvers when appropriate. The expression is as follows:

[0081]

[0082] when When , the maximum reward is given , indicating that the vehicle is changing lanes. When it is larger, according to Changes according to the changes in the lane, encouraging vehicles to complete lane changes quickly and accurately;

[0083] Step S35: Used to evaluate the vehicle's steering stability, The expression is as follows:

[0084]

[0085] in, is the vehicle’s yaw angle, which indicates the vehicle’s steering angle;

[0086] Step S36: Turn reward for the straight lane Turn rewards at intersections without signals The sum of In a straight lane, in order to ensure driving comfort and in line with actual driving habits, the vehicle should maintain a small steering wheel angle. , at an unsignaled intersection, when executing a left turn task, the vehicle is required to reach a yaw angle when passing through the unsignaled intersection. To successfully complete the turn, the vehicle's yaw angle and steering wheel angle The relationship diagram is as follows Figure 4 As shown, and The expressions are as follows:

[0087]

[0088]

[0089] The total state-dependent reward function All sub-rewards , , , , The sum of is expressed as follows:

[0090]

[0091] Among them, the coefficient Represents the weight coefficient, which is used to determine the relative weight of each sub-reward item in the total reward calculation;

[0092] Step S37: Initialize a policy network Actor and a value network Critic. The policy network is a deep neural network that can determine the best action based on the current implicit state representation. Through continuous training and learning, it can improve and optimize its decision-making ability in different states, so that the policy network can select actions that can bring higher cumulative rewards. The value network is used to evaluate the long-term expected return of the current strategy. The value network helps adjust the policy network by selecting actions that may lead to higher cumulative rewards. Figure 5 The figure shows the framework diagram of the autonomous driving strategy for non-signaled intersections, which is used to describe the structure and learning process of the network. Figure 5 The motion input includes four-wheel torque and steering wheel angle , directly acts on the vehicle model, SUMO as a traffic simulation environment, generates state information of other vehicles and traffic participants, and the gradient is passed through the reward function To calculate, used to update the policy network and value network, Represents the state space, including the vehicle state space , environment state space and the implicit state representation space ;

[0093] Step S38: Execute the actions of the policy network in the simulation environment or in the actual vehicle, and collect the implicit state representation and corresponding rewards , , , and , use these data to train and optimize the policy and value networks, use the Advantage Actor-Critic algorithm to train the policy network, and the A2C algorithm uses the advantage function to speed up the convergence of the network so that it can select the optimal action based on the implicit state representation. A2C is a parallel synchronization method based on the advantage function, which is used to evaluate the output of the network and reduce the estimated variance under the policy gradient. The expression of its advantage function is shown in the following formula:

[0094]

[0095] in, It is in strategy Next, status Take action The action value function and state value function Indicated in strategy Next, from the state The expected cumulative return from departure, Indicates from the state The expected cumulative return from departure, is in state Take action After receiving the instant reward, is the discount factor;

[0096] The expression of policy gradient is as follows:

[0097] J ( ?? ?? ) = E ?? ~ ?? ?? [ ?? t = 0 T A ? ( s t , a t ) ] ? ?? log ?? ?? ( a t | s t )

[0098] in, According to the strategy Sampled Trajectory expectations, represents the length of the trajectory, Indicates that in a given state Take action The probability of using the parameter To express the strategy, Indicates in status Take action Strategy The log probability of The gradient of the strategy with respect to the parameter The rate of change, Indicates in status Take action The advantage function depends on the parameters , adjusts the policy by evaluating the larger cumulative reward of the action in the current state.

Claims

1. An implicit state representation method based on an automatic driving system at an unsignalized intersection, characterized in that: The method comprises the following steps: Step S1: define the implicit state space; Step S2: Calculate two variable parameters of implicit state representation: lane keeping sign and target speed; Step S3: Optimize the driving strategy based on the implicit state representation and design the reward function.

2. The implicit state representation method based on an automatic driving system at an unsignalized intersection according to claim 1, characterized in that: In step S1, the state space is defined, and the following processing is required: Step S11: A series of state variables are defined to describe the state of the autonomous vehicle itself, the road environment, and the information of surrounding traffic participants. The state variables are divided into three different state spaces: the vehicle state space , environment state space and the implicit state representation space ; Step S12: Represents the state space of the vehicle, which is used to describe the state information related to the vehicle itself, including speed, position and attitude parameters, that is, the longitudinal speed of the vehicle and lateral speed , the vehicle's yaw angle , the slip rate of the four tires in front, behind, left and right ; Step S13: Represents the environment state space, which is used to describe the parameters contained in the surrounding traffic participants and road environment state space, including lane markings Identifier indicating lane information, lateral offset Indicates the vertical distance from the vehicle to the lane centerline in the horizontal direction and the lateral offset change rate Indicates the speed at which the vehicle deviates from the lane; Step S14: Represents the implicit state representation space, including 2 implicit state variable parameters, lane keeping sign and target speed ,in Used to implicitly indicate whether the vehicle should maintain the current lane or perform lane change during autonomous driving. Used to implicitly represent environmental information related to the vehicle's position, surrounding traffic participants, and road environment; Step S15: Represents the implicit state space, which is used to calculate the implicit state variables: and , describes the relative position and speed of the vehicle and the surrounding traffic participants, including: the longitudinal distances between the vehicle and the front and rear vehicles in the same lane and the left and right lanes are and the corresponding speeds are , the distance between the vehicle and the nearest traffic participant in the unsignalized intersection and the corresponding speed , the distance between the entrance lane and the pedestrian crossing of an unsignalized intersection , longitudinal safety distance , lane width , the lateral distance from the vehicle to the center line of the target lane .

3. The implicit state representation method based on an automatic driving system at an unsignalized intersection according to claim 1, characterized in that: The step S2 calculates and , the specific process is as follows: Step S21: Implicit state representation by and To determine the behavior of the vehicle, is a discrete value, which takes the value of 0 or 1, representing the following two situations: When , it means the vehicle stays in the current lane. When , it means the vehicle is changing lanes. It is a continuous value, which is set based on different scenarios and conditions of the vehicle in the unsignalized intersection autonomous driving task; Step S22: Scene without surrounding traffic participants, The changes are as follows: When the vehicle's lateral displacement , in meters, yaw angle , and the lateral distance from the vehicle to the center line of the target lane is greater than half the lane width, that is, ,at this time , indicating that the vehicle is changing lanes. When the vehicle crosses the boundary of the current lane, the lane change is completed. Until the next lane change task begins, when in an unsignaled intersection, the vehicle needs to follow the desired path generated by the Bezier curve to enter the left lane. , indicating that the vehicle follows the target path and does not change lanes; Step S23: Scene without surrounding traffic participants, It is mainly determined by factors such as road environment, lane position and location of unsignaled intersections. The calculation rules are as follows: On the straight lane, the vehicle in the outermost lane 10m / s, vehicles in other lanes 15m / s. In order to ensure safety at non-signaled intersections, vehicles It will drop to 10m / s when the vehicle is far from the intersection without signal. , in meters, at this time is a linear function, that is ; Step S24: Scene with surrounding traffic participants, The changes of are as follows: When the longitudinal relative distance between the vehicle and the vehicle in front of the same lane is greater than the longitudinal safety distance, that is, , and the vehicle's current speed is less than the target speed, that is ,at this time , indicating that the vehicle is changing lanes. When the vehicle is in the left turn lane or at an intersection without a signal, , indicating that the vehicle must stay in the current lane and drive along the predetermined route; Step S25: Scene with surrounding traffic participants, It is determined by the relative distance between the vehicle and surrounding traffic participants, speed, and traffic rules. The calculation rule of is as follows: When the longitudinal relative distance between the vehicle and the vehicle in front of it in the same lane is greater than the safety distance, , ,in , when the longitudinal relative distance between the vehicle and the vehicle behind in the left lane is less than the safety distance, that is , , when the vehicle is in the left turn lane, , when the vehicle is in an intersection without a signal, .

4. The implicit state representation method based on an automatic driving system at an unsignalized intersection according to claim 1, characterized in that: The step S3 designs a state-dependent reward function And through the reinforcement learning algorithm, the vehicle's lane keeping, lane changing and unprotected left turn behaviors in complex traffic environments are optimized. The specific implementation process is as follows: Step S31: Design a state-dependent reward function , the reward function consists of five parts: Speed ​​Reward , Lane Keeping Reward , Lane Change Reward , Yaw Angle Reward and turn reward ; Step S32: is with and The relevant penalties, By measuring and The gap between the two encourages the vehicle to stay within the desired speed range. The expression is as follows: in, is the current longitudinal velocity of the vehicle, is the absolute difference between the two, when The closer , the larger the reward value, the closer the reward is to 0, indicating no punishment. Leave The farther away, the smaller the reward value, the larger the absolute difference, and the greater the penalty. The system guides the vehicle to adjust its speed through penalties, thereby reducing speed deviation; Step S33: Used to assess whether the vehicle is staying in the center of the lane and prevent the vehicle from deviating from the lane. The expression is as follows: in, is the lateral offset of the vehicle from the centerline of the lane, is the rate of change of lateral offset, is the lane width, when When , the maximum reward is given , indicating that the vehicle is in the center of the lane. hour, , indicating that the vehicle is reducing the lateral deviation and returning to the center of the lane. When larger, Gradually decreases, and when the reward is negative, the vehicle is encouraged to take immediate measures to reduce its lateral deviation from the lane centerline to correct the lane departure; Step S34: Used to encourage vehicles to perform lane change maneuvers when appropriate. The expression is as follows: when When , the maximum reward is given , indicating that the vehicle is changing lanes. When it is larger, according to Changes according to the changes in the lane, encouraging vehicles to complete lane changes quickly and accurately; Step S35: Used to evaluate the vehicle's steering stability, The expression is as follows: in, is the vehicle’s yaw angle, which indicates the vehicle’s steering angle; Step S36: Turn reward for the straight lane Turn rewards at intersections without signals The sum of In a straight lane, in order to ensure driving comfort and in line with actual driving habits, the vehicle should maintain a small steering wheel angle. , at an unsignaled intersection, executing a left turn task requires the vehicle to reach a yaw angle when passing through the unsignaled intersection. To complete the turn smoothly, and The expression is as follows: The total state-dependent reward function All sub-rewards , , , , The sum of is expressed as follows: Among them, the coefficient Represents the weight coefficient, which is used to determine the relative weight of each sub-reward item in the total reward calculation; Step S37: Initialize a policy network Actor and a value network Critic. The policy network is a deep neural network that can determine the best action based on the current implicit state representation. Through continuous training and learning, it improves and optimizes its decision-making ability in different states, so that the policy network can select actions that can bring higher cumulative rewards. The value network is used to evaluate the long-term expected return of the current strategy. The value network helps adjust the policy network by selecting actions that may lead to higher cumulative rewards. Step S38: Execute the actions of the policy network in the simulation environment or in the actual vehicle, and collect the implicit state representation and corresponding rewards , , , and , use these data to train and optimize the policy and value network, use the Advantage Actor-Critic algorithm to train the policy network, the A2C algorithm uses the advantage function to speed up the convergence of the network, so that it can select the optimal action based on the implicit state representation. A2C is a parallel synchronization method based on the advantage function, which is used to evaluate the output of the network and reduce the estimated variance under the policy gradient. The expression of its advantage function is shown in the following formula: in, It is in strategy Next, status Take action The action value function and state value function Indicated in strategy Next, from the state The expected cumulative return from departure, Indicates from the state The expected cumulative return from departure, is in state Take action After receiving the instant reward, is the discount factor; The expression of policy gradient is as follows: in, According to the strategy Sampled trajectories expectations, represents the length of the trajectory, Indicates that in a given state Take action The probability of using the parameter To express the strategy, Indicates in status Take action Strategy The log probability of The gradient of the strategy with respect to the parameter The rate of change, Indicates in status Take action The advantage function depends on the parameters , adjusts the policy by evaluating the larger cumulative reward of the action in the current state.

Images