A method for perception and decision of NPC vehicles in autonomous driving simulation test

Abstract

The application provides a perception and decision method for NPC vehicles in automatic driving simulation test, first proposes a direct perception model for directly obtaining driving decision related indexes from a perception image; then connects the direct perception model to the NPC vehicle, so that the NPC vehicle can obtain important traffic information directly related to decision based on the direct perception model; then trains a decision model of the NPC vehicle based on reinforcement learning, and designs a reward function from four aspects of safety, efficiency, comfort and interference to the test vehicle; finally, the NPC vehicle forms a traffic flow, and the direct perception model and the trained decision model are connected, so that the NPC vehicle can interact around the test vehicle to continuously and quickly generate variable scenes to test the automatic driving capability of the host vehicle. This method can make the NPC vehicle imitate the driving behavior of the automatic driving vehicle, and at the same time, generate various dangerous behaviors to the test vehicle, so that the simulation test is more perfect and efficient.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving simulation testing, and specifically relates to a perception and decision-making method for NPC vehicles in autonomous driving simulation testing. Background Technology

[0002] The ability to simulate real-world traffic scenarios is a significant milestone in achieving safe and scalable autonomous driving. Autonomous driving algorithms require extensive testing before being deployed on public roads. However, due to limitations such as the difficulty, high cost, and high risks associated with setting up real-world scenarios for autonomous driving testing, simulation testing has gradually become a viable approach. It allows us to build rich virtual environments where we can improve autonomous vehicles and verify their safety and performance. To efficiently test autonomous driving algorithms, various risky scenarios are often constructed. How to construct these risky scenarios more efficiently remains a challenge. Summary of the Invention

[0003] To address the aforementioned issues, this invention discloses a perception and decision-making method for NPC vehicles in autonomous driving simulation testing. It uses concise, direct, yet meaningful availability representations as the perception output, and trains a driving decision-making model accordingly. By integrating the perception and decision-making model into the NPC vehicle to achieve intelligent traffic flow, and generating complex and varied driving behavior interactions around the test vehicle, rich scenarios can be rapidly generated for testing.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows:

[0005] A perception and decision-making method for NPC vehicles in autonomous driving simulation testing includes the following:

[0006] Deep learning ConvNets are used as direct perception models to map images to heuristic metrics. This invention focuses primarily on multi-lane road driving. From an egocentric perspective, the vehicle only needs to consider traffic in its current lane and the two adjacent (left / right) lanes when making decisions. Therefore, only these three lanes need to be modeled. A single ConvNet is trained to handle three lane configurations together: one-lane, two-lane, or three-lane roads. Sometimes, cars must drive along lane markings; in this case, only the lanes on either side of the lane markings need to be monitored.

[0007] To achieve the two main functions of lane perception and vehicle perception, this invention proposes three types of indicators to represent driving conditions: surrounding vehicle information, road information, and vehicle information. In summary, this invention proposes multiple availability indicators as representations of driving scenarios. These are the outputs of ConvNet, used as perception indicator estimates.

[0008] The direct perception model employs three types of availability indicators: surrounding vehicle information, road information, and vehicle information, to achieve key functions such as lane perception and vehicle perception. The surrounding vehicle information includes the coordinates (x, y) of vehicles in the front, rear, left front, left rear, right front, and right rear positions. i y i ), vehicle orientation α i and vehicle speed v i Road information includes road width w, number of lanes m, current lane (lane), road heading angle θ, and maximum speed limit v. maxL and minimum speed limit V minL The vehicle information includes the vehicle heading angle φ, the vehicle speed v0, and the vehicle position coordinates (x0, y0). The longitudinal distances d1, d2, d3, d4, d5, and d6 between the vehicle and the vehicle in the front, rear, left front, left rear, right front, and right rear positions are calculated from the coordinates.

[0009] The direct perception model is integrated into the NPC vehicle to train its decision-making process, which is based on Markov processes. A Markov process is a memoryless stochastic process, using tuples...<S,P> Let S be a finite set of states, and P be the state transition matrix.

[0010]

[0011] Where n is the number of states, the sum of elements in each row of the matrix is ​​1, and the state transition probabilities are:

[0012] P ss ′=P[S t+1 =s′|S t =s]

[0013] Adding a reward function R to the Markov process s And the decay coefficient γ, to train the decision model. Using a quadrupole array.<S,P,R,γ> The reward refers to the expected value of the reward obtained when the state transitions from state S at time t to state S+1 at time t+1.

[0014] R s =E[R t+1 |S t =s]=∑ s′，r P(s′,r|s)r(s)

[0015] Where the decay coefficient γ∈(0,1], the sum of all rewards decayed from time t onwards is denoted as the reward:

[0016]

[0017] The decay coefficient reflects the impact of future rewards on returns; γ approaching 0 indicates a focus on immediate gains, while γ approaching 1 indicates a preference for long-term benefits. The expected return under a given state is v(s) = E[G]. t |S t =s] is defined as the state value, used to estimate the worth of that state. After a state transition, the reward value can be decomposed into the sum of the reward value and the value of the next state:

[0018] G t =R t+1 +γv(S t+1 )

[0019] Define a finite set of actions A, consisting of tuples<S,A,P,R,γ> Description, where P and R are both related to behavior:

[0020]

[0021]

[0022] A set of behaviors (discrete behaviors) or a distribution of behaviors (continuous behaviors) is defined as a policy, which is the probability π(a|s) that behavior a can be taken in a given state s is given by P[A]. t =a|S t =s], the Markov reward process for a given policy is as follows: <S,p π ,R π ,γ>. The state transition matrix and immediate reward satisfy the formula:

[0023]

[0024]

[0025] When executing a policy, the probability of transitioning from state s to s′ is equal to the sum of a series of probabilities. This series of probabilities refers to the product of the probability of performing a specific action when executing the current policy and the probability that the action will transition the state from s to s′. In the current state s, the immediate reward obtained by executing the specified policy is the sum of the products of the rewards obtained by all possible actions under that policy and the probability of that action. A policy describes the probability of performing different actions in different states. Action value represents the expected gain obtained by performing a specific agent action a in the current state s when executing policy π. That is, when following the current policy π, the value of performing action a in the current state is measured: Action value:

[0026] q π (s, a) = E π [G t |S t =s,A t =a]

[0027] Similar to state value, behavioral value can be expanded into the sum of the immediate reward and the value of performing a certain behavior in the next state. Furthermore, based on the definition of expectation, the expectation of the behavioral value function can be expressed as a function of a random variable.

[0028]

[0029] Transformed into Bellman equations:

[0030]

[0031] From the behavior value functions generated by all strategies, select the state-behavior pairs.<s,a> The most valuable function q * =maxqπ(s, a). The optimal value function clarifies the possible performance of all strategies and derives the optimal value in each state during the decision-making process. At this point, the decision-making process is considered solved. For any decision-making process, there always exists a deterministic optimal strategy. This is the solution approach based on value-based reinforcement learning.

[0032] The state design scheme of this invention considers the state of the controlled vehicle and six obstacle vehicles around it, which are within the controlled vehicle's perception range (approximately 150m). If no obstacle vehicle meets the requirements, zeros are added. The vehicle has seven state variables, including its presence marker, longitudinal and lateral positions, velocity, and acceleration. The presence of a vehicle is represented by 1, otherwise by 0. The other variables are continuous and are relative values ​​to the controlled vehicle. Therefore, the state space consists of a total of 7×7 states.

[0033] The behavior space scheme of this invention is designed as a discrete space, comprising five actions:

[0034] a∈{"Turn left", "Turn right", "Keep straight", "Accelerate", "Decelerate"}

[0035] This invention comprehensively considers the safety, efficiency, comfort, and interference of lane-changing strategies, and designs a reward function for vehicle reinforcement learning:

[0036] r total =αr cl +βr ef +γr cf +δr el

[0037] Safety: This invention uses a portion of the data output by the direct perception model as a safety standard for penalizing vehicles. For example, when the action is a right turn, the minimum distance D is... min =min{d1, d2, d5, d6}, when D minIf the value is less than the preset value, penalties will be imposed according to the following rules, and the vehicle's driving angle will be constrained to prevent it from deviating from the road angle too much:

[0038]

[0039] Efficiency: This invention evaluates efficiency based on vehicle speed and distance traveled. If the vehicle's speed is lower than the minimum speed of surrounding vehicles, it is penalized. Conversely, the vehicle receives a reward related to the square of its speed and distance traveled. The faster the speed and the longer the distance traveled, the greater the reward.

[0040]

[0041] Comfort: Frequent acceleration (deceleration) and lane changes can affect passenger comfort, especially at higher speeds. Therefore, this invention designs a comfort reward function to penalize improper driving behavior:

[0042]

[0043] Where a0 is the vehicle's acceleration, which can be obtained from the speed and time step output by the direct sensing model.

[0044] Interference capability: The ability to interfere with the test vehicle is mainly designed through lane-changing frequency, collision time, and whether a collision occurs. The aim is to increase the risk and randomness of the scenario to improve testing efficiency.

[0045]

[0046]

[0047] Where TTC is the minimum collision time with surrounding vehicles, which can be obtained from the distance between vehicles and the speed; cnt is the number of lane changes within one cycle; and Step... max This represents the maximum time step within a single period.

[0048] The total reward function consists of four parts, taking into account factors such as safety, efficiency, comfort, and interference. Based on the range of the reward function, the reward value is mapped to a value between 0 and 1 to facilitate convergence in reinforcement learning training.

[0049] Finally, the direct perception model and the trained decision reinforcement model are connected to the NPC vehicle, allowing it to interact around the test vehicle. This can continuously generate varied scenarios with random risks, greatly improving the efficiency of autonomous driving simulation testing.

[0050] The beneficial effects of this invention are:

[0051] This invention discloses a perception and decision-making method for NPC vehicles in autonomous driving simulation testing. This method allows NPC vehicles to autonomously interfere with and influence the driving of the test vehicle, continuously and rapidly generating dangerous scenarios to improve testing efficiency. The NPC vehicle's decision-making is based on a direct perception model, which lies between indirect perception and behavior mapping. It neither analyzes the entire scene nor directly maps driving behavior without analysis. Instead, it learns the mapping of several meaningful availability indicators of road and traffic conditions from images, using this concise, direct, and meaningful availability representation as the perception output. Simultaneously, a reward function is designed based on this to train the decision-making model, enabling the vehicle to autonomously interfere with and test surrounding vehicles while ensuring its own safety, efficiency, and comfort. By integrating this model into NPC vehicles to achieve intelligent traffic flow, NPC vehicles can generate complex and varied driving behavior interactions around the test vehicle, rapidly generating rich scenarios to test the test vehicle's perception and decision-making capabilities. Attached Figure Description

[0052] Figure 1 This is a logical diagram illustrating a perception and decision-making method for NPC vehicles in an autonomous driving simulation test, as provided in an embodiment of this application.

[0053] Figure 2 This application provides a direct perception model index diagram for an NPC vehicle's perception and decision-making method in an autonomous driving simulation test.

[0054] Figure 3 This diagram illustrates the decision model training process for an NPC vehicle's perception and decision-making method in an autonomous driving simulation test, as provided in an embodiment of this application. Detailed Implementation

[0055] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] As shown in the figure, the perception and decision-making method for NPC vehicles in autonomous driving simulation testing, as described in this invention, uses a deep learning ConvNet as a direct perception model to map images to heuristic indicators. This invention primarily focuses on multi-lane road driving. From an egocentric perspective, the main vehicle only needs to consider traffic in its current lane and two adjacent (left / right) lanes when making decisions. Therefore, only these three lanes need to be modeled. A single ConvNet is trained to handle three lane configurations together: one-lane, two-lane, or three-lane roads. Sometimes, the car must drive along lane markings; in this case, only the lanes on either side of the lane markings need to be monitored.

[0057] To achieve the two main functions of lane perception and vehicle perception, this invention proposes three types of indicators to represent driving conditions: surrounding vehicle information, road information, and vehicle information. In summary, this invention proposes multiple availability indicators as representations of driving scenarios. These are the outputs of ConvNet, used as perception indicator estimates.

[0058] The direct perception model employs three types of availability indicators: surrounding vehicle information, road information, and vehicle information, to achieve key functions such as lane perception and vehicle perception. The surrounding vehicle information includes the coordinates (x, y) of vehicles in the front, rear, left front, left rear, right front, and right rear positions. i y i ), vehicle heading a i and vehicle speed v i Road information includes road width w, number of lanes m, current lane (lane), road heading angle θ, and maximum speed limit v. maxL and minimum speed limit V minL The vehicle information includes the vehicle heading angle φ, the vehicle speed v0, and the vehicle position coordinates (x0, y0). The longitudinal distances d1, d2, d3, d4, d5, and d6 between the vehicle and the vehicle in the front, rear, left front, left rear, right front, and right rear positions are calculated from the coordinates.

[0059] The direct perception model is integrated into the NPC vehicle to train its decision-making process, which is based on Markov processes. A Markov process is a memoryless stochastic process, using tuples...<S,P> Let S be a finite set of states, and P be the state transition matrix.

[0060]

[0061] Where n is the number of states, the sum of elements in each row of the matrix is ​​1, and the state transition probabilities are:

[0062] P ss ′=[S t+1 =s′|S t =s]

[0063] Adding a reward function R to the Markov process s And the decay coefficient γ, to train the decision model. Using a quadrupole array.<S,P,R,γ> The reward refers to the expected value of the reward obtained when the state transitions from state S at time t to state S+1 at time t+1.

[0064] R s =E[R t+1 |S t =s]=∑ s′，r P(s′,r|s)r(s)

[0065] Where the decay coefficient γ∈(0,1], the sum of all rewards decayed from time t onwards is denoted as the reward:

[0066]

[0067] The decay coefficient reflects the impact of future rewards on returns; γ approaching 0 indicates a focus on immediate gains, while γ approaching 1 indicates a preference for long-term benefits. The expected return under a given state is v(s) = E[G]. t |S t =s] is defined as the state value, used to estimate the worth of that state. After a state transition, the reward value can be decomposed into the sum of the reward value and the value of the next state:

[0068] G t =R t+1 +γv(S t+1 )

[0069] Define a finite set of actions A, consisting of tuples<S,A,P,R,γ> Description, where P and R are both related to behavior:

[0070]

[0071]

[0072] A set of behaviors (discrete behaviors) or a distribution of behaviors (continuous behaviors) is defined as a policy, which is the probability π(a|s) that behavior a can be taken in a given state s is given by P[A]. t =a|S t =s], the Markov reward process for a given policy is as follows: <S,P π ,R π ,γ>. The state transition matrix and immediate reward satisfy the formula:

[0073]

[0074]

[0075] When executing a policy, the probability of transitioning from state s to s′ is equal to the sum of a series of probabilities. This series of probabilities refers to the product of the probability of performing a specific action when executing the current policy and the probability that the action will transition the state from s to s′. In the current state s, the immediate reward obtained by executing the specified policy is the sum of the products of the rewards obtained by all possible actions under that policy and the probability of that action. A policy describes the probability of performing different actions in different states. Action value represents the expected gain obtained by performing a specific agent action a in the current state s when executing policy π. That is, when following the current policy π, the value of performing action a in the current state is measured: Action value:

[0076] q π (s, a) = E π [G t |S t =s,A t =a]

[0077] Similar to state value, behavioral value can be expanded into the sum of the immediate reward and the value of performing a certain behavior in the next state. Furthermore, based on the definition of expectation, the expectation of the behavioral value function can be expressed as a function of a random variable.

[0078]

[0079] Transformed into Bellman equations:

[0080]

[0081] From the behavior value functions generated by all strategies, select the state-behavior pairs.<s,a> The most valuable function q * =maxqπ(s, a). The optimal value function clarifies the possible performance of all strategies and derives the optimal value in each state during the decision-making process. At this point, the decision-making process is considered solved. For any decision-making process, there always exists a deterministic optimal strategy. This is the solution approach based on value-based reinforcement learning.

[0082] The state design scheme of this invention considers the state of the controlled vehicle and six obstacle vehicles around it, which are within the controlled vehicle's perception range (approximately 150m). If no obstacle vehicle meets the requirements, zeros are added. The vehicle has seven state variables, including its presence marker, longitudinal and lateral positions, velocity, and acceleration. The presence of a vehicle is represented by 1, otherwise by 0. The other variables are continuous and are relative values ​​to the controlled vehicle. Therefore, the state space consists of a total of 7×7 states.

[0083] The behavior space scheme of this invention is designed as a discrete space, comprising five actions:

[0084] a∈{"Turn left", "Turn right", "Keep straight", "Accelerate", "Decelerate"}

[0085] This invention comprehensively considers the safety, efficiency, comfort, and interference of lane-changing strategies, and designs a reward function for vehicle reinforcement learning:

[0086] r total =αr cl +βr ef +γr cf +δr el

[0087] Safety: This invention uses a portion of the data output by the direct perception model as a safety standard for penalizing vehicles. For example, when the action is a right turn, the minimum distance D is... min =min{d1, d2, d5, d6}, when D min If the value is less than the preset value, penalties will be imposed according to the following rules, and the vehicle's driving angle will be constrained to prevent it from deviating from the road angle too much:

[0088]

[0089] Efficiency: This invention evaluates efficiency based on vehicle speed and distance traveled. If the vehicle's speed is lower than the minimum speed of surrounding vehicles, it is penalized. Conversely, the vehicle receives a reward related to the square of its speed and distance traveled. The faster the speed and the longer the distance traveled, the greater the reward.

[0090]

[0091] Comfort: Frequent acceleration (deceleration) and lane changes can affect passenger comfort, especially at higher speeds. Therefore, this invention designs a comfort reward function to penalize improper driving behavior:

[0092]

[0093] Where a0 is the vehicle's acceleration, which can be obtained from the speed and time step output by the direct sensing model.

[0094] Interference capability: The ability to interfere with the test vehicle is mainly designed through lane-changing frequency, collision time, and whether a collision occurs. The aim is to increase the risk and randomness of the scenario to improve testing efficiency.

[0095]

[0096]

[0097] Where TTC is the minimum collision time with surrounding vehicles, which can be obtained from the distance between vehicles and the speed; cnt is the number of lane changes within one cycle; and Step... max This represents the maximum time step within a single period.

[0098] The total reward function consists of four parts, taking into account factors such as safety, efficiency, comfort, and interference. Based on the range of the reward function, the reward value is mapped to a value between 0 and 1 to facilitate convergence in reinforcement learning training.

[0099] Finally, the direct perception model and the trained decision reinforcement model are connected to the NPC vehicle, allowing it to interact around the test vehicle. This can continuously generate varied scenarios with random risks, greatly improving the efficiency of autonomous driving simulation testing.

[0100] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A perception and decision-making method for NPC vehicles in autonomous driving simulation testing, characterized in that, A perception and decision-making model is proposed to empower NPC vehicles in autonomous driving simulation test scenarios, enabling NPC vehicles to continuously and rapidly generate changing scenarios around the test vehicle. The decision-making model, the reinforcement learning model training process, is based on the output of the perception model and Markov rewards. The state design scheme considers each NPC car and its six surrounding cars. Each car has seven state variables, including its presence marker, longitudinal and lateral positions, velocity, and acceleration. The action space is designed as discrete, consisting of five actions: ； The reinforcement learning model described above, based on the output metrics of the perception model, designs a reward function for NPC vehicle decision-making from four perspectives: safety, efficiency, comfort, and interference with other vehicles. ； The reward value is then mapped to a range of 0-1 to facilitate convergence in reinforcement learning training. The design steps are as follows: S1. Rewards and penalties for designing safety based on the angle at which a vehicle deviates from the road and the safe distance from surrounding vehicles: ； in The minimum distance between the vehicle and surrounding vehicles participating in the behavioral interaction; S2. Rewards and penalties for design efficiency based on vehicle relative speed: ； S3. The comfort level is designed based on changes in vehicle speed: ； in The acceleration of this vehicle is obtained based on the speed output by the perception model and the time step. S4. Consider its ability to interfere with surrounding vehicles and design the reward function based on lane-changing frequency, collision time, and whether a collision occurs: ； ； in, The minimum collision time with surrounding vehicles is obtained by considering distance and speed, where cnt represents the number of lane changes per cycle. The maximum time step for one cycle.

2. The perception and decision-making method for NPC vehicles in autonomous driving simulation testing as described in claim 1, characterized in that, The perception model employs three types of availability indicators: surrounding vehicle information, road information, and vehicle information, to achieve lane perception and vehicle perception functions. The surrounding vehicle information includes the coordinates of vehicles directly in front, directly behind, to the left front, to the left rear, to the right front, and to the right rear. Vehicle orientation and vehicle speed Road information includes road width. Number of lane lines Current lane Road heading angle Maximum speed limit on roads and minimum speed limit This vehicle information includes the vehicle's heading angle. Vehicle speed The vehicle's location coordinates .

3. The perception and decision-making method for NPC vehicles in autonomous driving simulation testing as described in claim 1, characterized in that, The perception model and the decision-making model trained based on reinforcement learning are connected to the NPC vehicle, which interacts with the test vehicle to continuously generate varied scenarios with random risks.

Images