Multi-dimensional natural driving scene generation and simulation test method based on reinforcement learning

Abstract

The application relates to a multi-dimensional natural driving scene generation and simulation test method based on reinforcement learning, which comprises the following steps: quantifying driving behavior characteristics by combining a fuzzy logic classification model on the basis of dynamics characteristics, rule compliance, psychological characteristics and cooperation modes, and obtaining driving style fuzzy rules; constructing a driving environment and configuring the quantified characteristics of different driving styles according to the driving style fuzzy rules; using a reinforcement learning algorithm, based on the driving environment and the quantified characteristics of different driving styles, taking the driving style fuzzy rules as the target, calculating the rewards of different driving scenes according to a reward function to train and update an agent policy network, and obtaining the trained agent policy network; and the trained agent policy network is used for generating multi-dimensional natural driving scenes and performing simulation tests. The method can generate diversified behavior modes covering from cautious driving to aggressive driving, and obtain more complete and accurate behavior characteristic representations.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to a multi-dimensional natural driving scene generation and simulation test method based on reinforcement learning. BACKGROUND

[0002] In recent years, with the rapid development of intelligent driving technology, real vehicle testing plays a key role in verifying the performance and safety of automatic driving systems. However, real vehicle testing faces major challenges such as high cost, high risk, and complex scenarios that are difficult to fully cover. To address these issues, simulation testing, as an efficient and flexible alternative, is attracting widespread attention from the research and industrial communities.

[0003] Simulation testing can reproduce real driving scenarios through virtual environments, with significant advantages such as low cost, strong controllability, and high repeatability. For example, testing the perception system and decision-making algorithm of an autonomous vehicle in complex traffic scenarios, simulation tools such as Carla and LGSVL Simulator can simulate extreme conditions such as rain and snow weather, low visibility, and high-density traffic, which are difficult to achieve in actual testing. In addition, Waymo's research has shown that through simulation testing, billions of miles of autonomous driving data can be generated every day, and the cost of collecting these data is much lower than real vehicle testing, and it is easier to find boundary problems in long-tail scenarios.

[0004] However, simulation testing is not perfect, and its limitations mainly lie in the deviation between simulation models and the real world. Since the algorithms of simulation tools are based on idealized assumptions, they may not fully reflect the dynamic complexity of real driving, such as the random behavior of traffic participants or the uncertainty of the environment. In addition, the confidence of the simulation system itself depends on the accuracy of the underlying model and the data coverage range. Existing simulation testing methods and scene generation techniques face challenges in practical applications: first, the coverage of real-world complex driving scenarios is insufficient, affecting the simulation reality and universality; second, the modeling of driving behavior characteristics is limited, resulting in a lack of diversity and naturalness in generated behavior. These problems constrain the effectiveness of simulation testing and hinder the development of autonomous driving systems. SUMMARY

[0005] To solve the above problems in the prior art, the application provides a multi-dimensional natural driving scene generation and simulation test method based on reinforcement learning. The technical problems to be solved by the application are solved by the following technical solutions:

[0006] The embodiment of the application provides a multi-dimensional natural driving scene generation and simulation test method based on reinforcement learning, including the following steps:

[0007] S1, based on the kinetic characteristics, rule compliance, psychological characteristics and collaborative ways, the driving behavior characteristics are quantified by combining the fuzzy logic classification model, and the driving style fuzzy rule is constructed;

[0008] S2, the driving environment is constructed, and the quantified characteristics of different driving styles are configured according to the driving style fuzzy rule;

[0009] S3, the reinforcement learning algorithm is used, the driving environment and the quantified characteristics of different driving styles are used as the target of the driving style fuzzy rule, the reward of different driving scenes is calculated according to the reward function to train and update the agent policy network, and the trained agent policy network is obtained; the trained agent policy network is used for generating a multi-dimensional natural driving scene and performing simulation test.

[0010] In an embodiment of the present application, step S1 comprises:

[0011] S11, the vehicle kinematic state index is acquired, and the fuzzy values of different vehicle kinematic state indexes are defined by using fuzzy subsets, so as to obtain vehicle dynamic state fuzzy indexes;

[0012] S12, the traffic compliance state index is acquired, and the fuzzy values of different traffic compliance state indexes are defined by using fuzzy subsets, so as to obtain traffic compliance state fuzzy indexes;

[0013] S13, the risk aversion coefficient, the risk decision threshold and the risk transition coefficient are calculated and weighted to construct a risk preference comprehensive index;

[0014] S14, the collaboration initiative coefficient and the information sharing degree are calculated and weighted to construct a collaboration way comprehensive index;

[0015] S15, the driving style fuzzy rule is constructed according to the vehicle dynamic state fuzzy index, the traffic compliance state fuzzy index, the risk preference comprehensive index and the collaboration way comprehensive index.

[0016] In an embodiment of the present application, the vehicle kinematic state index comprises speed, acceleration, steering angle and yaw rate; wherein the fuzzy value of the speed comprises low speed, medium speed and high speed, the fuzzy value of the acceleration comprises low acceleration, medium acceleration and high acceleration, the fuzzy value of the steering angle comprises low steering angle and high steering angle, and the fuzzy value of the yaw rate comprises low yaw rate and high yaw rate;

[0017] The traffic compliance state indicators include a speed over-speed state, a following distance safety state and a lane keeping specification state, the fuzzy values of the speed over-speed state include speed over-speed and speed not over-speed, the fuzzy values of the following distance safety state include following distance safety and following distance not safety, and the lane keeping specification state includes lane keeping good, lane keeping general and lane keeping poor.

[0018] In one embodiment of the present application, the formula for calculating the risk avoidance coefficient is:

[0019]

[0020] wherein, is a longitudinal acceleration, is a lateral acceleration, is a steering angle, is a speed; is a weight coefficient;

[0021] The formula for calculating the risk decision threshold is:

[0022]

[0023] wherein, are a following distance, a headway time and a collision time, respectively, are weight coefficients;

[0024] The formula for calculating the risk transition coefficient is:

[0025]

[0026] wherein, are a longitudinal acceleration change amount, a lateral acceleration change amount, a speed change amount, a steering angle change amount and a yaw rate change amount, respectively;

[0027] The risk preference comprehensive indicator is:

[0028]

[0029] wherein, is a weight.

[0030] In one embodiment of the present application, the formula for calculating the cooperation initiative coefficient is:

[0031]

[0032] wherein, , are a lateral position and a lateral distance from a neighboring vehicle, respectively, , is a weight coefficient;

[0033] The formula for calculating the degree of information sharing is:

[0034]

[0035] in, For speed, Traffic flow density, , These are the weighting coefficients;

[0036] The comprehensive index of the collaboration method is:

[0037]

[0038] in, , These are the weighting coefficients.

[0039] In one embodiment of the present invention, the driving style fuzzy rules include: very defensive, high-priority defensive, low-priority defensive, high-priority normal movement, low-priority normal movement, very dynamic, high-priority aggressive, and low-priority aggressive.

[0040] In one embodiment of the present invention, step S2 includes:

[0041] A driving environment is constructed in the simulation platform, and quantitative features of different driving styles are configured according to the fuzzy rules of the driving style. The driving environment includes complex road conditions, dynamic traffic flow, and multiple types of vehicle models.

[0042] In one embodiment of the present invention, step S3 includes:

[0043] The vehicle's real-time status information, historical status trajectory, and environmental information are acquired, and the real-time status information, historical status trajectory, and environmental information are encoded using a Transformer encoder to obtain the vehicle status at the current moment.

[0044] Based on the current vehicle state, a driving action is generated using an Actor network, the driving action is executed, the vehicle state at the next moment is obtained, and a multi-objective reward function is used to calculate the reward for the current driving scenario.

[0045] Based on the time difference method, the Critic network is used to evaluate the vehicle state and driving behavior at the current moment; wherein, the Actor network and the Critic network form the agent policy network;

[0046] The Actor network and the Critic network are updated using the MAPPO reinforcement learning algorithm;

[0047] adjusting weights in the multi-objective reward function to optimize the agent policy network.

[0048] In an embodiment of the present application, the real-time state information comprises vehicle speed, acceleration, yaw rate, steering angle, overspeed state, following distance state and lane keeping state; the historical state trajectory comprises state of previous time steps; the environment information comprises road type and traffic flow;

[0049] The driving behavior action comprises continuous output and discrete output, the continuous output comprises acceleration and steering angle, and the discrete output comprises whether to change lane, decelerate or accelerate.

[0050] In an embodiment of the present application, the multi-objective reward function is:

[0051]

[0052] wherein, is a comprehensive dynamics reward, is a comprehensive traffic compliance reward, is a risk preference reward, is a collaborative manner reward, is a driving style matching reward, , is a weight coefficient;

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] wherein, is a speed reward, is an acceleration reward, is a yaw rate reward, is a steering angle reward, , , , is a weight coefficient;

[0059]

[0060]

[0061]

[0062]

[0063] in, For speed overload rewards, As a follow-distance bonus, Lane keeping reward; These are the weighting coefficients;

[0064]

[0065]

[0066] .

[0067] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0068] The method of this invention introduces risk preference and naturalness indicators of cooperation mode. Based on dynamic characteristics, rule compliance, psychological characteristics, and cooperation mode, it constructs fuzzy rules for driving style. Then, it uses the reward function in the reinforcement learning framework to train and update the agent policy network, realizing the dynamic generation of driving behavior models. This method can generate diverse behavior patterns covering cautious to aggressive driving, obtain more complete and accurate behavioral feature representations, and thus generate diverse driving behaviors. It provides scientific and efficient technical support for the research and development, verification, future technology standardization, safety assessment, and large-scale application of autonomous driving systems. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating a method for generating and simulating multidimensional natural driving scenarios based on reinforcement learning, provided in an embodiment of the present invention.

[0070] Figure 2 A schematic diagram of a multi-dimensional feature-based driving style modeling framework provided in an embodiment of the present invention;

[0071] Figure 3 The present invention provides a reinforcement learning-based driving style scene generation and simulation process. Detailed Implementation

[0072] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0073] Example 1

[0074] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for generating and simulating multidimensional natural driving scenarios based on reinforcement learning, provided in an embodiment of the present invention. The method includes the following steps:

[0075] S1, based on the kinetic characteristics, rule compliance, psychological characteristics and collaborative mode, the driving behavior characteristics are quantified by combining the fuzzy logic classification model, and a fuzzy rule of driving style is constructed.

[0076] See Figure 2 , Figure 2 A Multidimensional Metrics for Dynamic Characteristics, Rule Compliance, Risk Appetite and Collaboration (MDRC) driving style modeling framework provided by the embodiment of the application is shown in the figure. Figure 2 The framework covers four key dimensions of kinetic characteristics, rule compliance, psychological characteristics and collaborative mode. Traditional research mainly focuses on kinetic characteristics and rule compliance, such as speed, acceleration and following distance, but these methods fail to fully reflect the internal decision mechanism of the driver and the interactive behavior in complex traffic scenarios. Therefore, the embodiment adds two dimensions of psychological characteristics and collaborative mode to reveal the risk preference and decision tendency of the driver and the interactive willingness with the environment, thereby improving the accurate description of driving behavior and the adaptability to complex scenarios. Kinetic characteristics describe the motion state of the vehicle, rule compliance evaluates the driver's adherence to traffic rules, psychological characteristics reveal the driver's tendency in risk decision-making, and collaborative mode measures the interactive behavior of the driver and the environment. By constructing a multidimensional feature space, the framework can fully capture the complexity of driving behavior and refine the behavior analysis in different driving scenarios. To achieve classification, the embodiment introduces a fuzzy logic method to convert multidimensional features into classification basis for driving style. Fuzzy logic is good at handling uncertainty in complex systems, and has higher flexibility and robustness than hard boundary classification methods, which can accurately describe the diversity and transition of driving behavior and effectively reduce classification errors caused by fuzzy data division. The framework further supports mapping the classification results to a simulation platform to verify and optimize the driving behavior model, providing an important reference for the development of intelligent driving technology.

[0077] Step S1 specifically includes:

[0078] S11, obtain traffic compliance state indicators, and define fuzzy values of different traffic compliance state indicators using fuzzy subsets to obtain traffic compliance state fuzzy indicators.

[0079] Specifically, the kinematic characteristics of the vehicle are captured by key kinetic indicators such as speed, acceleration, steering angle, yaw rate, lateral position and following distance, and the complexity and nonlinearity of the variables are handled by fuzzy subsets to obtain traffic compliance state fuzzy indicators, as shown in Table 1.

[0080] Table 1 Vehicle kinematics state fuzzy index

[0081]

[0082] As shown in Table 1, the vehicle kinematics state index includes speed, acceleration, steering angle and yaw rate; wherein the fuzzy value of the speed includes low speed, medium speed and high speed, the fuzzy value of the acceleration includes low acceleration, medium acceleration and high acceleration, the fuzzy value of the steering angle includes low steering angle and high steering angle, and the fuzzy value of the yaw rate includes low yaw rate and high yaw rate.

[0083] Further, the speed threshold value and interval range is shown in Table 2, the acceleration threshold value and interval range is shown in Table 3, the yaw rate threshold value and interval range is shown in Table 4, and the steering angle threshold value and interval range is shown in Table 5, wherein the acceleration, the steering angle and the yaw rate are absolute values.

[0084] Table 2 Speed threshold value and interval range

[0085]

[0086] Table 3 Acceleration threshold value and interval range

[0087]

[0088] Table 4 Yaw rate threshold value and interval range

[0089]

[0090] Table 5 Steering angle threshold value and interval range

[0091]

[0092] S12, acquiring a traffic compliance state index, and using fuzzy subsets to define fuzzy values of different traffic compliance state indexes to obtain a traffic compliance state fuzzy index.

[0093] Specifically, focusing on whether the speed is overspeed, whether the following distance is safe and whether the lane keeping is standard and other core indexes of the traffic compliance state, and dividing into fuzzy subsets. Through fuzzy logic, accurate evaluation of the compliance state is realized, and the adaptability and robustness of the intelligent transportation system in complex environment are enhanced, and the traffic compliance state fuzzy index obtained by processing is shown in Table 6.

[0094] Table 6 Traffic compliance state fuzzy index

[0095]

[0096] As shown in Table 2, the traffic compliance state indicators include a speed over-speed state, a following distance safety state, and a lane keeping specification state. The fuzzy value of the speed over-speed state includes speed over-speed and speed not over-speed. The fuzzy value of the following distance safety state includes following distance safety and following distance unsafe. The lane keeping specification state includes lane keeping good, lane keeping general, and lane keeping poor.

[0097] Further, when judging the speed over-speed state, the keeping state is recorded once per frame, and the mode of the speed over-speed state in the total frame number is taken as the final state. When the speed is greater than the speed limit by 10%, it is the over-speed standard. The threshold range of the speed over-speed state is shown in Table 7.

[0098] Table 7 Threshold range of whether the speed is over-speed

[0099]

[0100] When judging the following distance safety state, the keeping state is recorded once per frame, and the mode of the following distance in the total frame number is taken as the final state. The threshold range of whether the following distance is safe is shown in Table 8.

[0101] Table 8 Threshold range of whether the following distance is safe

[0102]

[0103] When judging the lane keeping state, the keeping state is recorded once per frame, and the mode of the lane keeping state in the total frame number is taken as the final state. The threshold range of whether the lane keeping is specification is shown in Table 9.

[0104] Table 9 Threshold range of whether the lane keeping is specification

[0105]

[0106] S13, calculate the risk avoidance coefficient, the risk decision threshold, and the risk transition coefficient, and construct a risk preference comprehensive index by weighting.

[0107] To further reveal the inherent decision-making mechanism and risk attitude of the driver, the risk-taking tendency and coping ability of the driver are quantified. On this basis, the risk avoidance coefficient, the decision threshold, and the risk transition coefficient are calculated and weighted to construct a risk preference comprehensive index, so as to improve the accuracy of driving style classification. In dynamic driving behavior, the cooperation mode dimension is introduced to consider the interaction of the driver with the external environment, especially the intelligent transportation system. The comprehensive index is constructed by the cooperation initiative coefficient and the information sharing degree according to a specific weight, to evaluate the cooperation efficiency of the driver in the intelligent transportation system. This makes it possible to improve the adaptability and overall efficiency of driving behavior modeling in the intelligent transportation system, especially in multi-vehicle interaction and complex traffic scenarios.

[0108] Specifically, the risk avoidance coefficient (RAC) is used to measure the sensitivity of the driver to risk, reflecting his risk tolerance. The calculation formula of the risk avoidance coefficient is:

[0109]

[0110] wherein, is the longitudinal acceleration, is the lateral acceleration, is the steering angle, is the speed; is the weight coefficient, which needs to be determined by experimental data fitting.

[0111] The risk decision threshold (RDT) defines the critical point of the driver's risk avoidance behavior, reflecting his decision-making acuity. The calculation formula of the risk decision threshold is:

[0112]

[0113] wherein, is the following distance (reflecting the spatial distance between the vehicle and the front vehicle), the headway time (describing the time required to reach the front vehicle position at the current speed), and the collision time (representing the time when the collision may occur under the current state), are all weight coefficients.

[0114] The risk transition coefficient (RTC) reflects the adjustment rate of the driver's risk preference under different situations. The calculation formula of the risk transition coefficient is:

[0115]

[0116] wherein, is the longitudinal acceleration change, the lateral acceleration change, the speed change, the steering angle change and the yaw rate change

[0117] The risk preference comprehensive index is:

[0118]

[0119] wherein, is the weight.

[0120] S14, calculate the cooperation initiative coefficient and the information sharing degree, and construct the cooperation mode comprehensive index by weighting.

[0121] Specifically, a collaboration proactivity coefficient (CPC) is used to measure the willingness of a driver to actively collaborate in a traffic situation. The calculation formula of the collaboration proactivity coefficient is:

[0122]

[0123] wherein, , is a lateral position (indicating the degree of deviation of the vehicle in the lane), and a lateral distance from a neighboring vehicle, , is a weight coefficient.

[0124] An information sharing degree (ISD) is used to evaluate the communication ability of a driver in an intelligent transportation system.

[0125] The calculation formula of the information sharing degree is:

[0126]

[0127] wherein, is a speed, is a traffic flow density, , is a weight coefficient.

[0128] The collaboration mode comprehensive index is:

[0129]

[0130] wherein, , is a weight coefficient.

[0131] Further, the risk preference comprehensive index and the collaboration mode comprehensive index are 0-1 numerical values, and the threshold ranges of the two are shown in Table 10.

[0132] Table 10 Threshold range of risk preference comprehensive index and collaboration mode

[0133]

[0134] S15, constructing a driving style fuzzy rule according to the vehicle dynamics state fuzzy index, the traffic compliance state fuzzy index, the risk preference comprehensive index and the collaboration mode comprehensive index.

[0135] After completing the fuzzy processing of the indicators and setting the rules, we construct a series of fuzzy rules for the given input parameters based on expert experience. First, we operate on the actual data set by extracting relevant data such as speed, acceleration, following distance, etc. Calculate the fuzzy membership degree and generate the driving style classification results according to the preset fuzzy rules. For each data, the system automatically judges its driving style category according to the rule mapping table, as shown in Table 11, and counts the classification results to verify the overall distribution rationality and classification accuracy of the data set. In Table 11, the risk preference and cooperation mode values are finally normalized to values within 0 to 1, with low being 0.3 or below and extremely low being 0.15 or below; high is 0.8 or above, and extremely high is 0.9 or above; Lane keeping specification is more than 80% during the entire simulation test period, with a deviation of ≥50% from the lane center as frequent and large.

[0136] Table 11 Fuzzy rules of driving style

[0137]

[0138] As shown in Table 10, the fuzzy rules of driving style include: very defensive, high priority defensive, low priority defensive, high priority normal motion, low priority normal motion, very motion, high priority aggressive, and low priority aggressive.

[0139] Further, the classification results are mapped to the simulation platform (such as CARLA, SUMO). Specifically, according to the classification results of different driving styles, the dynamic parameters of the simulation vehicle are set. The simulation platform generates the behavior patterns of the corresponding driving styles by inputting these dynamic parameters. During the simulation process, the actual driving behavior data is used to verify and optimize the classification model, and the dynamic parameters are adjusted to make the simulation results more consistent with the characteristics of actual driving behavior, and finally realize the closed-loop optimization of the classification model and the simulation system, providing support for the development of intelligent transportation systems. Then, use these parameters to generate different styles of driving behavior in the simulation system. Finally, a system framework that can dynamically adjust the driving style is constructed to achieve smooth transition from defensive to aggressive, providing support for subsequent testing and optimization.

[0140] Through the above steps, this embodiment realizes efficient classification and simulation mapping of multi-dimensional driving style based on the comprehensive construction of driving behavior feature system, while effectively reducing the risk of data processing complexity and model adaptability. The fuzzy logic classification and dynamic simulation framework not only accurately reflects the behavior characteristics of different driving styles, but also has the advantages of high robustness and scalability in complex traffic scenarios. This system provides a solid technical foundation and reliable data support for further exploration of driving scene generation and simulation based on reinforcement learning, and creates more possibilities for the optimization design of intelligent driving systems.

[0141] S2, construct a driving environment and configure quantitative characteristics of different driving styles according to the driving style fuzzy rule.

[0142] For example, a diversified driving environment is constructed in the simulation platform SUMO, including complex road conditions (such as multi-lane, highway, intersection), dynamic traffic flow (supporting changes in different traffic density), and multiple types of vehicle models (such as cars, buses and trucks). Then by configuring the SUMO XML file (such as route.xml, network.xml), the quantitative characteristics of different driving styles are determined (such as the aggressive driving of rapid acceleration and rapid braking behavior, and the conservative driving of low-speed uniform driving characteristics). At the same time, multi-dimensional data is collected, including vehicle state (speed, acceleration), driving action (acceleration, steering angle) and environmental characteristics (traffic flow density), to support the initial input of the model.

[0143] S3, using a reinforcement learning algorithm, based on the driving environment and the quantitative characteristics of different driving styles, taking the driving style fuzzy rule as the target, calculating the reward of different driving scenes according to the reward function to train and update the agent policy network, obtaining the trained agent policy network; the trained agent policy network is used to generate multi-dimensional natural driving scenes and perform simulation testing.

[0144] This embodiment introduces the Transformer model into the optimization of multi-agent communication mechanism in view of the challenges in real-time traffic management and adaptability to complex environments. Compared with the low efficiency of traditional sequence processing methods (such as RNN or LSTM) in processing long sequences, the Transformer can process the entire sequence at the same time, greatly improving the computational efficiency. With the help of self-attention mechanism, the agent can accurately filter out the information most relevant to the current decision. To generate scenes that conform to multi-dimensional driving styles, this embodiment constructs the input and output structure of reinforcement learning, defines the state and action space, and combines the reward function to evaluate the quality of driving behavior. Then, the initial scene is generated by training the agent policy, the reward weight is dynamically optimized to ensure the diversity and driving style characteristics of the generated scene. Finally, a closed-loop feedback mechanism is established to analyze and optimize the scenes that do not pass, gradually improving the adaptability and comprehensiveness of scene generation, providing support for intelligent driving system testing. The overall process is shown in Figure 3 Figure 3 The driving style scene generation and simulation process of reinforcement learning provided by the embodiment of the present application.

[0145] ​Specifically, the input-output structure of reinforcement learning is constructed, including state space, action space, and reward function. First, the state space is used to describe the real-time state of the vehicle and the environment, such as vehicle speed, acceleration, front vehicle distance, and other environmental variables. Then, the action space defines the control input of driving behavior, such as speed, acceleration, steering angle, etc. Next, the reward function is designed to evaluate driving behavior and scene quality by combining short-term and long-term rewards. Short-term rewards are used to immediately feedback the compliance and safety of driving behavior, such as obstacle avoidance behavior, rule compliance, etc.; long-term rewards evaluate the overall rationality and complexity of the scene, such as whether the driving behavior generated by multi-dimensional features meets the expected goal. Finally, the input-output structure and reward function are integrated into the reinforcement learning framework, laying the foundation for scenario generation.

[0146] Specifically, the input of the state space includes real-time state information, historical state trajectory, and environmental information. Real-time state information covers vehicle dynamics state (speed, acceleration, yaw rate, steering angle), traffic compliance state (whether speeding, following distance, lane keeping state), which reflects the vehicle's current motion and traffic rule compliance. Historical state trajectory takes the state of previous time steps as additional feature input, providing the agent with information in the time dimension, which helps to analyze the trend of state changes and make more reasonable decisions. Environmental information includes road type, traffic flow, etc., environmental factors have an important influence on vehicle behavior decision, these information expands the state space, so that the agent can make different responses according to different environments. The role of the state space is to integrate the above types of information to provide a complete description of the current environmental state to the agent, so that the agent can select appropriate actions from the action space accordingly, therefore, the state space has no direct output.

[0147] Specifically, the input of the action space is the information provided by the agent according to the state space, and the decision-making process will consider the dimensions of the state (such as vehicle dynamics state, traffic compliance state, historical trajectory, environmental information, etc.), through certain policy network or decision logic to determine the executable action, these state information is the source of input of the action space. The output of the action space includes continuous output and discrete output; continuous output includes acceleration and steering angle, these continuous control variables are used to fine-tune the vehicle's motion state; discrete output includes whether to change lanes, decelerate or accelerate, etc., in the form of discrete decision instructions, so that the vehicle makes clear action selection in different traffic scenarios.

[0148] The logical relationship between the state space and the action space is:

[0149] State-driven action selection: The state space contains the observation information of the agent to the environment, reflecting the current situation of the environment. The agent decides to take what action according to the information in the state space, that is, the state is the basis for action selection. For example, in the automatic driving scene, the current speed, distance from the front car, lane position and other state information of the vehicle will prompt the agent to select the appropriate action from the action space (such as the action set of acceleration, deceleration, turning, etc.). Action affects state transition: After the agent selects and executes a certain action from the action space, the action will act on the environment, causing the state of the environment to change. For example, in the robot walking task, after the robot executes the "step forward" action, its position, posture and other state information in the environment will change accordingly, thus entering a new state. The new state will become the basis for the next action selection, and so on.

[0150] Further, an initial scene is generated using random actions and a reinforcement learning agent policy network is trained. First, the agent generates different driving scenes through a randomly initialized policy, for example, simulating the probability of pedestrian appearance, vehicle spacing and traffic flow density on urban roads. Then, the score of each scene is calculated according to the reward function to evaluate whether the scene meets the driving style classification and behavior logic. Next, the agent policy network is updated using a deep reinforcement learning algorithm to gradually generate high-quality scenes, such as aggressive driving style overtaking scenes on highways or defensive style avoidance behavior scenes at urban intersections. Finally, through multiple rounds of training, the scene generation policy converges to an optimal solution, ensuring the diversity and test coverage of the generated driving scenes.

[0151] In this embodiment, a fusion network is designed in combination with the Transformer framework, that is, the Transformer encoder is used to extract environment and vehicle state features, the Actor network generates driving strategies according to the features, and the Critic network evaluates the value of the action. The Actor network and the Critic network form the agent policy network. Further, the MAPPO reinforcement learning algorithm is used to update the Actor network and the Critic network.

[0152] The process of training and updating the agent policy network includes the following steps:

[0153] S1, network parameter initialization.

[0154] Specifically, the learning rate is initially set to 1e -4 , the discount factor is initially γ=0.99, and the time difference error (TD error) is used to optimize the Critic network.

[0155] S2, feature extraction and training process of environment state sequence. Specifically, it includes:

[0156] S21, acquire real-time state information, historical state trajectory and environment information of the vehicle, and encode the real-time state information, the historical state trajectory and the environment information by using a Transformer encoder to obtain a vehicle state at a current time.

[0157] Specifically, the real-time state information includes vehicle speed, acceleration, yaw rate, steering angle, overspeed state, following distance state and lane keeping state; the historical state trajectory includes a state at a previous time step; and the environment information includes road type and traffic flow. Illustratively, the vehicle speed, acceleration, steering angle and path deviation and the like are acquired from SUMO simulation and are encoded as time series data. Meanwhile, the Transformer encoder converts an input state feature vector into a high-dimensional representation.

[0158] S22, generate a driving behavior action by using an Actor network based on the vehicle state at the current time, execute the driving behavior action to obtain a vehicle state at a next time, and calculate a reward of a current driving scene by using a multi-objective reward function.

[0159] Specifically, the driving behavior action includes continuous output and discrete output, the continuous output includes acceleration and steering angle, and the discrete output includes whether to change lanes, decelerate or accelerate. Illustratively, a driving behavior action (such as acceleration, deceleration or lane changing) is generated based on the output features of the encoder, and the action is fed back to the SUMO simulation for execution, the simulation environment returns a new state, and a reward of a current driving scene is calculated by using a multi-objective reward function.

[0160] The multi-objective reward function is as follows:

[0161]

[0162] wherein, is a comprehensive dynamics reward, is a comprehensive traffic compliance reward, is a risk preference reward, is a collaborative manner reward, is a driving style matching reward, , is a weight coefficient;

[0163]

[0164]

[0165]

[0166]

[0167]

[0168] wherein, is a speed reward, is an acceleration reward, is a yaw rate reward, is a steering angle reward, , , , is a weight coefficient;

[0169]

[0170]

[0171]

[0172]

[0173] wherein, is a speed over-speed reward, is a following distance reward, is a lane keeping reward; is a weight coefficient;

[0174]

[0175]

[0176] .

[0177] It can be understood that when the range of risk preference and cooperation mode within the current driving style is low, medium or high, the reward is added by 0.01, otherwise, it is added by -0.02.

[0178] S23, based on the time difference method, using the Critic network to evaluate the vehicle state and the driving behavior action at the current moment; wherein, the Actor network and the Critic network form the intelligent agent policy network.

[0179] Illustratively, the Q value or V value of the current state and action is evaluated, the long-term income is predicted in combination with the next state feature, and then the TD error is used as the loss function of the Critic network, which is used to update the network weight, so that the evaluation gradually approaches the real environment feedback value.

[0180] The loss function based on the TD error is:

[0181]

[0182] wherein, is the time ta reward, is a discount factor, is a state is a value estimate, is a parameter of the Critic network.

[0183] S25, updating the Actor network and the Critic network using the MAPPO reinforcement learning algorithm, and stopping training when a preset number of iterations is reached.

[0184] S25, adjusting the weights in the multi-objective reward function to optimize the agent policy network.

[0185] Specifically, the first time can be set , 0.2, , , , 0.25, 0.33, according to the training effect of S21-S25, adjust the different weight coefficients for repeated training until the agent policy network converges to the optimal solution that meets the fuzzy rules of driving style, and obtain the trained agent policy network. The trained agent policy network can be used to generate multi-dimensional natural driving scenarios and perform simulation testing.

[0186] Further, after obtaining the trained agent policy network, a closed-loop feedback mechanism can also be established to analyze the scene adaptability and optimize the scene generation logic. First, for the scenes that fail in simulation testing, such as scenes where the driving style cannot match the expected target or the behavior is unreasonable, a systematic analysis is performed to find out the failure reasons. Then, combined with the analysis results, the scene generation logic is adjusted again, such as optimizing the reinforcement learning model parameters or adjusting the weight distribution of the reward function. Next, the optimized scene is input into the Carla simulation system again for testing to ensure that it can cover more boundary conditions and test scenarios. Finally, the scene parameters generated by multiple iterations are gradually converged to make the driving style scene generation more comprehensive and efficient. After that, the simulation data and real vehicle data are bidirectionally migrated and evaluated in a closed loop. Specifically, the system migrates the natural traffic environment and dynamic characteristics in real vehicle testing to the simulation platform to enhance the simulation realism; at the same time, the simulation results are used to optimize the real vehicle testing plan, and key boundary scenarios are tested first, finally forming a closed-loop feedback mechanism of data and model, thereby effectively improving the robustness and comprehensiveness of system performance.

[0187] The embodiment introduces risk preference and cooperation mode naturalness indicators, and is based on kinetic characteristics, rule compliance, psychological characteristics and cooperation mode to construct a driving style fuzzy rule, and then uses a reward function in a reinforcement learning framework to train and update an agent strategy network, so as to realize dynamic generation of a driving behavior model. The method can generate diversified behavior patterns covering from cautious driving to aggressive driving, obtain more complete and accurate behavior characteristic representation, and thus generate diversified driving behaviors, thereby providing scientific and efficient technical support for research and development, verification, future technical standardization, safety evaluation and large-scale application of an automatic driving system.

[0188] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, some simple deductions or substitutions can be made without departing from the concept of the present application, and all of them shall be regarded as falling within the protection scope of the present application.

Claims

1. A method for multi-dimensional naturalistic driving scenario generation and simulation testing based on reinforcement learning, characterized in that, The method comprises the steps of: S1, based on the kinetic characteristics, rule compliance, psychological characteristics and cooperation mode, the driving behavior characteristics are quantified by combining the fuzzy logic classification model, and the driving style fuzzy rule is constructed; step S1 comprises: S11, acquiring vehicle kinematics state indicators, and defining fuzzy values of different vehicle kinematics state indicators by using fuzzy subsets to obtain vehicle dynamics state fuzzy indicators; S12, acquiring traffic compliance state indicators, and defining fuzzy values of different traffic compliance state indicators by using fuzzy subsets to obtain traffic compliance state fuzzy indicators; S13, calculating the risk aversion coefficient, risk decision threshold and risk transition coefficient and constructing a risk preference comprehensive index by weighting; the calculation formula of the risk aversion coefficient is: wherein, is a longitudinal acceleration, is a lateral acceleration, is a steering angle, is a speed; is a weight coefficient; The calculation formula of the risk decision threshold is: wherein, respectively, the following distance, the headway time, the collision time, are weight coefficients; The calculation formula of the risk transition coefficient is: wherein, respectively, are a longitudinal acceleration variation, a lateral acceleration variation, a speed variation, a steering angle variation, and a yaw rate variation; S14, calculating the cooperation initiative coefficient and the information sharing degree and constructing a cooperation mode comprehensive index by weighting; The calculation formula of the cooperation initiative coefficient is: wherein, , respectively a lateral position, a lateral distance to an adjacent vehicle, , is a weight coefficient; The calculation formula of the information sharing degree is: wherein, is the speed, is the traffic flow density, , is the weight coefficient; S15, constructing a driving style fuzzy rule according to the vehicle dynamics state fuzzy indicators, the traffic compliance state fuzzy indicators, the risk preference comprehensive index and the cooperation mode comprehensive index; wherein the driving style fuzzy rule includes: very defensive, high priority defensive, low priority defensive, high priority normal motion, low priority normal motion, very motion, high priority aggressive and low priority aggressive; S2, constructing a driving environment and configuring quantified characteristics of different driving styles according to the driving style fuzzy rule; S3, using a reinforcement learning algorithm, based on the driving environment and the quantified characteristics of different driving styles, taking the driving style fuzzy rule as the target, calculating the reward of different driving scenes according to a reward function to train and update an agent policy network, and obtaining a trained agent policy network; the trained agent policy network is used to generate multi-dimensional natural driving scenes and perform simulation testing.

2. The multi-dimensional natural driving scene generation and simulation testing method based on reinforcement learning according to claim 1, wherein: The vehicle kinematics state indicators include speed, acceleration, steering angle and yaw rate; wherein the fuzzy values of the speed include low speed, medium speed and high speed, the fuzzy values of the acceleration include low acceleration, medium acceleration and high acceleration, the fuzzy values of the steering angle include low steering angle and high steering angle, and the fuzzy values of the yaw rate include low yaw rate and high yaw rate; The traffic compliance state indicators include speed over-speed state, following distance safety state and lane keeping specification state, the fuzzy values of the speed over-speed state include speed over-speed and speed not over-speed, the fuzzy values of the following distance safety state include following distance safety and following distance unsafe, and the lane keeping specification state includes lane keeping well, lane keeping generally and lane keeping poorly.

3. The method of claim 1, wherein the method is based on reinforcement learning. Step S2 comprises: A driving environment is constructed in the simulation platform, and quantitative characteristics of different driving styles are configured according to the driving style fuzzy rules, wherein the driving environment includes complex road conditions, dynamic traffic flow and multi-type vehicle models.

4. The method of claim 1, wherein the method is based on reinforcement learning. Step S3 comprises: obtaining real-time state information, historical state trajectory and environment information of the vehicle, and encoding the real-time state information, the historical state trajectory and the environment information by using a Transformer encoder to obtain a current vehicle state; based on the current vehicle state, generating a driving behavior action by using an Actor network, executing the driving behavior action to obtain a next vehicle state, and calculating a reward of a current driving scene by using a multi-objective reward function; based on a time difference method, evaluating the current vehicle state and the driving behavior action by using a Critic network; wherein the Actor network and the Critic network form the agent policy network; updating the Actor network and the Critic network by using a MAPPO reinforcement learning algorithm; adjusting the weight in the multi-objective reward function to optimize the agent policy network.

5. The multi-dimensional natural driving scene generation and simulation test method based on reinforcement learning according to claim 4, characterized in that: the real-time state information includes vehicle speed, acceleration, yaw rate, steering angle, overspeed state, following distance state and lane keeping state; the historical state trajectory includes the state of the previous time step; and the environment information includes road type and traffic flow; the driving behavior action includes continuous output and discrete output, the continuous output includes acceleration and steering angle, and the discrete output includes whether to change lanes, decelerate or accelerate.

6. The method of claim 4, wherein the method further comprises: the multi-objective reward function is: wherein, is a comprehensive dynamics reward, is a comprehensive traffic compliance reward, is a risk preference reward, is a collaborative manner reward, is a driving style matching reward, , is a weight coefficient; wherein, is a speed reward, is an acceleration reward, is a yaw rate reward, is a steering angle reward, , , , is a weight factor; wherein, is a speed over speed reward, is a following distance reward, is a lane keeping reward; is a weight coefficient; 。

Images