Adaptive autonomous driving method and system for complex road scene

Abstract

The application discloses a training method and system of an adaptive automatic driving model for a complex road scene, wherein the method comprises the following steps: training a plurality of first-level automatic driving models, wherein the reward functions of different models correspond to different training strategy tendencies; comprehensively scoring the trained plurality of first-level automatic driving models, and screening three models with the highest comprehensive scores as expert models; training a second-level automatic driving model, inputting data in a training set into the three expert models, then inputting the outputs of the three expert models into a reward function of the second-level automatic driving model for calculation, and in the case of keeping relatively consistent with expert demonstration actions, the second-level automatic driving model receives less punishment, and if deviating from the demonstration actions, the second-level automatic driving model receives greater deduction in the reward level, and the strategy of the trained second-level automatic driving model approaches the demonstration actions of the expert models. The application can improve the multi-scene adaptability of automatic driving, enrich the decision candidate space, and shorten the training cycle and reduce the difficulty.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, and in particular to an adaptive automatic driving method and system for complex road scenes. BACKGROUND

[0002] With the rapid development of intelligent network technology, the global automobile intelligentization process is accelerating. Intelligent networked vehicles are becoming an important direction for the development of domestic and foreign automobile industry due to their significant advantages in reducing safety accidents, reducing environmental pollution, relieving traffic congestion and reducing energy consumption. An automatic driving vehicle travels along a predetermined route, but the real traffic environment is not static. There are usually various traffic participants on the road, whose real-time state is constantly changing. Therefore, the automatic driving vehicle needs to dynamically predict the surrounding environment and implement following, avoiding and lane changing and other motion interaction strategies according to different traffic participants. This not only helps to adjust the driving strategy in real time, but also provides guidance for trajectory planning, ensuring that the vehicle can complete the driving task as efficiently and safely as possible. The automatic driving system can be regarded as a combination of traditional automobile technology and modern high-performance computer technology, covering environment perception, map positioning, prediction, decision-making, planning and control execution. In the aspect of perception and positioning, the automatic driving vehicle is equipped with laser radar, millimeter wave radar, camera and other sensors, and through sensor fusion technology combined with high-precision maps, it comprehensively perceives the static and dynamic environment around the vehicle, realizes high-precision positioning and global path planning.

[0003] At the decision and planning level, the system includes three aspects: trajectory prediction of traffic participants, vehicle behavior decision, and trajectory planning to generate a safe and collision-free driving trajectory that meets specific constraints. Finally, at the control level, the system uses vehicle dynamics control theory to output control commands such as accelerator, brake, and steering wheel angle, enabling precise control of vehicle motion through electronic control chassis, ensuring that the actual driving trajectory of the vehicle accurately tracks the planned trajectory. Currently, behavior decision methods are mainly divided into rule-based methods and data-based learning methods. Rule-based decision schemes rely on pre-set traffic rules and driving experience. However, this method cannot cover all possible scenarios, as real-world traffic scenarios are often extremely complex, and dynamic traffic participants may take completely opposite actions at a moment. Therefore, when faced with unknown scenarios, rule-based decision systems may encounter processing limitations, and even require driver intervention to handle complex situations. With the development of artificial intelligence technology, especially the success in image recognition, financial market analysis, and natural language processing, autonomous driving technology provides more solutions. Among them, reinforcement learning methods enable agents to make more reasonable behavior decisions based on environmental changes through automatic interaction and continuous optimization. Combined with the powerful representation ability of deep learning neural networks, reinforcement learning provides an important way to achieve autonomous driving decision-making technology, especially in handling high-dimensional complex problems, which can significantly improve the intelligence level of decision-making systems.

[0004] The behavior decision subsystem is a core part in complex dynamic urban traffic environments, and its main task is to ensure the safety of autonomous vehicles, comply with traffic regulations, and provide the necessary constraint information for smooth path and speed optimization. Quick and accurate behavior decision can effectively reduce accidents and is of great significance to personal safety. Having similar decision-making capabilities as a driver is a key indicator of high intelligence level for cars, which directly determines the level of autonomous driving systems. Depending on the architecture, the input information of the decision system may include structured road information such as current lane, merging lane, intersection, traffic sign information, obstacle status, and trajectory prediction information of surrounding traffic participants. Through comprehensive analysis of current environmental and traffic participant information, the system can output driving instructions, including following, free lane changing, forced lane changing, and overtaking, etc.

[0005] In the development process of automatic driving system, a core contradiction is that the model trained based on fixed scene performs excellently in simple environment, but is difficult to adapt to the complex road environment in real world. When the road type, traffic flow is relatively single and the behavior of pedestrians and vehicles is predictable, the driving model with good performance can be effectively trained through the artificially designed supervised signal and reward function; however, when the deployment scene is expanded to the actual road which is dynamic and changeable, the generalization ability of the model will be significantly degraded. This is essentially caused by the structural contradiction between the training paradigm and the real demand. The artificially designed reward function faces a fundamental dilemma in pursuing the dynamic balance between safety and efficiency: the conventional scene which occurs frequently requires the model to adopt aggressive strategy to improve the traffic efficiency, while the low-frequency high-risk scene (such as sudden obstacles, extreme weather) requires absolutely conservative safety strategy, and a single reward function is difficult to cover such scene-sensitive strategy requirements. More seriously, the probability of occurrence of key safety events in real road is often less than one in ten thousand, which leads to the serious lack of coverage of high-risk scenes in training data, so that the decision-making ability of the model will be greatly reduced when facing sudden conditions.

[0006] This contradiction is further intensified at the model architecture level: when the model optimizes to the performance extreme in a specific scene (such as highway cruising), the strategy network parameters will be fixed to form path dependence, which is difficult to quickly switch the strategy tendency when the scene mutates (such as turning into a congested urban road). Although the introduction of human expert supervision can accelerate the convergence of the model in a specific scene, when facing the full-scene demand including hundreds of road types and thousands of traffic participants, the cost of expert labeling will increase exponentially. More importantly, human driving behavior itself has a 23% decision-making error rate, and the supervision signal provided by it may contain suboptimal or even incorrect patterns, which will be further amplified in cross-regional and cross-cultural scenes. SUMMARY

[0007] The main purpose of the present application is to provide an efficient, safe and reliable adaptive automatic driving method and system for complex road scenes in complex dynamic environments.

[0008] The technical scheme adopted by the present application is:

[0009] A training method of an adaptive automatic driving model for complex road scenes is provided, comprising the following steps:

[0010] A plurality of primary automatic driving models are trained, the reward functions of different models correspond to different training strategy tendencies, and specifically correspond to different weights of the trade-off between safety reward and driving efficiency reward; the input of the primary automatic driving model is road environment information, self-vehicle and other vehicle state information, and the output is the control amount of vehicle automatic driving action, including the control amount of vehicle steering wheel angle, accelerator and brake pedal action;

[0011] The trained plurality of first-level automatic driving models are comprehensively scored, and the three models with the highest comprehensive scores are selected as expert models; the specific process of comprehensive scoring includes: measuring the driving performance of the ego vehicle and scoring by using a pre-constructed individual evaluation model, and measuring the interaction safety of the ego vehicle with other surrounding vehicles and scoring by using a pre-constructed social evaluation model, and then weighting and summing the two scores according to a preset weight to obtain the comprehensive score;

[0012] The pre-constructed second-level automatic driving model is trained, the input and output of the second-level automatic driving model are the same as those of the first-level automatic driving model, and during the training, the data in the training set is input into the three expert models, and then the outputs of the three expert models are input into the reward function of the second-level automatic driving model for calculation, and in the case of keeping relatively consistent with the expert demonstration action, it will be less punished, and if it deviates from the demonstration action, it will be greatly reduced in terms of reward, and the strategy of the trained second-level automatic driving model approaches the demonstration action of the expert model.

[0013] According to the above technical scheme, the reward function of the second-level automatic driving model fuses the environment original reward, the individualized behavior discriminator reward and the individualized transition discriminator reward, wherein the environment original reward is specifically the basic return obtained after interacting with the environment at the current time, which is used to ensure that the training target is consistent with the task demand; the individualized behavior discriminator reward is specifically the output function value of the individualized behavior discriminator, which calculates the matching degree of the state-action of the current vehicle with the state-action of the expert model; and the individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the matching degree of the state transition of the current and executed vehicle state-action with the state transition of the expert model.

[0014] According to the above technical scheme, the weights of the safety reward and the driving efficiency reward in the reward function of the first-level automatic driving model are specifically obtained by sampling two Gaussian distributions.

[0015] According to the above technical scheme, the individual evaluation model specifically scores the individual characteristics of the ego vehicle, wherein the individual characteristics include the ego vehicle motion state, the ego vehicle driving behavior and the individual characteristics based on uncertainty, the ego vehicle motion state includes speed, acceleration and acceleration change rate; the ego vehicle driving behavior includes waiting time, the difference between actual speed and lane speed limit; and the individual characteristics based on uncertainty include the percentage of time spent driving according to the lane line, and the trajectory abnormal value.

[0016] According to the above technical scheme, the score of the individual characteristics is specifically: multiplying the minimum value of each individual characteristic at all time steps by the corresponding weight, and then summing to obtain.

[0017] According to the technical scheme, the social evaluation model scores according to different interaction indexes between driving subjects, and the interaction indexes include a time headway, a collision time, a required deceleration for avoiding collision, and a conflict time difference of mutual interaction of two vehicles.

[0018] According to the technical scheme, the scoring of the social evaluation model is specifically as follows: after obtaining the minimum value of each interaction index in the interaction process of a pair of vehicles, a predefined weight vector is used for weighted summation.

[0019] The application further provides a training system of an adaptive automatic driving model for a complex road scene, comprising:

[0020] The first-level automatic driving model training module is configured to train a plurality of pre-constructed first-level automatic driving models, wherein different reward functions of different models correspond to different training strategy tendencies, and specifically correspond to different weights of trade-off between safety rewards and driving efficiency rewards; the input of the first-level automatic driving model is road environment information, self-vehicle and other vehicle state information, and the output is a control amount of vehicle automatic driving action, including a control amount of vehicle steering wheel angle, accelerator and brake pedal action;

[0021] The parallel space-time prediction module is configured to comprehensively score the trained plurality of first-level automatic driving models, and select three models with the highest comprehensive scores as expert models; the specific process of the comprehensive scoring includes: measuring and scoring the driving performance of the self-vehicle by using a pre-constructed individual evaluation model, and measuring and scoring the interaction safety of the self-vehicle and other surrounding vehicles by using a pre-constructed social evaluation model, and then weighting and summing the two scores according to a preset weight to obtain the comprehensive score;

[0022] The second-level automatic driving model training module is configured to train a pre-constructed second-level automatic driving model, wherein the input and output of the second-level automatic driving model are the same as those of the first-level automatic driving model; during the training, the data in the training set is input into the three expert models, and the outputs of the three expert models are input into the reward function of the second-level automatic driving model for calculation; in the case of keeping relatively consistent with the expert demonstration action, the second-level automatic driving model receives less punishment, and if deviating from the demonstration action, the second-level automatic driving model will receive greater deduction in the reward level; the strategy of the trained second-level automatic driving model approaches the demonstration action of the expert model.

[0023] In the above technical solution, the reward function of the secondary automatic driving model fuses an environment original reward, an individualized behavior discriminator reward and an individualized transfer discriminator reward, wherein the environment original reward is specifically a basic return obtained after interaction with the environment at the current time, and is used to ensure that the training target is consistent with the task requirement; the individualized behavior discriminator reward is specifically an output function value of the individualized behavior discriminator, which calculates the matching degree of the state-action of the current vehicle with the state-action of the expert model; and the individualized transfer discriminator reward is an output function value of the individualized transfer discriminator, which calculates the matching degree of the state transfer of the current and executed vehicle state-action with the state transfer of the expert model.

[0024] The application further provides a computer storage medium, which has stored therein a computer program executable by a processor, and the computer program executes the training method of the adaptive automatic driving model for complex road scenes.

[0025] The application has the following advantages: the application proposes an innovative framework of hierarchical expert distillation: firstly, a plurality of different primary automatic driving models are normally trained (such as training based on fixed scenes and simple reward functions), and then a model library is formed based on these models; and then, the decisions of the experts are fused, and knowledge distillation is used to encode the scene adaptability into a secondary automatic driving model. The architecture of the application breaks through the traditional mode and provides a new technical path for solving the generalization dilemma of automatic driving. Since each primary automatic driving model has a specific strategy tendency (for example, emphasizing safety or efficiency), a single model is difficult to maintain optimal performance in all scenes. By integrating a plurality of models with different decision characteristics into a secondary automatic driving model, the following advantages are achieved: (1) improving multi-scene adaptability: in different driving environments such as congestion, smoothness, high speed or urban roads, the secondary automatic driving model can flexibly "borrow" the primary model that best fits the current scene, thereby achieving better performance in safety or efficiency. (2) enriching the decision candidate space: a plurality of models accumulate differentiated decision experience, and through parallel prediction and scoring, the advantages and disadvantages of multiple driving strategies can be quickly evaluated at the same time, and the optimal strategy is selected for execution. (3) shortening the training period and reducing the difficulty: compared with training a single model that covers all working conditions from scratch, the secondary model only needs to be further strengthened on the basis of a plurality of primary models (equivalent to "expert demonstration"), and therefore can converge faster and avoid unnecessary repeated exploration.

[0026] Further, the application takes full advantage of the advantage that a specific style automatic driving model is easy to train based on a simple reward function. By training a plurality of primary automatic driving models for a plurality of simple reward functions, and combining the diversified reward function design method based on Gaussian sampling proposed by the application, comprehensive coverage of the optimal driving strategy for all scenes is achieved.

[0027] Of course, implementing any product of the application does not necessarily require that all of the above-mentioned advantages be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0029] Figure 1 is a self-adaptive automatic driving method flow chart for complex road scenes according to an embodiment of the present application;

[0030] Figure 2 is a first-level automatic driving model training flow chart according to an embodiment of the present application;

[0031] Figure 3 is a second-level automatic driving model training flow chart according to an embodiment of the present application;

[0032] Figure 4 is a first-level automatic driving model training process chart according to an embodiment of the present application;

[0033] Figure 5 is a second-level automatic driving model training process chart according to an embodiment of the present application;

[0034] Figure 6 is a parallel space-time prediction flow chart according to an embodiment of the present application;

[0035] Figure 7 is a parallel space-time prediction technical route chart according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0037] It should be noted that the diagrams provided in the embodiments of the present application only illustrate the basic concept of the present application in a schematic manner, and therefore, only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component can be randomly changed in terms of shape, number and proportion, and the layout pattern of the components can also be more complex.

[0038] In the present application, it also needs to be explained that, as the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like appear, the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, as the terms "first", "second" appear, they are only for description and distinction purposes, and cannot be understood as indicating or implying relative importance.

[0039] In addition, it also needs to be explained that the features of various embodiments of the present application can be combined or integrated partially or entirely, and as understood by those skilled in the art, can interact and operate in different ways. Each embodiment can be realized independently of each other, or in an associated relationship.

[0040] To adapt to the dynamic and variable driving environment, the present application first constructs multiple automatic driving models with different driving styles based on a relatively simple reward function and a general deep reinforcement learning (DRL) method, collectively referred to as a first-level automatic driving model. Due to the simple design of the reward function of the model, the training difficulty is relatively low, and good automatic driving performance can be achieved in a specific scenario. Subsequently, these first-level automatic driving models are included in a model library, and a more complex second-level automatic driving model is further constructed on this basis. The second-level automatic driving model can fully utilize the advantages of each first-level model to achieve optimal performance in different environments. Thus, the present application aims to solve the following two core problems: ① how to construct a first-level automatic driving model that covers as many driving styles as possible; ② how to use the above model library to construct an automatic driving model with stronger adaptability that can maintain optimal performance in multiple scenarios.

[0041] In one embodiment of the present application, as shown in Figure 1 The training method of the adaptive automatic driving model for complex road scenes includes the following steps:

[0042] S1, training a plurality of pre-constructed first-level automatic driving models, the reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights of the trade-off between safety reward and driving efficiency reward; the input of the first-level automatic driving model is the road environment information, the state information of the ego vehicle and other vehicles, and the output is the control amount of the vehicle automatic driving action, including the control amount of the vehicle steering wheel angle, the throttle, and the brake pedal action;

[0043] S2, score the trained plurality of first-level automatic driving models comprehensively, and select three models with the highest comprehensive scores as expert models; the specific process of comprehensive scoring includes: measuring the driving performance of the ego vehicle and scoring by using a pre-constructed individual evaluation model, and measuring the interaction safety of the ego vehicle with other surrounding vehicles and scoring by using a pre-constructed social evaluation model, and then weighting and summing the two scores according to a preset weight to obtain a comprehensive score;

[0044] S3, training a pre-constructed second-level automatic driving model, the input and output of the second-level automatic driving model being the same as those of the first-level automatic driving model, and during the specific training, the data in the training set is input into the three expert models, and the outputs of the three expert models are input into a reward function of the second-level automatic driving model for calculation, and in the case of keeping relatively consistent with the expert demonstration action, the second-level automatic driving model receives less punishment, and if deviating from the demonstration action, the second-level automatic driving model will receive greater deduction in the reward level, and the strategy of the trained second-level automatic driving model approaches the demonstration action of the expert model.

[0045] Further, the reward function of the second-level automatic driving model fuses an environment original reward, an individualized behavior discriminator reward and an individualized transition discriminator reward, wherein the environment original reward is specifically a basic return obtained after interacting with the environment at the current time, and is used to ensure that the training target is consistent with the task demand; the individualized behavior discriminator reward is specifically the output function value of the individualized behavior discriminator, which calculates the matching degree of the state-action of the current vehicle with the state-action of the expert model; and the individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the matching degree of the state transition of the current and executed vehicle state-action with the state transition of the expert model.

[0046] Further, the weights of the safety reward and the driving efficiency reward in the reward function of the first-level automatic driving model are specifically obtained by sampling two Gaussian distributions.

[0047] Further, the individual evaluation model specifically scores the individual characteristics of the ego vehicle, wherein the individual characteristics include the ego vehicle motion state, the ego vehicle driving behavior and the individual characteristics based on uncertainty, the ego vehicle motion state includes the speed, the acceleration and the change rate of the acceleration; the ego vehicle driving behavior includes the waiting time, the difference between the actual speed and the speed limit of the lane; and the individual characteristics based on uncertainty include the percentage of time of driving according to the lane line and the trajectory abnormal value.

[0048] Further, the score of the individual characteristics is specifically: multiplying the minimum value of each individual characteristic at all time steps by the corresponding weight, and then summing to obtain.

[0049] Further, the social evaluation model scores according to different interaction indexes between different driving subjects, and the interaction indexes include time headway, collision time, deceleration required to avoid collision, and conflict time difference of interaction between two vehicles.

[0050] Further, the score of the social evaluation model is specifically obtained by weighting and summing by using a predefined weight vector after obtaining the minimum value of each interaction index in the interaction process of a pair of vehicles.

[0051] To implement the above method, the training system of the adaptive automatic driving model for complex road scenes in the embodiment of the application comprises:

[0052] The first-level automatic driving model training module is configured to train a plurality of first-level automatic driving models constructed in advance, wherein the reward functions of different models correspond to different training strategy tendencies, and specifically correspond to different weights of trade-off between safety reward and driving efficiency reward; the input of the first-level automatic driving model is road environment information, self-vehicle and other vehicle state information, and the output is a control amount of vehicle automatic driving action, including a control amount of vehicle steering wheel angle, accelerator and brake pedal action;

[0053] The parallel space-time prediction module is configured to comprehensively score the plurality of trained first-level automatic driving models, and select three models with the highest comprehensive scores as expert models; the specific process of the comprehensive score includes: measuring and scoring the driving performance of the self-vehicle by using a pre-constructed individual evaluation model, and measuring and scoring the interaction safety of the self-vehicle and other surrounding vehicles by using a pre-constructed social evaluation model, and then weighting and summing the two scores according to a preset weight to obtain the comprehensive score;

[0054] The second-level automatic driving model training module is configured to train a pre-constructed second-level automatic driving model, wherein the input and output of the second-level automatic driving model are the same as those of the first-level automatic driving model; during the training, the data in the training set is input into the three expert models, and the outputs of the three expert models are input into the reward function of the second-level automatic driving model for calculation; in the case of keeping relatively consistent with the expert demonstration action, the second-level automatic driving model receives less punishment, and if deviating from the demonstration action, the second-level automatic driving model will receive greater deduction in the reward level; the strategy of the trained second-level automatic driving model approaches the demonstration action of the expert model.

[0055] Further, the reward function of the secondary automatic driving model fuses an environment original reward, an individualized behavior discriminator reward and an individualized transfer discriminator reward, wherein the environment original reward is specifically a basic return obtained after interacting with the environment at the current time, and is used to ensure that the training target is consistent with the task requirements; the individualized behavior discriminator reward is specifically the output function value of the individualized behavior discriminator, which calculates the matching degree of the state-action of the current vehicle with the state-action of the expert model; and the individualized transfer discriminator reward is the output function value of the individualized transfer discriminator, which calculates the matching degree of the state transition of the current and executed vehicle state-action with the state transition of the expert model.

[0056] Specifically, in constructing the primary automatic driving model, a common SAC deep reinforcement learning training paradigm can be adopted, and multiple automatic driving agents are trained for different "policy tendencies". The input of each agent includes: self-vehicle coordinates, speed, acceleration; road coordinates; coordinates, speed and acceleration of surrounding environment vehicles. The output is the control amount of the vehicle (steering wheel angle and accelerator, brake pedal). The "policy tendency" referred to herein mainly reflects the different emphasis of the agent on the trade-off between safety and efficiency (or vehicle speed). For example, when the weight of speed and efficiency is larger and the weight of safety item is relatively smaller in the reward function, the agent tends to pursue higher speed; on the contrary, when the weight of safety item is dominant, the agent will give priority to driving safety, thereby maintaining a relatively conservative driving strategy.

[0057] The "policy tendency" mainly reflects the different trade-off between safety and efficiency (or vehicle speed) of the agent. Specifically, when the weight of speed and efficiency is larger and the weight of safety item is relatively smaller in the reward function, the agent tends to drive at a higher speed; and when the weight of safety item is dominant, the agent will give priority to driving safety, thereby adopting a more conservative driving strategy.

[0058] On the basis of the traditional deep reinforcement learning training process, such as Figure 2 , 4As shown, the present application proposes a diversified reward function design method based on Gaussian sampling to generate agent models of different driving styles. First, the simulation environment is initialized, which includes the ego vehicle and several environment vehicles, and a random choice of highway, roundabout or crossroads is made as the road scene. Then the Actor and Critic network parameters of the SAC agent are initialized, and the experience pool is established according to the state space and action space structure. In the early stage of training, the agent controls the ego vehicle to interact with the environment for 10,000 steps with a random strategy. After each interaction, the reward value of the current state-action pair is calculated based on the reward function of the first-level autonomous driving model, and the state, action, reward and next state are stored in the experience pool. After the initial 10,000 steps, the training phase begins. Every 5 interactions, 1024 experience tuples are randomly sampled from the experience pool for network training: the Critic network is updated by minimizing the mean square error (MSE) of Q value estimation, and the Actor network parameters are updated using the Adam optimizer. New experiences are continuously added to the experience pool during training until its capacity reaches the upper limit. The total training is 200,000 steps, and the final model parameters are saved as the first-level autonomous driving strategy model.

[0059] Specifically, the reward function of the autonomous driving task can be divided into the following two parts:

[0060] (1) Efficiency reward

[0061] Used to measure efficiency-related indicators such as vehicle speed, driving distance, and early arrival at the target.

[0062] (2) Safety reward

[0063] Used to punish unsafe behaviors such as collisions, sudden braking, and insufficient distance from the front vehicle.

[0064] The comprehensive reward can be expressed as:

[0065]

[0066] wherein and are the weights of the efficiency and safety parts, respectively. In order to reflect different decision-making preferences among multiple agents, the present application sets two Gaussian distributions: and to sample the weights of efficiency and safety. Among them, and are used to control the dispersion degree of the sampled weights.

[0067] The original weights and After that, it also needs to be mapped to a comparable and stable range through normalization operation. If the standard deviation of the Gaussian distribution is large, it means that the dispersion of weight sampling is higher, so that more diverse strategy tendencies can be obtained in the same batch of agents, but it also means that more sampling and training times are needed to obtain uniform style samples. Normalization formula:

[0068]

[0069] As shown in Figure 3 , 5 , the construction of the secondary autonomous driving model is a new model formed by integrating multiple one-level autonomous driving models with different "strategy tendencies" based on the original one-level autonomous driving model. First, initialize the simulation environment, which includes the ego vehicle and several environment vehicles, and randomly select a highway, roundabout, or crossroads as the road scene. Then initialize the Actor and Critic network parameters of the SAC agent, and construct the experience pool according to the state space and action space structure. In the early stage of training, the ego vehicle interacts with the environment with a random strategy under the control of the SAC agent for 10,000 steps. After each step, call the parallel space-time prediction module, select 3 expert models from the one-level autonomous driving model library, and input the current state into these 3 models to obtain the corresponding expert behaviors. Then, based on the reward function of the secondary autonomous driving model, calculate the reward value corresponding to each expert behavior, and select the highest reward as the final reward for the current step. Store the experience tuple composed of state, action, final reward, and next state in the experience pool. After completing 10,000 steps, enter the training phase. Every 5 steps of interaction, randomly sample 1024 experience tuples from the experience pool, update the Critic network by minimizing the mean square error (MSE) of Q value estimation, and update the Actor network using the Adam optimizer. At the same time, continuously add newly generated experiences to the experience pool until the capacity limit is reached. The entire training process lasts for 200,000 steps, and after training is complete, the model parameters are saved as a secondary autonomous driving strategy model.

[0070] The input and output of the secondary automatic driving model are completely same as the primary automatic driving model, that is, the same self-vehicle coordinates, speed, acceleration, road environment information, and other vehicle states (position, speed, acceleration, etc.) are received, and the output is still the decision of the control amount of the vehicle steering wheel angle, accelerator, brake pedal, etc. However, compared with the primary model, the secondary automatic driving model no longer simply relies on a single strategy for training, but can "learn experience" from multiple trained models to achieve better driving performance in more complex environments. The primary automatic driving model is trained by using the conventional "action-feedback" experience sampling method; the secondary automatic driving model selects the best performing models as "experts" from multiple primary models with the help of the parallel space-time prediction module proposed in the application, and uses the information of these experts to guide its own training. The primary automatic driving model is usually independently trained for specific trade-offs (such as more emphasis on safety or more emphasis on efficiency); the secondary automatic driving model combines multiple driving styles and is more adaptable and stable in more complex or variable driving environments. The primary model is usually a "single style" deep reinforcement learning when constructed, while the secondary model realizes the mode of "multi-model guidance + self-learning", and has expert scoring and guidance in its training process, making it more adaptable to environmental changes.

[0071] The core innovation of the application is to introduce a "parallel space-time prediction module", which comprehensively evaluates the driving behavior of each model in the primary automatic driving model library within a future period of time, obtains the "individual score" and "social score" of each model, and further selects the best performing three models as experts in the training stage of the secondary automatic driving model.

[0072] As shown in Figure 6 , 7 The technical route and main workflow of the parallel space-time prediction module are as follows:

[0073] (1) Start all primary automatic driving models in parallel: under the same environmental information (including lane structure, lane boundary, position, speed, acceleration of self-vehicle and surrounding vehicles, etc.), let all models in the primary automatic driving model library run in parallel.

[0074] (2) Iterative update of future steps: ① For each primary model, first update the coordinates and motion state of the ego vehicle at the next time step according to the ego vehicle and environment state, and the control amount (direction, throttle, brake, etc.) output by the model. ② At the same time, based on the conventional lateral and longitudinal driver model (IDM+MOBIL), predict and update the motion state of the surrounding vehicles at the next time step. ③ Input the updated environment state into these primary models again, and repeat for about 20 time steps (20 steps are taken as an example in the present application; the more steps, the more accurate the prediction, but the calculation overhead will also increase). The lateral and longitudinal driver model (IDM+MOBIL) constructed in the present application belongs to the conventional practice, and is not the core innovation point of the present application, so it will not be described here.

[0075] (3) Scoring based on "individual evaluation model" and "social evaluation model": Individual evaluation model: focuses on measuring the driving performance of the ego vehicle itself, including speed, acceleration, jerk, waiting time, lane-keeping ratio, trajectory deviation, etc.; Social evaluation model: focuses on measuring the interaction safety with other vehicles, including headway (THW), collision time (TTC), deceleration required to avoid collision (DRAC), and conflict time difference (TCD) between two vehicles, etc. In the entire prediction period, in order to capture the highest risk state that may occur, the minimum or maximum value (depending on the risk characteristics) of each safety index is taken to obtain the comprehensive score of the ego vehicle in "individual" and "social" aspects.

[0076] (4) Select the top three models with the highest score as experts: sum the comprehensive score (individual + social) of each primary autonomous driving model to obtain the overall performance score of the model in this period (or in this scenario). Take the top three models with the highest score, output their control amount, and provide guidance to the secondary autonomous driving model in subsequent training.

[0077] Among them, the individual evaluation model mainly defines individual features (Individual Features) for describing the motion state of the ego vehicle in a period of time, including individual features based on the motion state of the ego vehicle, individual features based on the driving behavior of the ego vehicle, and individual features based on uncertainty.

[0078] The individual features based on the motion state of the ego vehicle include the speed , acceleration and jerk (i.e. , which describes the rate of change of acceleration) at time ; The individual features based on the driving behavior of the ego vehicle include the waiting time (Waiting Period, WP) and the difference between the actual speed and the lane speed limit ​Individual characteristics based on uncertainty include the percentage of time spent driving along lane lines. and trajectory outliers .

[0079] Among them, the waiting period This typically refers to the cumulative time a vehicle waits continuously at low speeds (or near a standstill). A speed threshold can be selected. ,when At that time, we considered the vehicle to be in a waiting state. Therefore, the waiting time... It can be defined as:

[0080]

[0081] in This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

[0082] The difference between actual speed and lane speed limit Specifically:

[0083] Assuming in each The lane speed limit at that time is Then in The velocity difference at time points can be written as:

[0084]

[0085] Calculate the average value for the entire trajectory:

[0086]

[0087] Percentage of time vehicles travel in lanes It mainly utilizes lane geometry information to calculate the vehicle's position. Lateral deviation at time (Refers to the distance between the vehicle and the lane centerline). Set an allowable deviation threshold. ,when At that time, the vehicle is considered to be "driving in its lane". Therefore, this indicator can be expressed as:

[0088]

[0089] This indicates the percentage of time a vehicle spends within its lane throughout the entire trajectory.

[0090] Trajectory outliers The design intent is to measure the deviation between the vehicle's trajectory and the overall average trajectory. Let the vehicle trajectory be denoted as... (two-dimensional trajectory) ). A number of cluster centers of all the trajectories of the agents in the first-level autonomous driving model library are obtained by clustering method, denoted as set . Trajectory outliers are defined as the average distance between the ego vehicle trajectory and all cluster centers:

[0091]

[0092] wherein represents the similarity between the whole trajectory and the cluster center measured by the Euclidean distance. Such a definition can reflect the deviation of the ego vehicle trajectory from the average trajectory in the first-level model library as a whole.

[0093] Based on these features, a set of individual features is constructed, the value of which in the entire observed trajectory changes over time . In order to ensure that the highest risk performance in the entire trajectory is captured, for each individual feature , the minimum value is calculated to ensure that the high risk appearing at a short moment is captured.

[0094]

[0095]

[0096]

[0097] represents the individual feature based on the ego vehicle motion state, represents the individual feature based on the ego vehicle driving behavior, represents the individual feature based on uncertainty.

[0098] After obtaining the minimum value of each feature in the prediction duration, the next step is to linearly combine these minimum values. A pre-set weight vector is introduced, wherein each element corresponds to the importance weight of a feature. The mathematical expression is:

[0099]

[0100] Here, “.” represents the dot product operation of the vector. That is, the minimum value of each feature at all time steps is multiplied by the corresponding weight, and then summed up to finally obtain the individual feature score of the ego vehicle.

[0101] In the social evaluation model, social features are first defined. These features are mainly used to describe the interactions between different driving agents. This invention adopts some widely studied and accepted safety agent indicators, including time head-down distance (THW), time to collision (TTC), deceleration required to avoid a collision (DRAC), and the time difference of conflict between the two vehicles. Indicators such as ).

[0102] Time Headway (THW) represents the time required for a following vehicle to reach the current position of the vehicle in front, converted from distance to time. It represents the time distance between the two vehicles, defined as the distance between them divided by the current speed of the following vehicle. Intuitively, THW reveals the "reaction" time the following vehicle has before colliding with the vehicle in front.

[0103] Time-to-Collision (TTC) is the time expected to pass before two road users collide, based on the current state of the road. TTC predicts the remaining time before a collision occurs between vehicles if their current speeds and directions of travel remain constant. The calculation typically relies on the relative speeds and distances between the vehicles, i.e., the expected time for the two vehicles to meet or collide in the current state.

[0104] The Deceleration Rate to Avoid Crash (DRAC) is essentially the minimum average delay required to avoid an accident at a given speed and distance between vehicles. DRAC quantifies the deceleration required to avoid a collision with a vehicle ahead or to the side in the current driving state. It reflects how quickly a vehicle must decelerate significantly to avoid a collision if it does not brake or adjust its trajectory in time.

[0105] The time difference to conflict point between the two vehicles. The time difference between two road users passing through a conflict point is a measure of their interaction. It represents the minimum time difference between the arrival times of two vehicles on their respective trajectories at the conflict point (a location where a collision may occur or a critical road intersection). This metric measures the degree to which two vehicles arrive at a potential conflict area synchronously: if the arrival times of the two vehicles at the conflict point are very close, it indicates a higher risk of collision during their interaction.

[0106] These indicators constitute a set of social characteristics. Each feature All with time In each pair of interacting vehicle trajectories All of the above can be calculated, denoted as To capture the highest risk that can occur throughout the interaction, we take the minimum value of each social feature in the calculation process, i.e., select the minimum value of the indicator throughout the time series. For each social feature , we calculate The purpose of this is to ensure that even if a risk occurs at a momentary moment (for example, a very low TTC or a very high DRAC), it can be captured and given enough weight in the subsequent score.

[0107]

[0108]

[0109]

[0110]

[0111] In the above formula, represents the collision time feature, represents the headway feature, represents the deceleration required to avoid a collision, represents the conflict time difference feature, and represent the coordinates of the ego vehicle and the interacting vehicle, respectively, and represent the speed of the ego vehicle and the interacting vehicle, respectively, represents the distance of the ego vehicle from the potential conflict point, represents the average speed of the ego vehicle to reach the potential conflict point, and represent the distance of the interacting vehicle from the potential conflict point and the average speed to reach the point.

[0112] After obtaining the minimum value of each social feature in the interaction between a pair of vehicles, a predefined weight vector is used to linearly combine these indicators. The formula is:

[0113]

[0114] where “.” represents the dot product operation of the vector. This process integrates each social feature according to its importance in safety risk assessment, resulting in an overall social risk score between a pair of vehicles.

[0115] The autonomous vehicle in the scenario , the social interaction between the vehicle and all other vehicles will generate a social score. Ultimately, the vehicle The overall risk score at the social level is obtained by summing all the social scores of other vehicles .

[0116] The final score expression of each autonomous driving model in the primary autonomous driving model library is as follows:

[0117]

[0118] Here, is the individual feature score of the autonomous driving vehicle, and the latter half is the cumulative social feature score of all the vehicles interacting with the ego vehicle. Not only the risk of a single vehicle is considered, but also the potential safety hazards generated by the interaction of all other vehicles are fully taken into account in the evaluation system. We select the top 3 primary autonomous driving models with the highest comprehensive score as the experts to guide the training of the secondary autonomous driving model at this time node.

[0119] The training of the secondary autonomous driving model still uses the conventional SAC reinforcement learning algorithm, but the most core difference lies in the design of the reward function, which integrates the information from the behavior demonstration of the experts (i.e., the selected three primary models). Through the combination of expert demonstration and original environmental feedback, the secondary autonomous driving model can learn the optimal strategy more efficiently.

[0120] First, let the original environmental reward be , which represents the basic return obtained by the agent after interacting with the environment at the current time, and is used to ensure that the training target is consistent with the task requirements. Then, the invention introduces the outputs of two discriminators: the individualized behavior discriminator and the individualized transition discriminator . The comprehensive reward function of the secondary autonomous driving model can be expressed as:

[0121]

[0122] Wherein, the meanings of various symbols are as follows:

[0123] represents the original environmental reward, i.e., the immediate reward received by the agent from the environment at the current time. It is the basic feedback signal of the task, reflecting the performance of the agent in the environment.

[0124] is a weight hyperparameter used to balance the influence between the environmental reward and the information from the expert demonstration (reward adjustment item obtained through the discriminator), adjusting the contribution size of the demonstration information in the reward, and ensuring a good compromise between demonstration guidance and environmental feedback.

[0125] represents the output function of the individualized behavior discriminator, and the parameters are determined by denotes the input is with , denotes the state space of the vehicle , denotes the action taken by the vehicle in state . The output is a value in the interval , denoting the degree of matching of the current state-action pair with the expert individualized demonstration data. If the matching is good, the value is larger; if deviating from the demonstration, the value is smaller.

[0126] denotes the output function of the individualized transition discriminator, whose parameters are denoted by . The input is , and , denotes the next state space reached by the ego vehicle after the action is executed. It indicates whether the state transition caused by the current state-action pair is consistent with the local transition pattern observed in the expert demonstration. The higher the output, the more the state transition matches the demonstration, thus enhancing the weight of the demonstration information in the reward; otherwise, it reduces the positive contribution of the reward.

[0127] denotes the logarithmic penalty term on the output of the behavior discriminator. When is high (i.e., the current behavior is close to the expert demonstration), is lower, and its logarithmic value is more negative, so the deduction to the reward after being multiplied by the negative sign is smaller; if is low (i.e., the behavior deviates from the expert demonstration), is larger, and the deduction part will increase, thus punishing the behavior deviating from the demonstration.

[0128] Under this design, guarantees that the model can eventually complete the basic driving task, while the reward modification brought by the discriminator output further utilizes the expert demonstration information: the individualized behavior discriminator is used to judge whether the current state-action combination "imitates" the expert; the individualized transition discriminator measures whether the state transition caused by the action also approximates the local pattern demonstrated by the expert. The agent will only receive less punishment when it is relatively consistent with the expert demonstration, and if it deviates from the demonstration, it will receive greater deduction in the reward level, thus guiding the strategy of the secondary autonomous driving model to approach the expert demonstration.

[0129] It can be seen that the application firstly selects three models with the highest comprehensive scores from the first-level automatic driving model library as "experts" in the training process of the second-level automatic driving model through the parallel space-time prediction module. At each training time, the actions given by the three expert models are respectively brought into the above reward function for evaluation; the state-action pair with the highest reward is selected and stored in the experience replay pool of SAC. The second-level automatic driving model takes into account the exploration of the real environment and the reference of the expert demonstration, and can more quickly and stably converge to the optimal solution with a high-level driving strategy.

[0130] As for the specific training process, the application follows the conventional practice of SAC reinforcement learning, and both the first-level and second-level automatic driving models can be trained using the SAC algorithm. SAC is an advanced reinforcement learning algorithm that can not only stably learn the strategy, but also effectively balance the exploration and utilization of the agent in the exploration process by controlling the action entropy. In the application, the SAC algorithms used to train the first-level and second-level automatic driving models are basically the same except for the reward function. The SAC algorithm used in the application has three modules, namely the state input module, the decision module and the action output module. Among them, the state input module is mainly responsible for receiving driving environment information and vehicle state information, including vehicle coordinates, speed, acceleration, road environment information and other vehicle states (position, speed, acceleration, etc.). The intelligent decision module mainly uses SAC algorithm as the core, combines Actor (actor) and Critic (critic) two networks, and updates the strategy by constantly interacting with the environment. The action output module mainly generates vehicle steering wheel angle, throttle, brake pedal and other control quantities based on the information processed by the decision module. These output actions directly act on the lateral and longitudinal control of the vehicle.

[0131] Among them, the Actor network includes several fully connected layers, which calculate the mean and standard deviation of steering, acceleration and braking. The activation function mainly has two parts, the steering part uses tanh function to normalize to [-1, 1], and the acceleration and braking part uses sigmoid function to normalize the output to [0, 1] range, and the standard deviation is also normalized by sigmoid function. The design of the Critic network aims to evaluate the value of action selection in the current state, and its input also comes from the multi-dimensional state information of the vehicle. The network estimates the state value through a multi-layer fully connected structure (including using GELU activation function). In the parameter update, a soft update strategy is used to gradually move the target network closer to the current network, ensuring the stability of the training process.

[0132] Training process and algorithm details:

[0133] Experience Acquisition and Network Update: In the simulation environment, the vehicle behaves according to the current policy and records empirical tuples of state, action, reward, and next state. Using this empirical data, the Critic network is updated by minimizing the mean squared error (MSE) of the Q-value estimation, while the Actor network's parameters are updated using the Adam gradient descent method.

[0134] Exploration and Entropy Regularization: A core feature of the SAC algorithm is that it encourages exploration by maximizing action entropy. This allows the model to explore the state space more fully in the early stages, while improving the overall policy convergence by balancing exploration and utilization.

[0135] In one embodiment of the present invention, the SAC algorithm model includes a Network and four Network, four The networks are respectively state value estimation and Networks, and action-state value estimation and network. The network input is the state space defined in the patent (vehicle coordinates, speed, acceleration; road coordinates; coordinates, speed, acceleration of surrounding vehicles), and the output is the action probability distribution parameters defined in the patent (steering wheel angle and accelerator and brake pedals). The network input is the state space defined in the patent, and the output is the value of the state and action-state pairs (i.e., the expected reward value in this driving scenario). Network output is An estimate representing the value of a state. The network output is Represents the action-state pair value.

[0136] In the SAC algorithm, if The more the network output action can enable The greater the value, the better. The value of network output actions The more accurate (according to the Bellman equation), the better. Whether it is accurate depends on The more accurate the result, the better. Network output state value The more accurate, the better.

[0137] Experience pool generation process: Given a state ,pass The network obtains the parameters of the probability distribution of all actions (mean and variance of a Gaussian distribution), and then samples the actions according to the probability distribution. Then Input into the environment and get and This gives us a : Then Add to experience pool The purpose of the experience pool is to eliminate... The correlation between actions is determined because actions are usually strongly correlated in reinforcement learning. These correlations are broken down and placed into an experience pool, and then randomly selected from the experience pool during neural network training. A batch of people were selected This makes neural network training more stable and robust.

[0138] Network update process:

[0139] From the experience pool Data collected from China conduct Network updates, For example, based on the optimal Bellman equation, As a state The true value estimate will be based on the actual actions taken. of Value as state Predictive value estimation, and finally use As Functions for neural networks and Conduct training. (Take...) This means from the experience pool Take one The data is averaged, that is: .

[0140] Network update process:

[0141] From the experience pool Data collected from China conduct Network updates, For example, the state value is estimated using the following formula, i.e., the following formula is used as... The actual value output by the network:

[0142]

[0143] : double network takes the minimum value, two sets of networks are usually maintained to alleviate the overestimation problem. Here, the smaller value of the two is taken, which makes the target more conservative and stable.

[0144] : double is the log probability of the current policy in state down-sampled to action . After multiplying by the coefficient , it makes the target value contain an "encourage exploration" entropy term: . In this way, the policy not only pursues a high value, but also maintains a certain randomness to avoid converging to a suboptimal deterministic policy too early.

[0145] : double : state value is defined as the expectation of the value of the action sampled along the current policy in state .

[0146] The output of the network is used as the predicted value, and finally is used as the function to train the neural network .

[0147] The update process of the network:

[0148] : double : double : double . For each , the current is resampled (or all discrete actions are enumerated directly).

[0149] : double : double : double : double : double

[0150] : double ​, the gradient can be automatically derived by the deep learning framework (such as PyTorch / TensorFlow) according to the above formula.

[0151] With Adam or SGD, the learning rate is , Generally , the update frequency can be not completely synchronized (such as updating every step, updating every few steps), to ensure that it is more stable to learn first.

[0152] The application also provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., which stores a computer program, and the program is executed by a processor to realize the corresponding function. The computer readable storage medium of the embodiment is executed by the processor to realize the training method of the adaptive automatic driving model for complex road scenes of the method embodiment.

[0153] In summary, the application integrates multiple models with different decision-making characteristics into a two-level automatic driving model, which has the following advantages: ① Improve multi-scene adaptability: In different driving environments such as congestion, smoothness, high speed or urban roads, the two-level automatic driving model can flexibly "borrow" the first-level model that best fits the current scene, thereby achieving better performance in safety or efficiency. ②Rich decision-making candidate space: Multiple models accumulate differentiated decision-making experience, through parallel prediction and scoring, multiple driving strategies can be quickly evaluated for their pros and cons at the same time, and the optimal strategy is selected for execution. ③Shorten the training period and reduce the difficulty: Compared with training a single model that covers all working conditions from scratch, the two-level model only needs to further strengthen on the basis of multiple first-level models (equivalent to "expert demonstration"), so it can converge faster and avoid unnecessary repeated exploration.

[0154] It should be noted that, according to the needs of implementation, each step / component described in the present application can be split into more steps / components, or two or more steps / components or part of the operation of the steps / components can be combined into a new step / component to achieve the purpose of the present application.

[0155] The size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0156] It is to be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

Claims

1. A training method for an adaptive autonomous driving model for complex road scenarios, characterized in that, Includes the following steps: Multiple pre-built Level 1 autonomous driving models are trained, and the reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input to the Level 1 autonomous driving model is road environment information, vehicle status information and other vehicle status information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. The training process involves comprehensively evaluating multiple Level 1 autonomous driving models and selecting the three models with the highest comprehensive scores as expert models. The specific process of comprehensive evaluation includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, and measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model. The two scores are then weighted and summed according to preset weights to obtain the comprehensive score. The pre-built Level 2 autonomous driving model is trained. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. Specifically, during training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action, and receives greater deduction in the reward level if it deviates from the demonstration action. After training, the strategy of the Level 2 autonomous driving model moves closer to the demonstration action of the expert model.

2. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the expert model's state-action. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the expert model's state transition based on the current and subsequent vehicle state-action.

3. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The weights of safety rewards and driving efficiency rewards in the reward function of the Level 1 autonomous driving model are specifically obtained by sampling two Gaussian distributions.

4. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The individual evaluation model specifically scores the individual characteristics of the vehicle, which include the vehicle's motion state, driving behavior, and uncertainty-based individual characteristics. The vehicle's motion state includes speed, acceleration, and rate of change of acceleration. Autonomous driving behavior includes waiting time, the difference between actual speed and lane speed limit; uncertain individual characteristics include the percentage of time spent driving along lane lines, and trajectory anomalies.

5. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 4, characterized in that, The scoring of individual characteristics is specifically obtained by multiplying the minimum value of each individual characteristic at all time steps by its corresponding weight and then summing the results.

6. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The social evaluation model scores drivers based on interaction metrics between different drivers. These metrics include head-on distance, collision time, deceleration required to avoid a collision, and the time difference between the interaction between the two vehicles.

7. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 6, characterized in that, The scoring of the social evaluation model is as follows: after obtaining the minimum value of each interaction index in a pair of vehicle interactions, the scores are obtained by weighted summation using a predefined weight vector.

8. A training system for an adaptive autonomous driving model for complex road scenarios, characterized in that, include: The Level 1 autonomous driving model training module is used to train multiple pre-built Level 1 autonomous driving models. The reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input to the Level 1 autonomous driving model is road environment information, vehicle status information and other vehicle status information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. The parallel spatiotemporal prediction module is used to comprehensively score multiple trained Level 1 autonomous driving models and select the three models with the highest comprehensive scores as expert models. The specific process of comprehensive scoring includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, and measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model. Then, the two scores are weighted and summed according to preset weights to obtain the comprehensive score. The Level 2 autonomous driving model training module is used to train a pre-built Level 2 autonomous driving model. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. During training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action, and receives greater deduction in reward if it deviates from the demonstration action. After training, the strategy of the Level 2 autonomous driving model moves closer to the demonstration action of the expert model.

9. The training system for an adaptive autonomous driving model for complex road scenarios according to claim 8, characterized in that, The reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the expert model's state-action. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the expert model's state transition based on the current and subsequent vehicle state-action.

10. A computer storage medium, characterized in that, It contains a computer program that can be executed by a processor, which performs the training method for an adaptive autonomous driving model for complex road scenarios as described in any one of claims 1-7.

Images