Self-adaptive automatic driving method and system for complex road scene

Abstract

The invention discloses a complex road scene-oriented self-adaptive automatic driving model training method and system, and the method comprises the steps: training a plurality of primary automatic driving models, and enabling reward functions of different models to correspond to different training strategy tendencies; performing comprehensive scoring on the plurality of trained first-level automatic driving models, and screening out three models with the highest comprehensive scores as expert models; a secondary automatic driving model is trained, data in a training set are input into three expert models, then outputs of the three expert models are substituted into a reward function of the secondary automatic driving model for calculation, and the secondary automatic driving model is subjected to less punishment under the condition of keeping relatively consistent with expert demonstration actions. And if the second-level automatic driving model deviates from the demonstration action, the second-level automatic driving model is greatly deducted from the award level, and the strategy of the trained second-level automatic driving model approaches the demonstration action of the expert model. According to the method, the multi-scene adaptability of automatic driving can be improved, decision candidate space is enriched, the training period is shortened, and the difficulty is reduced.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to an adaptive autonomous driving method and system for complex road scenarios. Background Technology

[0002] With the rapid development of intelligent network technology, the global automotive intelligence process is accelerating. Intelligent connected vehicles, with their significant advantages in reducing safety accidents, environmental pollution, traffic congestion, and energy consumption, are becoming an important direction for the development of the automotive industry both domestically and internationally. Autonomous vehicles travel along predetermined routes, but the real traffic environment is not static. Roads typically have a variety of traffic participants whose real-time states are constantly changing. Therefore, autonomous vehicles need to dynamically predict the surrounding environment and implement movement interaction strategies such as following, avoiding, and changing lanes based on different traffic participants. This not only helps to adjust driving strategies in real time but also provides guidance for trajectory planning, ensuring that the vehicle can complete driving tasks as efficiently and safely as possible. Autonomous driving systems can be seen as a combination of traditional automotive technology and modern high-performance computing technology, encompassing environmental perception, map localization, prediction, decision-making, planning, and control execution. At the perception and localization level, autonomous vehicles are equipped with various sensors such as LiDAR, millimeter-wave radar, and cameras, and through sensor fusion technology combined with high-precision maps, they comprehensively perceive the static and dynamic environment around the vehicle, achieving high-precision localization and global path planning.

[0003] At the decision-making and planning level, the system comprises three aspects: trajectory prediction of traffic participants, vehicle behavior decision-making, and trajectory planning, to generate a safe, collision-free driving trajectory that meets specific constraints. Finally, at the control level, the system utilizes vehicle dynamics control theory to output control commands such as accelerator, brake, and steering wheel angles, achieving precise control of vehicle movement through an electronically controlled chassis, ensuring that the vehicle's actual driving trajectory accurately tracks the planned trajectory. Currently, behavioral decision-making methods are mainly divided into rule-based methods and data-based learning methods. Rule-based decision-making schemes rely on preset traffic rules and driving experience. However, this method cannot cover all possible scenarios because real-world traffic scenarios are often extremely complex, and dynamic traffic participants may take completely opposite actions in an instant. Therefore, when facing unknown scenarios, rule-based decision-making systems may encounter limitations in processing power and may even require driver intervention to cope with complex situations. With the development of artificial intelligence technology, especially its success in areas such as image recognition, financial market analysis, and natural language processing, autonomous driving technology offers more solutions. Among them, reinforcement learning methods enable intelligent agents to make more rational behavioral decisions based on environmental changes through automatic interaction and continuous optimization. By combining the powerful representational capabilities of deep learning neural networks, reinforcement learning provides an important approach to realizing autonomous driving decision-making technology, especially in dealing with high-dimensional and complex problems, and can significantly improve the intelligence level of decision-making systems.

[0004] The behavioral decision-making subsystem is a core component of complex and dynamic urban traffic environments. Its main tasks are to ensure the safety of autonomous vehicles, comply with traffic regulations, and provide the constraint information needed for smooth path and speed optimization. Fast and accurate behavioral decisions can effectively reduce accidents and are of great significance for improving personal safety. Having driver-like decision-making capabilities is a key indicator of a vehicle's high level of intelligence, directly determining the level of the autonomous driving system. Depending on the architecture, the input information of the decision-making system may include structured road information such as the current lane, merging lanes, and intersections, traffic sign information, obstacle states, and trajectory prediction information of surrounding traffic participants. Through comprehensive analysis of the current environment and traffic participant information, the system can output driving commands, including following, free lane changing, forced lane changing, and overtaking.

[0005] A core contradiction in the development of autonomous driving systems is that models trained on fixed scenarios perform exceptionally well in simple environments but struggle to adapt to the complex road conditions of the real world. When road types and traffic flow are relatively uniform and pedestrian and vehicle behavior is predictable, manually designed supervisory signals and reward functions can effectively train high-performing driving models. However, when deployment scenarios expand to dynamically changing real-world roads, the model's generalization ability significantly degrades. This essentially stems from a structural contradiction between the training paradigm and real-world needs—the manually designed reward function faces a fundamental dilemma in pursuing a dynamic balance between safety and efficiency: high-frequency, routine scenarios require the model to adopt aggressive strategies to improve traffic efficiency, while low-frequency, high-risk scenarios (such as sudden obstacles or extreme weather) require absolutely conservative safety strategies. A single reward function cannot cover these scenario-sensitive policy requirements. Even more critically, the probability of critical safety events occurring on real roads is often less than one in ten thousand, resulting in a severe lack of coverage of high-risk scenarios in training data, causing a sharp decline in the model's decision-making ability when facing unexpected situations.

[0006] This contradiction is further exacerbated at the model architecture level: when a model reaches its performance limit through optimization in a specific scenario (such as highway cruising), its policy network parameters become fixed and path-dependent, making it difficult to quickly switch policy tendencies when the scenario changes abruptly (such as turning into congested urban roads). Although introducing human expert supervision can accelerate model convergence in specific scenarios, the cost of expert annotation will increase exponentially when facing the needs of the entire scenario, which includes hundreds of road types and thousands of traffic participants. More importantly, human driving behavior itself has a 23% decision-making error rate, and the supervision signals it provides may contain suboptimal or even erroneous patterns. This unreliability will be further amplified in cross-regional and cross-cultural scenarios. Summary of the Invention

[0007] The main objective of this invention is to provide an efficient, safe, and reliable adaptive autonomous driving method and system for complex road scenarios in complex dynamic environments.

[0008] The technical solution adopted in this invention is: A training method for an adaptive autonomous driving model for complex road scenarios is provided, comprising the following steps: Multiple pre-built Level 1 autonomous driving models are trained. The reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input of the Level 1 autonomous driving model is road environment information, vehicle and other vehicle state information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. The training process involves comprehensively evaluating multiple Level 1 autonomous driving models and selecting the three models with the highest comprehensive scores as expert models. The specific process of comprehensive evaluation includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, and measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model. The two scores are then weighted and summed according to preset weights to obtain the comprehensive score. The pre-built Level 2 autonomous driving model is trained. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. Specifically, during training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action, and receives greater deduction in the reward level if it deviates from the demonstration action. After training, the strategy of the Level 2 autonomous driving model moves closer to the demonstration action of the expert model.

[0009] Following the above technical solution, the reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the state-action of the expert model. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the state transition of the expert model based on the current and subsequent vehicle state-action.

[0010] Following the above technical solution, the weights of safety rewards and driving efficiency rewards in the reward function of the Level 1 autonomous driving model are specifically obtained by sampling two Gaussian distributions.

[0011] Following the above technical solution, the individual evaluation model specifically scores the individual characteristics of the vehicle. These individual characteristics include the vehicle's motion state, driving behavior, and uncertainty-based individual characteristics. The vehicle's motion state includes speed, acceleration, and the rate of change of acceleration. The vehicle's driving behavior includes waiting time and the difference between actual speed and lane speed limit. Uncertainty-based individual characteristics include the percentage of time spent driving along lane lines and trajectory anomalies.

[0012] Following the above technical solution, the scoring of individual features is specifically as follows: multiply the minimum value of each individual feature at all time steps by the corresponding weight, and then sum them up.

[0013] Following the above technical solution, the social evaluation model scores based on the interaction indicators between different driving entities. These interaction indicators include head-on distance, collision time, deceleration required to avoid a collision, and the time difference of conflict between the two vehicles.

[0014] Following the above technical solution, the scoring of the social evaluation model is as follows: after obtaining the minimum value of each interaction index in a pair of vehicle interactions, a weighted sum is obtained using a predefined weight vector.

[0015] This invention also provides a training system for an adaptive autonomous driving model for complex road scenarios, comprising: The Level 1 autonomous driving model training module is used to train multiple pre-built Level 1 autonomous driving models. The reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input of the Level 1 autonomous driving model is road environment information, vehicle and other vehicle state information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. The parallel spatiotemporal prediction module is used to comprehensively score multiple trained Level 1 autonomous driving models and select the three models with the highest comprehensive scores as expert models. The specific process of comprehensive scoring includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model, and then weighting and summing the two scores according to preset weights to obtain the comprehensive score. The Level 2 autonomous driving model training module is used to train a pre-built Level 2 autonomous driving model. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. During training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action, and receives greater deduction in reward if it deviates from the demonstration action. After training, the strategy of the Level 2 autonomous driving model moves closer to the demonstration action of the expert model.

[0016] Following the above technical solution, the reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the state-action of the expert model. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the state transition of the expert model based on the current and subsequent vehicle state-action.

[0017] The present invention also provides a computer storage medium storing a computer program executable by a processor, the computer program executing the training method for an adaptive autonomous driving model for complex road scenarios described in the above technical solution.

[0018] The beneficial effects of this invention are as follows: This invention proposes an innovative framework of hierarchical expert distillation: First, multiple different Level 1 autonomous driving models are trained normally (e.g., trained based on fixed scenarios and simple reward functions), and then a model library is formed based on these models; then, the decisions of each expert are integrated, and the scenario adaptability is encoded into the Level 2 autonomous driving model through knowledge distillation. This breakthrough in the architecture of "professional division of labor - intelligent scheduling" provides a new technical path for solving the generalization dilemma of autonomous driving. Since each Level 1 autonomous driving model has a specific strategy tendency (e.g., emphasizing safety or emphasizing efficiency), it is difficult for a single model to maintain the best performance in all scenarios. By integrating multiple models with different decision characteristics into a Level 2 autonomous driving model, the following advantages are generated: (1) Improved multi-scenario adaptability: In different driving environments such as congestion, smooth traffic, highways or urban roads, the Level 2 autonomous driving model can flexibly "borrow" the Level 1 model that best fits the current scenario, thereby achieving better performance in terms of safety or efficiency. (2) Enriched decision candidate space: Multiple models have accumulated differentiated decision experience, and through parallel prediction and scoring, the advantages and disadvantages of multiple driving strategies can be quickly evaluated at the same time, and the optimal strategy can be selected for execution. (3) Shorten the training cycle and reduce the difficulty: Compared with training a single model covering all working conditions from scratch, the second-level model only needs to be further strengthened on the basis of multiple first-level models (equivalent to "expert demonstrations"), so it can converge faster and avoid unnecessary repeated exploration.

[0019] Furthermore, this invention fully leverages the advantage of easily training autonomous driving models with specific styles based on simple reward functions. By training multiple Level 1 autonomous driving models for multiple simple reward functions respectively, and combining this invention with the proposed Gaussian sampling-based diversified reward function design method, comprehensive coverage of optimal driving strategies across all scenarios is achieved.

[0020] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an adaptive autonomous driving method for complex road scenarios according to an embodiment of the present invention; Figure 2 This is a flowchart of the training process for a Level 1 autonomous driving model according to an embodiment of the present invention; Figure 3 This is a flowchart of the training process for a Level 2 autonomous driving model according to an embodiment of the present invention; Figure 4 This is a diagram illustrating the training process of a Level 1 autonomous driving model according to an embodiment of the present invention; Figure 5 This is a diagram illustrating the training process of a Level 2 autonomous driving model according to an embodiment of the present invention; Figure 6 This is a flowchart of a parallel spatiotemporal prediction process according to an embodiment of the present invention; Figure 7 This is a roadmap of parallel spatiotemporal prediction technology according to an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] It should be noted that the illustrations provided in the embodiments of the present invention are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0025] In this invention, it should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used only for descriptive and distinguishing purposes and should not be construed as indicating or implying relative importance.

[0026] Furthermore, it should be noted that the features of the various embodiments of the present invention can be combined or integrated in whole or in part, and as those skilled in the art will understand, they can interact and operate in different ways. Each embodiment can be implemented independently of each other or in association with one another.

[0027] To adapt to dynamic and ever-changing driving environments, this invention first constructs multiple autonomous driving models with different driving styles based on a relatively simple reward function and a general deep reinforcement learning (DRL) method, collectively referred to as Level 1 autonomous driving models. Because the reward function of this model is relatively simple to design, the training difficulty is also relatively low, achieving good autonomous driving performance in specific scenarios. Subsequently, these Level 1 autonomous driving models are incorporated into a model library, and a more complex Level 2 autonomous driving model is further constructed on this basis. The Level 2 autonomous driving model can fully utilize the advantages of each Level 1 model to achieve optimal performance in different environments. Therefore, this invention aims to solve the following two core problems: ① How to construct Level 1 autonomous driving models that cover as many driving styles as possible; ② How to utilize the aforementioned model library to construct an autonomous driving model with stronger adaptability that can maintain optimal performance in multiple scenarios.

[0028] In one embodiment of the present invention, such as Figure 1 As shown, the training method for adaptive autonomous driving models for complex road scenarios includes the following steps: S1. Train multiple pre-built Level 1 autonomous driving models. The reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input of the Level 1 autonomous driving model is road environment information, vehicle and other vehicle state information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. S2. A comprehensive score is given to the multiple trained Level 1 autonomous driving models, and the three models with the highest comprehensive scores are selected as expert models. The specific process of comprehensive scoring includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, and measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model. The two scores are then weighted and summed according to preset weights to obtain the comprehensive score. S3. Train the pre-built Level 2 autonomous driving model. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. Specifically, during training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action. If it deviates from the demonstration action, it will be subject to greater deduction in the reward level. After training, the strategy of the Level 2 autonomous driving model will move closer to the demonstration action of the expert model.

[0029] Furthermore, the reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the state-action of the expert model. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the state transition of the expert model based on the current and subsequent vehicle state-action.

[0030] Furthermore, the weights of safety rewards and driving efficiency rewards in the reward function of the Level 1 autonomous driving model are specifically obtained by sampling two Gaussian distributions.

[0031] Furthermore, the individual evaluation model specifically scores the individual characteristics of the vehicle, which include the vehicle's motion state, driving behavior, and uncertainty-based individual characteristics. The vehicle's motion state includes speed, acceleration, and rate of change of acceleration; the vehicle's driving behavior includes waiting time and the difference between actual speed and lane speed limit; and the uncertainty-based individual characteristics include the percentage of time spent driving along lane lines and trajectory anomalies.

[0032] Furthermore, the scoring of individual characteristics is specifically obtained by multiplying the minimum value of each individual characteristic at all time steps by its corresponding weight and then summing the results.

[0033] Furthermore, the social evaluation model scores drivers based on interaction metrics between different drivers, including head-on distance, collision time, deceleration required to avoid a collision, and the time difference of the interaction between the two vehicles.

[0034] Furthermore, the scoring of the social evaluation model is as follows: after obtaining the minimum value of each interaction index in a pair of vehicle interactions, a weighted sum is obtained using a predefined weight vector.

[0035] To implement the above method, the training system for adaptive autonomous driving models oriented towards complex road scenarios in this embodiment of the invention includes: The Level 1 autonomous driving model training module is used to train multiple pre-built Level 1 autonomous driving models. The reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input of the Level 1 autonomous driving model is road environment information, vehicle and other vehicle state information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. The parallel spatiotemporal prediction module is used to comprehensively score multiple trained Level 1 autonomous driving models and select the three models with the highest comprehensive scores as expert models. The specific process of comprehensive scoring includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model, and then weighting and summing the two scores according to preset weights to obtain the comprehensive score. The Level 2 autonomous driving model training module is used to train a pre-built Level 2 autonomous driving model. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. During training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action, and receives greater deduction in reward if it deviates from the demonstration action. After training, the strategy of the Level 2 autonomous driving model moves closer to the demonstration action of the expert model.

[0036] Furthermore, the reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the state-action of the expert model. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the state transition of the expert model based on the current and subsequent vehicle state-action.

[0037] Specifically, when constructing a Level 1 autonomous driving model, the common SAC deep reinforcement learning training paradigm can be adopted, and various autonomous driving agents can be designed and trained for different "policy preferences." The inputs to each agent include: the vehicle's coordinates, speed, and acceleration; road coordinates; and the coordinates, speed, and acceleration of surrounding vehicles. The outputs are the vehicle's control variables (steering wheel angle, accelerator, and brake pedals). The "policy preference" here mainly reflects the agent's different emphasis on the trade-off between safety and efficiency (or vehicle speed). For example, when the reward function has a larger weight for speed and efficiency, and a relatively smaller weight for safety, the agent tends to pursue higher speeds; conversely, when safety is dominant, the agent prioritizes driving safety, thus maintaining a relatively conservative driving strategy.

[0038] "Strategic bias" is mainly reflected in the different trade-offs that agents make between safety and efficiency (or speed). Specifically, when the weight of speed and efficiency is large and the weight of safety is relatively small in the reward function, the agent tends to drive at a higher speed; while when the weight of safety is dominant, the agent will prioritize driving safety and thus adopt a more conservative driving strategy.

[0039] Based on the traditional deep reinforcement learning training process, such as Figure 2 , 4As shown, this invention proposes a method for designing diversified reward functions based on Gaussian sampling to generate agent models with different driving styles. First, the simulation environment is initialized, containing the autonomous vehicle and several environmental vehicles, with highways, roundabouts, or intersections randomly selected as road scenarios. Then, the Actor and Critic network parameters of the SAC agent are initialized, and an experience pool is established based on the state space and action space structure. In the initial training phase, the agent controls the interaction between the autonomous vehicle and the environment with a stochastic strategy for 10,000 steps. After each interaction, the reward value of the current state-action pair is calculated based on the reward function of the Level 1 autonomous driving model, and the state, action, reward, and next state are stored in the experience pool. After the initial 10,000 steps, the training phase begins. Every 5 interactions, 1024 experience tuples are randomly sampled from the experience pool for network training: the Critic network is updated by minimizing the mean squared error (MSE) of the Q-value estimation, while the Actor network parameters are updated using the Adam optimizer. New experiences are continuously added to the experience pool during training until its capacity reaches its limit. A total of 200,000 steps are trained, and the model parameters are finally saved as the Level 1 autonomous driving policy model.

[0040] Specifically, the reward function for autonomous driving tasks can be broken down into the following two parts: (1) Driving efficiency reward

[0041] Indicators used to measure efficiency-related metrics such as vehicle speed, distance traveled, and arrival time at the destination.

[0042] (2) Security rewards

[0043] It is used to punish unsafe behaviors such as collisions, sudden braking, and insufficient distance from the vehicle in front.

[0044] The comprehensive reward can be expressed as:

[0045] in, and The weights are assigned to efficiency and safety, respectively. To reflect different decision preferences among multiple agents, this invention sets two Gaussian distributions: and This is used to sample the weights for efficiency and safety. and These are used to control the dispersion of the sampled weights.

[0046] The original weights are obtained by sampling from these two Gaussian distributions respectively. and Then, normalization is needed to map the Gaussian distribution to a comparable and stable range. A larger standard deviation of the Gaussian distribution means higher dispersion in the weight sampling, resulting in more diverse policy tendencies within the same batch of agents. However, this also means more sampling and training iterations are required to obtain uniform style samples. Normalization formula:

[0047] like Figure 3 , 5 As shown, the construction of the Level 2 autonomous driving model is a new model formed by integrating multiple Level 1 autonomous driving models with different "policy tendencies" based on the original Level 1 autonomous driving model. First, the simulation environment is initialized, containing the autonomous vehicle and several environmental vehicles, with highways, roundabouts, or intersections randomly selected as road scenarios. Then, the Actor and Critic network parameters of the SAC agent are initialized, and an experience pool is constructed based on the state space and action space structure. In the initial training phase, the autonomous vehicle interacts with the environment for 10,000 steps under the control of the SAC agent using a random policy. After each step, the parallel spatiotemporal prediction module is invoked to select three expert models from the Level 1 autonomous driving model library, and the current state is input into these three models to obtain the corresponding expert behaviors. Subsequently, based on the reward function of the Level 2 autonomous driving model, the reward value corresponding to each expert behavior is calculated, and the highest reward is selected as the final reward for the current step. The experience tuple consisting of the state, action, final reward, and next state is stored in the experience pool. After completing 10,000 steps, the training phase begins. Every 5 interaction steps, 1024 experience tuples are randomly sampled from the experience pool. The Critic network is updated by minimizing the mean squared error (MSE) of the Q-value estimation, and the Actor network is updated using the Adam optimizer. Newly generated experiences are continuously added to the experience pool until its capacity is reached. The entire training process lasts for 200,000 steps. After training, the model parameters are saved as a Level 2 autonomous driving policy model.

[0048] The inputs and outputs of the Level 2 autonomous driving model are identical to those of the Level 1 model. They receive the vehicle's coordinates, speed, acceleration, road environment information, and other vehicle states (position, speed, acceleration, etc.). The output remains decisions regarding steering wheel angle, accelerator, and brake pedal control. However, unlike the Level 1 model, the Level 2 model no longer relies solely on a single strategy for training. Instead, it learns from multiple pre-trained models to achieve superior driving performance in more complex environments. While the Level 1 model uses a conventional "action-feedback" experience sampling method for training, the Level 2 model, aided by the parallel spatiotemporal prediction module proposed in this invention, selects the best-performing models from multiple Level 1 models as "experts" and uses their information to guide its own training. Level 1 autonomous driving models are typically trained independently for specific trade-offs (e.g., prioritizing safety or efficiency); the Level 2 model integrates multiple driving styles, resulting in greater adaptability and stability in more complex or variable driving environments. Level 1 models are often built using a "single-style" deep reinforcement learning approach, while Level 2 models implement a "multi-model guidance + self-learning" model. During their training process, they receive expert scoring and guidance, making them more adaptable to changes in the environment.

[0049] The core innovation of this invention lies in the introduction of a "parallel spatiotemporal prediction module". By comprehensively evaluating the driving behavior of each model in the Level 1 autonomous driving model library over a period of time in the future, the "individual score" and "social score" of each model are obtained, and then the three best-performing models are selected as experts in the training stage of the Level 2 autonomous driving model.

[0050] like Figure 6 , 7 As shown, the technical approach and main workflow of the parallel spatiotemporal prediction module are as follows: (1) Start all Level 1 autonomous driving models in parallel: Under the same environmental information (including lane structure, lane boundaries, position, speed, acceleration, etc. of the vehicle and surrounding vehicles), all models in the Level 1 autonomous driving model library are run in parallel.

[0051] (2) Iteratively update the future time steps: ① For each first-level model, firstly, based on the vehicle's and environment's states, use the control quantities (direction, throttle, brake, etc.) output by the model to update the vehicle's coordinates and motion state in the next time step. ② Simultaneously, based on the conventional lateral and longitudinal driver model (IDM+MOBIL), predict and update the motion state of surrounding vehicles in the next time step. ③ Input the updated environment state back into these first-level models, repeating this process for approximately 20 time steps (20 steps are used as an example in this invention; the more steps, the more accurate the prediction, but the computational cost will also increase). The lateral and longitudinal driver model (IDM+MOBIL) constructed in this invention is a conventional approach and not the core innovation of this invention, so it will not be elaborated here.

[0052] (3) Scoring is based on the "individual evaluation model" and the "social evaluation model": Individual evaluation model: focuses on measuring the vehicle's own driving performance, including indicators such as speed, acceleration, jerk, waiting time, lane-keeping ratio, and trajectory deviation; Social evaluation model: focuses on measuring the safety of interaction with other vehicles, including headway (THW), time to collision (TTC), deceleration required to avoid a collision (DRAC), and the time difference of conflict between the two vehicles ( Indicators such as [list of indicators]. Throughout the prediction period, in order to capture the highest possible risk conditions, the minimum or maximum values ​​of each safety indicator (based on risk characteristics) are statistically analyzed to obtain a comprehensive score for the vehicle in both "individual" and "social" aspects.

[0053] (4) Select the three highest-scoring models as experts: Sum the comprehensive scores (individual + social) of each Level 1 autonomous driving model to obtain the overall performance score of the model in that time period (or scenario). Take the three models with the highest scores, output their control variables, and provide guidance for the Level 2 autonomous driving model in subsequent training.

[0054] The individual evaluation model mainly defines individual features to describe the vehicle's motion state over a period of time. These features include individual features based on the vehicle's motion state, individual features based on the vehicle's driving behavior, and individual features based on uncertainty.

[0055] Individual characteristics based on the vehicle's motion state include Speed ​​at any moment acceleration With jerk (i.e.) This describes the rate of change of acceleration; individual characteristics based on autonomous driving behavior include waiting period (WP) and the difference between actual speed and lane speed limit. Individual characteristics based on uncertainty include the percentage of time spent driving along lane lines. and trajectory outliers .

[0056] Among them, the waiting period This typically refers to the cumulative time a vehicle waits continuously at low speeds (or near a standstill). A speed threshold can be selected. ,when At that time, we considered the vehicle to be in a waiting state. Therefore, the waiting time... It can be defined as:

[0057] in This indicates an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

[0058] The difference between actual speed and lane speed limit Specifically: Assuming in each The lane speed limit at that time is Then in The velocity difference at time points can be written as:

[0059] Calculate the average value for the entire trajectory:

[0060] Percentage of time vehicles travel in lanes It mainly utilizes lane geometry information to calculate the vehicle's position. Lateral deviation at time (Refers to the distance between the vehicle and the lane centerline). Set an allowable deviation threshold. ,when At that time, the vehicle is considered to be "driving in its lane". Therefore, this indicator can be expressed as:

[0061] This indicates the percentage of time a vehicle spends within its lane throughout the entire trajectory.

[0062] Trajectory outliers The design intent is to measure the deviation between the vehicle's trajectory and the overall average trajectory. Let the vehicle trajectory be denoted as... (two-dimensional trajectory) Clustering methods were used to obtain several cluster centers for the motion trajectories of all agents in the Level 1 autonomous driving model library, denoted as set. Trajectory outliers Defined as the average distance between the vehicle's trajectory and all cluster centers:

[0063] in This indicates that Euclidean distance is used to measure the entire trajectory. With cluster center The degree of similarity between them. This definition can reflect the deviation between the average trajectories of the overall vehicle trajectory and the upper-level model library.

[0064] Based on these features, a set of individual features is constructed. The value of this feature changes over time throughout the entire observation trajectory. Changes. To ensure the capture of the highest-risk performance throughout the entire trajectory, for each individual characteristic... ,calculate This ensures that high risks that occur in brief moments are captured.

[0065]

[0066]

[0067]

[0068] This represents individual characteristics based on the vehicle's motion state. This indicates individual characteristics based on autonomous driving behavior. It represents individual characteristics based on uncertainty.

[0069] After obtaining the minimum value of each feature during the prediction duration, the next step is to linearly combine these minimum values. A pre-defined weight vector is introduced. Each element corresponds to an importance weight for a feature. The mathematical expression is:

[0070] Here, "." represents the dot product of vectors. That is, the minimum value of each feature at all time steps is multiplied by its corresponding weight, and then the results are summed to obtain the individual feature score of the vehicle.

[0071] In the social evaluation model, social features are first defined. These features are mainly used to describe the interactions between different driving agents. This invention adopts some widely studied and accepted safety agent indicators, including time head-down distance (THW), time to collision (TTC), deceleration required to avoid a collision (DRAC), and the time difference of conflict between the two vehicles. Indicators such as ).

[0072] Time Headway (THW) represents the time required for a following vehicle to reach the current position of the vehicle in front, converted from distance to time. It represents the time distance between the two vehicles, defined as the distance between them divided by the current speed of the following vehicle. Intuitively, THW reveals the "reaction" time the following vehicle has before colliding with the vehicle in front.

[0073] Time-to-Collision (TTC) is the time expected to pass before two road users collide, based on the current state of the road. TTC predicts the remaining time before a collision occurs between vehicles if their current speeds and directions of travel remain constant. The calculation typically relies on the relative speeds and distances between the vehicles, i.e., the expected time for the two vehicles to meet or collide in the current state.

[0074] The Deceleration Rate to Avoid Crash (DRAC) is essentially the minimum average delay required to avoid an accident at a given speed and distance between vehicles. DRAC quantifies the deceleration required to avoid a collision with a vehicle ahead or to the side in the current driving state. It reflects how quickly a vehicle must decelerate significantly to avoid a collision if it does not brake or adjust its trajectory in time.

[0075] The time difference to conflict point between the two vehicles. The time difference between two road users passing through a conflict point is a measure of their interaction. It represents the minimum time difference between the arrival times of two vehicles on their respective trajectories at the conflict point (a location where a collision may occur or a critical road intersection). This metric measures the degree to which two vehicles arrive at a potential conflict area synchronously: if the arrival times of the two vehicles at the conflict point are very close, it indicates a higher risk of collision during their interaction.

[0076] These indicators constitute a set of social characteristics. Each feature All with time In each pair of interacting vehicle trajectories All of the above can be calculated, denoted as To capture the highest potential risks throughout the interaction process, we employ a minimization strategy for each social feature during the computation, selecting the minimum value of the indicator over the entire time series. For each social feature... ,calculate The purpose of this is to ensure that even a risk that occurs for a brief moment (e.g., a momentary low TTC or a very high DRAC) can be detected and given sufficient weight in subsequent scoring.

[0077]

[0078]

[0079]

[0080]

[0081] In the above formula Indicates the characteristics of collision time. Indicates the time distance characteristic of the vehicle's front end. This indicates the deceleration required to avoid a collision. Indicates the time difference characteristics of the conflict. and These represent the coordinates of the autonomous vehicle and the coordinates of the interacting vehicle, respectively. and These represent the speeds of the vehicle and the interacting vehicle, respectively. This indicates the distance of the vehicle from the potential conflict point. This represents the average speed at which the vehicle reaches the potential point of conflict. and This represents the distance of the interacting vehicle from the potential conflict point and its average speed at which it reaches that point.

[0082] After obtaining the minimum value of each social feature during a pair of vehicle interactions, a predefined weight vector is used. These indicators are then combined using a weighted linear combination. The formula is:

[0083] Here, "." represents the dot product operation of vectors. This process integrates various social features according to their importance in the safety risk assessment, thereby obtaining the overall social risk score between a pair of vehicles.

[0084] autonomous vehicles in the scene This vehicle is different from all other vehicles. Each social interaction between the vehicles generates a social score. Ultimately, the vehicle... The overall risk score at the social level is obtained by summing it with all the social scores of other vehicles. .

[0085] The final score expression for each autonomous driving model in the Level 1 autonomous driving model library is as follows:

[0086] Here, The first part is the individual characteristic score of the autonomous vehicle, while the second part is the sum of the social characteristic scores of all vehicles interacting with it. Not only is the risk of a single vehicle considered, but the potential safety hazards arising from its interactions with all other vehicles are also fully incorporated into the evaluation system. We selected the three highest-scoring Level 1 autonomous driving models as the training data for the Level 2 autonomous driving model at that time point, guided by experts.

[0087] The training of the Level 2 autonomous driving model still employs the conventional SAC reinforcement learning algorithm, but the most crucial difference lies in the design of the reward function, which incorporates information from the behavioral demonstrations of experts (i.e., the three selected Level 1 models). By combining expert demonstrations with raw environmental feedback, the Level 2 autonomous driving model can learn the optimal policy more efficiently.

[0088] First, let the initial environmental reward be... This represents the basic reward the agent receives after interacting with the environment at the current moment, used to ensure that the training objective aligns with task requirements. Then, this invention introduces the outputs of two discriminators on this basis: an individualized behavior discriminator. and individualized transfer discriminant The reward function of the resulting Level 2 autonomous driving model can be expressed as:

[0089] The meanings of each symbol are as follows: This represents the initial reward from the environment, that is, the immediate reward the agent receives from the environment at the current moment. It is the basic feedback signal for the task, reflecting the agent's performance within the environment.

[0090] This is a weighted hyperparameter used to balance the influence between environmental rewards and reward adjustments from expert demonstration information (obtained through a discriminator), adjusting the contribution of demonstration information to the rewards and ensuring a good compromise between demonstration guidance and environmental feedback.

[0091] This represents the output function of the individualized behavior discriminator, with parameters consisting of... This indicates that the input is and , Indicates vehicle The state space, Indicates vehicle In state The action taken. The output is a value in The probability within the interval represents the degree of match between the current state-action pair and the expert-individualized demonstration data. A larger value indicates a better match, while a smaller value indicates a deviation from the demonstration.

[0092] The output function of the individualized transfer discriminator is represented by the parameters of the individualized transfer discriminator. This indicates that the input is... , as well as , Indicates action The next state space the vehicle reaches after execution. It indicates whether the state transition caused by the current state-action pair matches the local transition pattern observed in the expert demonstration. A higher output indicates a better match between the state transition and the demonstration, thus increasing the weight of the demonstration information in the reward; conversely, a lower output reduces the positive contribution of the reward.

[0093] This represents the logarithmic penalty term applied to the output of the behavior discriminator. When High (i.e., the current behavior is close to the expert demonstration). The lower the value, the more negative its logarithm; therefore, multiplying by a negative sign results in a smaller reduction in the reward. However, if... Lower (i.e., behavior deviates from expert demonstration). The larger the penalty, the greater the deduction, thus punishing behaviors that deviate from the model.

[0094] Under this design This ensures the model can ultimately complete basic driving tasks, while the reward adjustments from the discriminator output further utilize expert demonstration information: the individualized behavior discriminator determines whether the current state-action combination "imitates" the expert; the individualized transition discriminator measures whether the state transition resulting from the action also closely resembles the local pattern demonstrated by the expert. The agent receives less penalty when it maintains relative consistency with the expert demonstration, and suffers greater deduction in rewards if it deviates from the demonstration, thus guiding the Level 2 autonomous driving model's strategy towards the expert demonstration.

[0095] As can be seen, this invention first selects the three models with the highest comprehensive scores from the Level 1 autonomous driving model library through a parallel spatiotemporal prediction module, serving as "experts" in the training process of the Level 2 autonomous driving model. At each training time, the actions given by these three expert models are evaluated using the aforementioned reward function; the state-action pair with the highest reward is then stored in the SAC experience replay pool. The Level 2 autonomous driving model balances exploration of the real environment with learning from expert demonstrations, enabling it to converge to the optimal solution with a high-level driving strategy more quickly and stably.

[0096] Regarding the specific training process, this invention follows the conventional approach of SAC reinforcement learning, and both Level 1 and Level 2 autonomous driving models can be trained using the SAC algorithm. SAC is an advanced reinforcement learning algorithm that not only stably learns policies but also effectively balances the exploration and utilization processes of the agent by controlling action entropy during the exploration process. In this invention, the SAC algorithm used to train the Level 1 and Level 2 autonomous driving models is basically the same except for the reward function. The SAC algorithm used in this invention has three modules: a state input module, a decision module, and an action output module. The state input module is mainly responsible for receiving driving environment information and vehicle state information, including vehicle coordinates, speed, acceleration, road environment information, and other vehicle states (position, speed, acceleration, etc.). The intelligent decision module mainly uses the SAC algorithm as its core, combined with two networks, Actor and Critic, to update the policy through continuous interaction with the environment. The action output module mainly generates control quantities such as steering wheel angle, accelerator, and brake pedal based on the information processed by the decision module. These output actions directly affect the lateral and longitudinal control of the vehicle.

[0097] The Actor network comprises several fully connected layers that calculate the mean and standard deviation for steering, acceleration, and braking, respectively. Two activation functions are used: the tanh function is used to normalize the steering output to [-1, 1], while the sigmoid function is used to normalize the acceleration and braking outputs to the [0, 1] range, with the standard deviation also normalized using the sigmoid function. The Critic network is designed to evaluate the value of action choices in the current state, and its input also comes from the vehicle's multi-dimensional state information. The network estimates state values ​​through a multi-layer fully connected structure (including the use of the GELU activation function). A soft update strategy is used in parameter updates, gradually aligning the target network with the current network to ensure a smooth training process.

[0098] Training process and algorithm details: Experience Acquisition and Network Update: In the simulation environment, the vehicle behaves according to the current policy and records empirical tuples of state, action, reward, and next state. Using this empirical data, the Critic network is updated by minimizing the mean squared error (MSE) of the Q-value estimation, while the Actor network's parameters are updated using the Adam gradient descent method.

[0099] Exploration and Entropy Regularization: A core feature of the SAC algorithm is that it encourages exploration by maximizing action entropy. This allows the model to explore the state space more fully in the early stages, while improving the overall policy convergence by balancing exploration and utilization.

[0100] In one embodiment of the present invention, the SAC algorithm model includes a Network and four Network, four The networks are for state value estimation. and Networks, and action-state value estimation and network. The network input is the state space defined in the patent (vehicle coordinates, speed, acceleration; road coordinates; coordinates, speed, acceleration of surrounding vehicles), and the output is the action probability distribution parameters defined in the patent (steering wheel angle and accelerator and brake pedals). The network input is the state space defined in the patent, and the output is the value of the state and action-state pairs (i.e., the expected reward value in this driving scenario). Network output is An estimate representing the value of a state. The network output is Represents the action-state pair value.

[0101] In the SAC algorithm, if The more the network output action can enable The greater the value, the better. The value of network output actions The more accurate (according to the Bellman equation), the better. Whether it is accurate depends on The more accurate the result, the better. Network output state value The more accurate, the better.

[0102] Experience pool generation process: Given a state ,pass The network obtains the parameters of the probability distribution of all actions (mean and variance of a Gaussian distribution), and then samples the actions according to the probability distribution. Then Input into the environment and get and This gives us a : Then Add to experience pool The purpose of the experience pool is to eliminate... The correlation between actions is determined because actions are usually strongly correlated in reinforcement learning. These correlations are broken down and placed into an experience pool, and then randomly selected from the experience pool during neural network training. A batch of people were selected This makes neural network training more stable and robust.

[0103] Network update process: From the experience pool Data collected from China conduct Network updates, For example, based on the optimal Bellman equation, As a state The true value estimate will be based on the actual actions taken. of Value as state Predictive value estimation, and finally use As Functions for neural networks and Conduct training. (Take...) This means from the experience pool Take one The data is averaged, that is: .

[0104] Network update process: From the experience pool Data collected from China conduct Network updates, For example, the state value is estimated using the following formula, i.e., the following formula is used as... The actual value output by the network:

[0105] :pair The network takes the minimum value. China typically maintains two sets Networks are used to mitigate the overestimation problem. Taking the smaller of the two values ​​here makes the target more conservative and stable.

[0106] : legitimate and proper item Is the current strategy in the state? Downsampling to action The logarithmic probability. Multiplied by a coefficient. Then, the target value includes an entropy term that "encourages exploration": This strategy not only aims for high... The value should also maintain a certain degree of randomness to avoid prematurely adopting a suboptimal deterministic strategy.

[0107] Expectation of policy distribution State value Defined in state Next, follow the current strategy. Sample an action, then evaluate that action. Expected value.

[0108] use The network output is used as the predicted value, and finally, it is also used... As Functions for neural networks Conduct training.

[0109] Network update process: From the experience pool Draw one from : For each , using the current Resampling action (Or directly enumerate all discrete actions).

[0110] For each calculate Take the negative sign and adjust accordingly. Average, to get the final .

[0111] Using automatic differential calculation Deep learning frameworks (such as PyTorch / TF) will automatically derive the gradient based on the above formula.

[0112] Use Adam or SGD, based on the learning rate. Do generally The update frequency may be similar to Incomplete synchronization (e.g.) Updated every step of the way. Update every few steps to ensure Learn to be relatively stable first.

[0113] This application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. In this embodiment, the computer-readable storage medium, when executed by a processor, implements the training method for an adaptive autonomous driving model for complex road scenarios.

[0114] In summary, this invention integrates multiple models with different decision-making characteristics into a Level 2 autonomous driving model, resulting in the following advantages: ① Improved adaptability to multiple scenarios: In different driving environments such as congestion, smooth traffic, highways, or urban roads, the Level 2 autonomous driving model can flexibly "borrow" the Level 1 model that best suits the current scenario, thereby achieving better performance in terms of safety or efficiency. ② Enriched decision candidate space: Multiple models have accumulated differentiated decision-making experience. Through parallel prediction and scoring, the advantages and disadvantages of multiple driving strategies can be quickly evaluated at the same time, and the optimal strategy can be selected for execution. ③ Shortened training cycle and reduced difficulty: Compared to training a single model covering all working conditions from scratch, the Level 2 model only needs to be further strengthened based on multiple Level 1 models (equivalent to "expert demonstrations"), thus achieving faster convergence and avoiding unnecessary repeated exploration.

[0115] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0116] The order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A training method for an adaptive autonomous driving model for complex road scenarios, characterized in that, Includes the following steps: Multiple pre-built Level 1 autonomous driving models are trained, and the reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input to the Level 1 autonomous driving model is road environment information, vehicle status information and other vehicle status information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. The training process involves comprehensively evaluating multiple Level 1 autonomous driving models and selecting the three models with the highest comprehensive scores as expert models. The specific process of comprehensive evaluation includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, and measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model. The two scores are then weighted and summed according to preset weights to obtain the comprehensive score. The pre-built Level 2 autonomous driving model is trained. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. Specifically, during training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action, and receives greater deduction in the reward level if it deviates from the demonstration action. After training, the strategy of the Level 2 autonomous driving model moves closer to the demonstration action of the expert model.

2. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the expert model's state-action. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the expert model's state transition based on the current and subsequent vehicle state-action.

3. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The weights of safety rewards and driving efficiency rewards in the reward function of the Level 1 autonomous driving model are specifically obtained by sampling two Gaussian distributions.

4. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The individual evaluation model specifically scores the individual characteristics of the vehicle, which include the vehicle's motion state, driving behavior, and uncertainty-based individual characteristics. The vehicle's motion state includes speed, acceleration, and rate of change of acceleration. Autonomous driving behavior includes waiting time, the difference between actual speed and lane speed limit; uncertain individual characteristics include the percentage of time spent driving along lane lines, and trajectory anomalies.

5. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 4, characterized in that, The scoring of individual characteristics is specifically obtained by multiplying the minimum value of each individual characteristic at all time steps by its corresponding weight and then summing the results.

6. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 1, characterized in that, The social evaluation model scores drivers based on interaction metrics between different drivers. These metrics include head-on distance, collision time, deceleration required to avoid a collision, and the time difference between the interaction between the two vehicles.

7. The training method for an adaptive autonomous driving model for complex road scenarios according to claim 6, characterized in that, The scoring of the social evaluation model is as follows: after obtaining the minimum value of each interaction index in a pair of vehicle interactions, the scores are obtained by weighted summation using a predefined weight vector.

8. A training system for an adaptive autonomous driving model for complex road scenarios, characterized in that, include: The Level 1 autonomous driving model training module is used to train multiple pre-built Level 1 autonomous driving models. The reward functions of different models correspond to different training strategy tendencies, specifically corresponding to different weights in the trade-off between safety rewards and driving efficiency rewards. The input to the Level 1 autonomous driving model is road environment information, vehicle status information and other vehicle status information, and the output is the control quantity of the vehicle's autonomous driving actions, including the control quantity of the vehicle's steering wheel angle, accelerator, and brake pedal actions. The parallel spatiotemporal prediction module is used to comprehensively score multiple trained Level 1 autonomous driving models and select the three models with the highest comprehensive scores as expert models. The specific process of comprehensive scoring includes: measuring and scoring the driving performance of the vehicle through a pre-built individual evaluation model, measuring and scoring the interaction safety of the vehicle with other surrounding vehicles through a pre-built social evaluation model, and then weighting and summing the two scores according to preset weights to obtain the comprehensive score. The Level 2 autonomous driving model training module is used to train a pre-built Level 2 autonomous driving model. The input and output of the Level 2 autonomous driving model are the same as those of the Level 1 autonomous driving model. During training, the data in the training set is input into three expert models, and the outputs of the three expert models are then fed into the reward function of the Level 2 autonomous driving model for calculation. The Level 2 autonomous driving model receives less penalty when it is relatively consistent with the expert demonstration action, and receives greater deduction in reward if it deviates from the demonstration action. After training, the strategy of the Level 2 autonomous driving model moves closer to the demonstration action of the expert model.

9. The training system for an adaptive autonomous driving model for complex road scenarios as described in claim 8, characterized in that, The reward function of the Level 2 autonomous driving model integrates the original environmental reward, the individualized behavior discriminator reward, and the individualized transition discriminator reward. The original environmental reward is the basic reward obtained after interacting with the environment at the current moment, used to ensure that the training objective is consistent with the task requirements. The individualized behavior discriminator reward is the output function value of the individualized behavior discriminator, which calculates the degree of matching between the current vehicle state-action and the expert model's state-action. The individualized transition discriminator reward is the output function value of the individualized transition discriminator, which calculates the degree of matching between the state transition and the expert model's state transition based on the current and subsequent vehicle state-action.

10. A computer storage medium, characterized in that, It contains a computer program that can be executed by a processor, which performs the training method for an adaptive autonomous driving model for complex road scenarios as described in any one of claims 1-7.

Images