Automatic driving model training method and device combining GAIL and PPO

Abstract

The embodiment of the invention provides an automatic driving model training method and device combining GAIL and PPO, and the method comprises the steps: collecting an expert driving track sequence through a simulation environment by designing a multi-stage training frame; a double-branch discriminator network is constructed, a frame-level branch analyzes single-frame behavior characteristics based on an attention mechanism, and a track-level branch adopts a recurrent neural network to process a continuous driving sequence so as to realize multi-scale characteristic fusion. Iterative training is carried out by adopting a near-end strategy optimization algorithm, a multi-task learning objective function is constructed, driving action generation and trajectory prediction are optimized at the same time, and an adaptive confidence interval constraint and experience playback mechanism is introduced. According to the method, the defects of the traditional technology in the aspects of driving behavior evaluation, strategy optimization and the like are effectively overcome, and the robustness and generalization ability of an automatic driving model are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and particularly to a method and device for training an autonomous driving model that combines GAIL and PPO. Background Art

[0002] Existing methods for training autonomous driving models have obvious deficiencies. Traditional methods usually use the Generative Adversarial Imitation Learning (GAIL) or Proximal Policy Optimization (PPO) algorithm alone, lacking multi-scale analysis and continuous evaluation of driving behaviors, and it is difficult to ensure the robustness and generalization ability of the model.

[0003] In addition, there are bottlenecks in the discriminator design of the existing technology. Most systems only focus on the similarity evaluation of single-frame driving behaviors, ignoring the continuous characteristics of driving trajectories, which affects the adaptability of the model to complex scenarios.

[0004] There are technical shortcomings in the policy optimization of existing systems. Lack of an effective multi-task learning mechanism and adaptive confidence interval constraints, making it difficult to balance the exploration and exploitation of the model. Solving these problems is of great significance for improving the training effect of autonomous driving models. Summary of the Invention

[0005] Aiming at the problems in the existing technology, this application provides a method and device for training an autonomous driving model that combines GAIL and PPO, which can effectively solve the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improve the robustness and generalization ability of the autonomous driving model.

[0006] To solve at least one of the above problems, this application provides the following technical solutions:

[0007] In the first aspect, this application provides a method for training an autonomous driving model that combines GAIL and PPO, including:

[0008] Collect training data and preprocess it. Build multiple test sections in the autonomous driving simulation environment, collect the continuous driving trajectory sequences of expert drivers in the test sections, record the environmental image information, navigation instruction information, vehicle speed information and corresponding control actions during the expert driving process, and construct the environmental image information, navigation instruction information, vehicle speed information and control actions into a training sample sequence according to the temporal relationship;

[0009] Construct a multi-stage training framework. First, use generative adversarial imitation learning to train the discriminator network and the policy generation network. The discriminator network includes a frame-level discrimination branch and a trajectory-level discrimination branch. The frame-level discrimination branch extracts single-frame driving behavior features based on the attention mechanism. The trajectory-level discrimination branch uses a recurrent neural network to process continuous driving behavior sequences, fuses the features output by the frame-level discrimination branch and the trajectory-level discrimination branch, and then outputs a similarity score. Train the policy generation network to minimize the similarity score of the discriminator network;

[0010] Use the proximal policy optimization algorithm to iteratively optimize the policy generation network, construct a multi-task learning objective function to optimize the driving action generation task and the trajectory prediction task simultaneously, set an adaptive confidence interval threshold based on historical training data, optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, store the state transition sequence during the training process in the experience replay pool, sample historical data from the experience replay pool to participate in policy update, and output the trained autonomous driving model through cyclic iteration.

[0011] Furthermore, it also includes: constructing a test road section map in the autonomous driving simulation environment, dividing the test road section map into a training area and a test area, setting multiple sampling starting points and ending points in the training area, generating a reference trajectory from the starting point to the ending point based on the path planning algorithm, and inputting the reference trajectory into the simulation environment control module;

[0012] Collect the driving data of expert drivers in the training area, deploy on-vehicle forward cameras and wide-angle cameras to collect environmental image information, collect the navigation instruction information output by the navigation system and the vehicle speed information output by the vehicle speed sensor, record the throttle control amount, steering control amount and braking control amount during the expert driving process, and construct a training sample set with the environmental image information, navigation instruction information, vehicle speed information and control amount according to the sampling timestamp.

[0013] Furthermore, it also includes: preprocessing the environmental image information, normalizing the RGB image to a preset pixel range, performing data augmentation using the random cropping and rotation method, performing one-hot encoding processing on the navigation instruction information, performing maximum-minimum normalization processing on the vehicle speed information, and linearly mapping the control actions to a standard interval to generate standardized training features;

[0014] Construct a training sequence based on a sliding time window, set a time window with a fixed length, sort the standardized training features within the same time window according to the timestamp to construct a subsequence, perform temporal alignment on the subsequence, construct the aligned subsequence into a training sample sequence, and calculate the state transition relationship between adjacent frames in the training sample sequence.

[0015] Furthermore, it further includes: constructing a dual-branch structure of the discriminator network, arranging a spatial attention module and a channel attention module in the frame-level discrimination branch, where the spatial attention module calculates the weight coefficients at different spatial positions on the image feature map, the channel attention module calculates the weight coefficients of different feature channels, and the weight coefficients are weighted and combined with the original features to obtain enhanced features. In the trajectory-level discrimination branch, a long short-term memory network is arranged to extract temporal features, and the frame-level features and the trajectory-level features are input into a multi-layer perceptron for fusion;

[0016] Construct a policy generation network structure. Use a residual convolutional neural network as the environmental image feature extractor. Use a fully connected layer to process the navigation instruction information and vehicle speed information to obtain a state feature vector. Input the state feature vector into the policy network to generate the probability distribution of vehicle control actions. Use the reparameterization trick to sample from the probability distribution to obtain a deterministic action. Based on the similarity score output by the discriminator network, construct a generative adversarial loss function to optimize the parameters of the policy generation network.

[0017] Furthermore, it further includes: inputting the frame-level features and the trajectory-level features into a feature fusion module, using a weighted summation method with adaptive weights to fuse the two features. The adaptive weights are calculated based on the variance and mean of the feature vectors. Input the fused features into the output layer of the discriminator to obtain a similarity score, and perform sigmoid activation processing on the similarity score to obtain a discriminant probability value between 0 and 1;

[0018] Adopt an alternating training strategy to optimize the discriminator network and the policy generation network. Fix the parameters of the policy generation network and train the discriminator network to maximize the discrimination accuracy of expert data and generated data. Fix the parameters of the discriminator network and train the policy generation network to minimize the discrimination probability value of the generated data. During the training process, use a gradient penalty term to constrain the gradient norm of the discriminator, and dynamically adjust the training ratio of the two networks based on the discrimination accuracy on the validation set.

[0019] Furthermore, it further includes: constructing a multi-task learning objective function, weighting and combining the policy gradient loss of the driving action generation task and the mean squared error loss of the trajectory prediction task, calculating the KL divergence between the action distributions of the new policy and the baseline policy, calculating an adaptive confidence interval threshold based on the mean and standard deviation of the KL divergence in the recent N rounds of iterations, and using the confidence interval threshold as a constraint condition for policy update;

[0020] The importance sampling method is used to estimate the expected return of the new policy. The advantage function value of each time step is calculated based on Generalized Advantage Estimation (GAE), and the obtained advantage function value is normalized. The normalized advantage function value is multiplied by the log-likelihood of the action probability to obtain the policy gradient, and parameter updates are performed in the direction of the policy gradient. When the KL divergence between the new policy and the baseline policy exceeds the confidence interval threshold, this round of update is terminated in advance.

[0021] Further, it also includes: constructing an experience replay pool structure, constructing state transition tuples from the states, actions, rewards, and next states collected in each training episode, calculating the temporal difference error of each state transition tuple, setting priority weights for the state transition tuples based on the temporal difference error, storing the state transition tuples and their priority weights in the experience replay pool, and removing the historical data with the lowest priority when the capacity of the experience replay pool reaches the upper limit.

[0022] Importance sampling is performed according to the priority weights from the experience replay pool, the sampled historical data is mixed with the new data generated by the current policy to construct a training batch, the policy loss and value loss are calculated based on the mixed training data, the parameters of the policy generation network are updated by gradient, and it is judged whether the performance of the policy network converges on the validation set. When there is no significant improvement in the validation performance for multiple consecutive rounds, the training is stopped and the model parameters are output.

[0023] In a second aspect, the present application provides an autonomous driving model training device combining GAIL and PPO, including:

[0024] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the autonomous driving model training method combining GAIL and PPO are implemented.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the autonomous driving model training method combining GAIL and PPO are implemented.

[0026] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the autonomous driving model training method combining GAIL and PPO are implemented.

[0027] As can be seen from the above technical solutions, the present application provides a method and device for training an autonomous driving model combining GAIL and PPO. By designing a multi-stage training framework, expert driving trajectory sequences are collected through a simulation environment. A dual-branch discriminator network is constructed. The frame-level branch analyzes single-frame behavior features based on the attention mechanism, and the trajectory-level branch uses a recurrent neural network to process continuous driving sequences to achieve multi-scale feature fusion. The proximal policy optimization algorithm is used for iterative training, a multi-task learning objective function is constructed to optimize both driving action generation and trajectory prediction, and an adaptive confidence interval constraint and an experience replay mechanism are introduced. This method effectively solves the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improves the robustness and generalization ability of the autonomous driving model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0029] Figure 1 It is a schematic flowchart of the method for training an autonomous driving model combining GAIL and PPO in the embodiments of the present application;

[0030] Figure 2 It is a structural diagram of the device for training an autonomous driving model combining GAIL and PPO in the embodiments of the present application;

[0031] Figure 3 It is a schematic structural diagram of the electronic device in the embodiments of the present application.

[0032] Reference Numerals:

[0033] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0035] In the technical solution of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant provisions of national laws and regulations.

[0036] Considering the problems existing in the prior art, this application provides a method and device for training an autonomous driving model combining GAIL and PPO. By designing a multi-stage training framework, expert driving trajectory sequences are collected through a simulation environment. A dual-branch discriminator network is constructed. The frame-level branch analyzes single-frame behavior features based on the attention mechanism, and the trajectory-level branch uses a recurrent neural network to process continuous driving sequences to achieve multi-scale feature fusion. The proximal policy optimization algorithm is used for iterative training, a multi-task learning objective function is constructed to optimize both driving action generation and trajectory prediction, and an adaptive confidence interval constraint and an experience replay mechanism are introduced. This method effectively solves the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improves the robustness and generalization ability of the autonomous driving model.

[0037] In order to effectively solve the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improve the robustness and generalization ability of the autonomous driving model, this application provides an embodiment of a method for training an autonomous driving model combining GAIL and PPO. Refer to Figure 1 The method for training an autonomous driving model combining GAIL and PPO specifically includes the following content:

[0038] Step S101: Collect and preprocess training data. Construct multiple test sections in the autonomous driving simulation environment, collect the continuous driving trajectory sequences of expert drivers in the test sections, record the environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process, and construct the environmental image information, navigation instruction information, vehicle speed information, and control actions into a training sample sequence according to the time sequence relationship;

[0039] In this embodiment, a complete simulation environment construction scheme is first designed. Based on the CARLA simulator, an autonomous driving test environment highly restoring the real scene is built, including various typical road sections such as urban roads, highways, and rural roads. The design of the test road sections fully considers factors such as road curvature changes, slope changes, and road surface materials, and simulates different weather conditions and lighting conditions. To enhance the authenticity of the scene, dynamic traffic participants are added to the road sections, including other vehicles, pedestrians, and non-motor vehicles, and their behavior patterns are obtained based on real traffic data statistics.

[0040] In this embodiment, a professional driving data acquisition scheme is implemented. Professional drivers with more than ten years of driving experience are selected for demonstration driving, and the acquisition routes cover test road sections of different difficulty levels. During driving, environmental image information is collected through three on-vehicle cameras: the forward main camera is responsible for collecting the road scene within a 120-degree field of view in front of the vehicle, and the wide-angle cameras on the left and right sides are responsible for collecting the lateral traffic conditions. The installation positions and angles of the cameras are carefully designed to ensure the complete coverage of the field of view and the stability of the image quality.

[0041] In this embodiment, an efficient navigation information processing mechanism is developed. The navigation system generates detailed navigation instructions based on a high-precision map, including information such as the type of the next key action point (going straight, turning left, turning right, etc.), the distance to reach this action point, and the recommended lane. To improve the practicality of the navigation instructions, the continuous navigation information is discretized into several key decision points, and each decision point corresponds to a clear navigation action. This design enables the model to better understand and execute the navigation intention.

[0042] In this embodiment, an accurate vehicle speed information acquisition system is constructed. A high-precision Hall sensor is used to collect the wheel rotation speed information in real time, and the vehicle speed is obtained through wheel circumference conversion. At the same time, a Kalman filter is introduced to smooth the original vehicle speed data and eliminate the influence of sensor noise. The sampling frequency of the vehicle speed information is set to 100Hz to ensure the accurate capture of the vehicle's dynamic characteristics. The vehicle speed information is synchronized with other sensor data through timestamps to ensure the temporal consistency of the data.

[0043] In this embodiment, a complete control action recording scheme is designed. The control data of expert drivers is collected through the CAN bus, including the throttle pedal opening, the braking pedal force, and the steering wheel angle. The sampling frequency of the control signal is the same as that of the vehicle speed sensor, which is also 100Hz. To ensure the quality of the collected data, the effectiveness of each control quantity is monitored in real time during the data recording process, and outliers and noise interference are filtered out. At the same time, the duration and change trend of the control actions are recorded to provide a basis for subsequent behavior analysis.

[0044] This embodiment realizes an innovative training sample construction method. Based on the sliding time window mechanism, training data is organized with the window size set to 2 seconds and the sliding step to 0.1 second. Within each time window, the environmental image, navigation instruction, vehicle speed information, and control action are aligned according to timestamps to construct a complete state-action pair. To enhance the expressive ability of the data, the change trends of the state and action within the window are calculated to capture the dynamic characteristics of driving behavior.

[0045] This embodiment establishes a reliable data preprocessing process. The environmental image undergoes de-distortion, illumination normalization, and size adjustment, and is uniformly processed into an RGB image with a resolution of 480×640. The navigation instruction is converted into a one-hot encoded vector through a look-up table method for easy processing by the neural network. The vehicle speed information is mapped to the interval [0,1] through min-max normalization. The control action is also normalized: the throttle and brake control amounts are mapped to the interval [0,1], and the steering control amount is mapped to the interval [-1,1].

[0046] Through the above technological innovations, this embodiment effectively solves multiple key problems in the training of autonomous driving models, such as unstable training data quality, incomplete scenario coverage, inaccurate action annotation, etc. In practical applications, this solution can collect high-quality expert driving data, providing a reliable data foundation for subsequent model training. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through professional data collection and processing processes, the quality and practicality of the training data are significantly improved. The systematicness and integrity of this solution enable it to support the model training requirements in different scenarios, and through a standardized data processing process, efficient organization and management of training data are achieved.

[0047] Step S102: Construct a multi-stage training framework. First, use generative adversarial imitation learning to train the discriminator network and the policy generation network. The discriminator network includes a frame-level discriminant branch and a trajectory-level discriminant branch. The frame-level discriminant branch extracts single-frame driving behavior features based on the attention mechanism. The trajectory-level discriminant branch uses a recurrent neural network to process continuous driving behavior sequences. The features output by the frame-level discriminant branch and the trajectory-level discriminant branch are fused and then a similarity score is output. Train the policy generation network to minimize the similarity score of the discriminator network;

[0048] Optionally, this embodiment first designs a dual-branch discriminator network architecture. The frame-level discriminant branch uses ResNet-50 as the backbone network, and a spatial attention module and a channel attention module are added after each residual block. The spatial attention module calculates the importance weights of each position on the feature map, enabling the network to focus on key areas such as the vehicle ahead and traffic signs. The channel attention module learns the importance of different feature channels, enhancing the feature expression related to driving decisions. This design enables the network to accurately capture the key driving information in a single-frame image.

[0049] This embodiment realizes an efficient trajectory-level feature extraction mechanism. The trajectory-level discriminant branch adopts a two-layer LSTM network structure with a hidden layer dimension of 256, which is used to process continuous driving behavior sequences. Through the gating mechanism and memory unit, the LSTM network can effectively model the temporal dependence relationship of driving behaviors. The input of the network includes the state features and control actions of consecutive frames, and the output is the encoded trajectory features. To enhance the temporal modeling ability, residual connections are added between the LSTM layers to alleviate the vanishing gradient problem in long sequence training.

[0050] This embodiment constructs an innovative feature fusion strategy. The adaptive attention mechanism is used to fuse the frame-level features and trajectory-level features. The attention weights are generated by a learnable mapping function, which takes into account the complementarity and redundancy of the two types of features. During the fusion process, the network dynamically adjusts the importance weights of the features of the two branches, enabling flexible utilization of single-frame information and historical information in different driving scenarios. At the same time, residual connections are introduced to ensure the complete transmission of information.

[0051] This embodiment designs a stable policy generation network. The network adopts an encoder-decoder structure. The encoder uses a pre-trained visual backbone network to extract environmental features, and the decoder generates control actions through a multi-layer fully connected network. To improve the diversity and stability of the generated actions, a mixture of Gaussian distributions is used to model at the action output layer. The network simultaneously predicts the mean and variance of the actions, and realizes action sampling through the reparameterization trick to ensure the differentiability of the gradient.

[0052] This embodiment realizes a reliable adversarial training mechanism. The training objective of the discriminator is to maximize the discrimination between expert data and generated data, while the policy network attempts to generate driving actions that can deceive the discriminator. To prevent the discriminator from being over-trained, a gradient penalty term is introduced to limit the discrimination ability of the discriminator. At the same time, an experience pool is used to store the generated trajectory samples, and importance sampling is used to improve the training efficiency. The training ratio of the discriminator and the policy network is dynamically adjusted to maintain the adversarial balance.

[0053] This embodiment establishes a complete training process management. A phased training strategy is adopted. First, the discriminator network is pre-trained to establish the basic discrimination ability, and then it enters the adversarial training stage. During the adversarial training, the parameters of the discriminator and the policy network are alternately updated. After each round of training, the performance of the policy network is evaluated, including indicators such as trajectory similarity and control smoothness. When the performance of the policy network reaches the preset threshold, it enters the policy optimization stage.

[0054] In this embodiment, a comprehensive evaluation index system is constructed. The similarity score output by the discriminator not only considers the rationality of single-frame actions, but also evaluates the continuity and smoothness of the entire driving trajectory. The calculation of the score adopts a weighted method, combining the frame-level discriminant score and the trajectory-level discriminant score. The weight coefficients are dynamically adjusted according to the performance on the validation set to ensure the accuracy and stability of the score. At the same time, multiple auxiliary indicators are introduced, such as trajectory deviation, speed distribution, etc., to comprehensively evaluate the quality of the generation strategy.

[0055] Through the above technical innovations, this embodiment effectively solves multiple key problems in the training of autonomous driving models: unstable single-frame decision-making, insufficient temporal modeling, unbalanced adversarial training, etc. In practical applications, this solution can learn the driving styles and decision-making patterns of expert drivers and generate safe and smooth driving trajectories. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through multi-level feature extraction and adversarial learning, it significantly improves the driving ability and generalization performance of the model. The innovation and systematicness of this solution enable it to handle various driving scenarios, and through continuous adversarial training and optimization, it realizes the gradual improvement of driving strategies.

[0056] Step S103: Use the proximal policy optimization algorithm to iteratively optimize the policy generation network, construct a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task simultaneously, set an adaptive confidence interval threshold based on historical training data, optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, store the state transition sequence during the training process in the experience replay pool, sample historical data from the experience replay pool to participate in policy update, and output the trained autonomous driving model through loop iteration.

[0057] Optionally, this embodiment first designs an innovative multi-task learning framework. Construct a joint optimization objective function, including the policy gradient loss of the driving action generation task and the mean squared error loss of the trajectory prediction task. The driving action generation task is responsible for outputting real-time control instructions, including throttle, brake, and steering control. The trajectory prediction task predicts the movement trajectory of the vehicle within the next 5 seconds, including position, speed, and heading angle. The two tasks achieve efficient utilization of perception information by sharing the feature representation of the encoder network.

[0058] This embodiment realizes a stable policy optimization mechanism. The proximal policy optimization (PPO) algorithm is used for policy network training, and the expected return of the new policy is estimated by the importance sampling method. At each training step, first collect the trajectory data of the interaction between the current policy and the environment, and calculate the advantage function value at each time step. The advantage function reflects the quality of the current action relative to the average performance and provides a reliable guiding signal for policy update. At the same time, an entropy regularization term is introduced to prevent the policy from converging to a local optimal solution prematurely.

[0059] This embodiment constructs an adaptive confidence interval constraint. Based on the KL divergence statistical characteristics of policy updates in the most recent N rounds of iteration, the confidence interval threshold for policy updates is dynamically set. When the KL divergence between two policies exceeds the threshold, this round of update is terminated in advance to ensure the continuity and stability of policy changes. This design is particularly suitable for the autonomous driving scenario because excessive policy changes may lead to unsafe driving behaviors.

[0060] This embodiment designs an efficient experience replay mechanism. A prioritized experience replay pool is constructed, and the state transition sequences during the training process are sorted and stored according to the temporal difference error magnitude. The temporal difference error reflects the learning value of the current experience, and the larger the error, the more important the information contained in this experience. When performing policy updates, experience data with larger temporal difference errors is preferentially sampled to improve the training efficiency. When the capacity of the experience pool reaches the upper limit, the historical data with the lowest priority is removed.

[0061] This embodiment implements an innovative policy network structure. A dual-head network design is adopted. The action head is responsible for generating the probability distribution of control instructions, and the trajectory head is responsible for predicting future trajectories. The two tasks share the feature extraction network but have independent output layers. The feature extraction network adopts a residual structure, effectively alleviating the problem of gradient disappearance in the training of deep networks. At the same time, an attention module is added to the network to enhance the perception ability of key environmental features.

[0062] This embodiment establishes a complete training process control mechanism. An early stopping strategy is adopted to monitor the model training process, and the model performance is regularly evaluated on the validation set. When the validation performance does not improve significantly for multiple consecutive rounds of iteration, the training is automatically stopped and the optimal model parameters are saved. To prevent overfitting, a dropout layer and L2 regularization are introduced to improve the generalization ability of the model. At the same time, a model checkpoint mechanism is implemented to regularly save the training state and support resuming training from a breakpoint.

[0063] This embodiment constructs a reliable performance evaluation system. Evaluation scenarios with multiple difficulty levels are set in the test environment to comprehensively test the driving ability of the model. The evaluation metrics include trajectory tracking accuracy, speed control smoothness, steering operation continuity, etc. By comparing with expert driving data, the performance of the model in different scenarios is verified. The evaluation results serve as an important basis for model iteration and optimization.

[0064] This embodiment designs an innovative parallel training architecture. An asynchronous parallel method is used to simultaneously run multiple training environments, and each environment independently collects interaction data. The central policy network regularly collects experience data from each environment for batch updates. This design significantly improves the data collection efficiency and training speed. At the same time, a progressive curriculum learning strategy is implemented, gradually transitioning from simple scenarios to complex scenarios to help the model establish basic driving capabilities.

[0065] Through the above technological innovations in this embodiment, multiple key problems in the training of autonomous driving models are effectively solved: unstable policy updates, low training efficiency, poor generalization ability, etc. In practical applications, this solution can stably and efficiently train an intelligent model with good driving capabilities. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through mechanisms such as multi-task learning and experience replay, the learning efficiency and generalization ability of the model are significantly improved. The adaptive characteristics of this solution enable it to handle various driving scenarios, and through continuous optimization and iteration, the stable evolution of the intelligent driving model is achieved.

[0066] As can be seen from the above description, the method for training an autonomous driving model combining GAIL and PPO provided by the embodiments of this application can, by designing a multi-stage training framework, collect expert driving trajectory sequences through a simulation environment. Construct a dual-branch discriminator network. The frame-level branch analyzes single-frame behavior features based on the attention mechanism, and the trajectory-level branch uses a recurrent neural network to process continuous driving sequences to achieve multi-scale feature fusion. Use the proximal policy optimization algorithm for iterative training, construct a multi-task learning objective function to optimize both driving action generation and trajectory prediction simultaneously, and introduce an adaptive confidence interval constraint and an experience replay mechanism. This method effectively solves the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improves the robustness and generalization ability of the autonomous driving model.

[0067] In an embodiment of the method for training an autonomous driving model combining GAIL and PPO of this application, the following specific content may also be included:

[0068] Step S201: Construct a test section map in the autonomous driving simulation environment, divide the test section map into a training area and a test area, set multiple sampling start points and end points in the training area, generate a reference trajectory from the start point to the end point based on a path planning algorithm, and input the reference trajectory into the simulation environment control module;

[0069] Step S202: Collect the driving data of an expert driver in the training area, deploy an in-vehicle front camera and a wide-angle camera to collect environmental image information, collect the navigation instruction information output by the navigation system and the vehicle speed information output by the vehicle speed sensor, record the throttle control amount, steering control amount, and braking control amount during the expert's driving process, and construct a training sample set from the environmental image information, navigation instruction information, vehicle speed information, and control amount according to the sampling timestamp.

[0070] Optionally, in this embodiment, a complete test section construction plan is first designed. Diversified road environments are constructed based on open-source map data, including typical scenarios such as urban arterial roads, secondary roads, highways, and rural roads. The section design takes into account key parameters such as road width, number of lanes, road surface curvature, and slope changes to ensure the rationality and authenticity of the road network structure. At the same time, traffic signs, signal lights, speed bumps and other infrastructure are added to the section, and the traffic flow density at different times is simulated to create a real driving environment.

[0071] This embodiment implements a scientific regional division strategy. The road network is divided into a training area and a test area using a hierarchical clustering algorithm. The training area covers common driving scenarios, while the test area contains more challenging complex road conditions. The two areas maintain a certain similarity in road type and difficulty distribution to ensure the effectiveness and generalization of model training. To increase scenario diversity, multiple functional blocks are set in the training area, such as commercial areas, residential areas, and school areas, to simulate the traffic characteristics of different regions.

[0072] This embodiment constructs an intelligent sampling point generation mechanism. A heuristic algorithm is used to set sampling start and end point pairs in the training area to ensure coverage of driving tasks with different distances and complexities. The selection of sampling points takes into account the topological structure and traffic flow characteristics of the road network, and key nodes with representativeness are preferentially selected. At the same time, a dynamic sampling mechanism is implemented, and according to the performance during model training, the distribution of sampling points is adaptively adjusted, with a focus on scenarios where the model performs poorly.

[0073] This embodiment designs an efficient path planning algorithm. An improved A* algorithm is used to generate a global reference trajectory, and influencing factors such as road grade and number of turns are added to the traditional heuristic function. The planning result not only considers the path length, but also weighs the road traffic conditions and driving difficulty. To improve the practicality of the planned trajectory, curve smoothing processing is introduced to generate a continuous trajectory that conforms to the vehicle dynamics characteristics. At the same time, a multi-level trajectory optimization mechanism is established to improve driving comfort while ensuring safety.

[0074] This embodiment implements a professional sensor layout plan. The forward main camera uses high dynamic range imaging technology, with a field of view of 120 degrees, a resolution of 1920×1080, and a frame rate of 30fps, specifically for capturing the front road scene. Wide-angle cameras are installed on both the left and right sides, with a field of view of 180 degrees, enhancing the perception of side vehicles and obstacles. The installation position and angle of the cameras are carefully designed to ensure a dead-angle-free coverage of the field of view.

[0075] In this embodiment, a complete data acquisition link is established. The navigation system provides real-time navigation instructions based on a high-precision map, including information such as the type, distance, and recommended lane of the next key action point. The vehicle speed information is collected by a high-precision Hall sensor with a sampling frequency of 100 Hz, and a Kalman filter is introduced to eliminate measurement noise. The control quantity is collected using a high-precision digital potentiometer to record the positions of the accelerator, brake pedals, and steering wheel angle, and the sampling frequency is synchronized with the vehicle speed sensor.

[0076] In this embodiment, a reliable time synchronization mechanism is designed. All sensor data is marked with a unified timestamp system and microsecond-level time synchronization is achieved based on the Network Time Protocol (NTP). The data acquisition system adopts a multi-buffer design to ensure the real-time performance and integrity of different sensor data. At the same time, a data integrity check mechanism is implemented to automatically identify and handle data loss or delay situations.

[0077] In this embodiment, an innovative training sample organization method is constructed. The collected sensor data is stored in a distributed database and an index structure is established according to the timestamp. The database supports high-concurrency read and write and fast retrieval, facilitating subsequent data analysis and model training. At the same time, a data compression storage mechanism is implemented to perform lossless compression on image data and reduce storage space occupancy. To ensure data security, a complete data backup and recovery mechanism is established.

[0078] Through the above technological innovations in this embodiment, several key problems in autonomous driving data acquisition are effectively solved: incomplete scenario coverage, unstable data quality, poor sensor synchronization, etc. In practical applications, this solution can collect high-quality expert driving data, providing a reliable data foundation for subsequent model training. It is particularly suitable for autonomous driving tasks in complex traffic environments, and significantly improves the quality and usability of training data through professional data acquisition and processing processes. The systematicness and integrity of this solution enable it to support model training requirements in different scenarios, and achieve efficient organization and management of training data through a standardized data processing process.

[0079] In an embodiment of the autonomous driving model training method combining GAIL and PPO in this application, it may also specifically include the following content:

[0080] Step S301: Preprocess the environmental image information, normalize the RGB image to a preset pixel range, perform data augmentation using random cropping and rotation methods, perform one-hot encoding processing on the navigation instruction information, perform maximum-minimum normalization processing on the vehicle speed information, linearly map the control actions to a standard interval, and generate standardized training features;

[0081] Step S302: Construct a training sequence based on a sliding time window. Set a time window with a fixed length, sort the normalized training features within the same time window according to timestamps to construct subsequences, perform temporal alignment on the subsequences, construct the aligned subsequences into a training sample sequence, and calculate the state transition relationship between adjacent frames in the training sample sequence.

[0082] Optionally, in this embodiment, a comprehensive image preprocessing process is first designed. For the RGB images collected by the vehicle-mounted camera, the adaptive histogram equalization technique is used to enhance the image contrast and effectively process the imaging differences under different lighting conditions. The image normalization adopts a per-channel processing method to map the pixel values to the [0,1] interval, and the calculation method is pixel_norm = (pixel - mean) / std, where mean and std are the mean and standard deviation of the training set respectively. This normalization process improves the stability and convergence speed of model training.

[0083] This embodiment implements a rich data augmentation strategy. The random cropping operation samples from the four corners and the central region while maintaining the aspect ratio of the original image to increase the spatial diversity of the data. The range of the random rotation angle is limited between [-5,5] degrees to simulate the small changes in vehicle posture. At the same time, operations of randomly adjusting brightness, contrast, and saturation are introduced to enhance the model's adaptability to different lighting conditions. To ensure the quality of the augmented images, an automatic quality assessment mechanism is designed to filter out severely distorted samples.

[0084] This embodiment constructs an efficient navigation instruction encoding method. The navigation instructions are converted into one-hot encoding vectors, and the vector dimension corresponds to the number of possible navigation actions. The basic actions include going straight, turning left, turning right, lane changing, etc., and each action corresponds to a unique encoding position. To enhance the expressive ability of the encoding, information on the urgency of action execution, such as the distance to the next action point, is added and encoded through an additional dimension. This design enables the model to better understand and execute navigation intentions.

[0085] This embodiment designs an accurate vehicle speed information processing mechanism. The maximum - minimum normalization method is used to process the vehicle speed data, and the normalization formula is speed_norm = (speed - speed_min) / (speed_max - speed_min), where speed_min and speed_max are the minimum and maximum vehicle speeds in the training data respectively. To handle the influence of outliers, an adaptive truncation threshold is set to limit the vehicle speed values outside the normal range within a reasonable interval. At the same time, the trend information of the vehicle speed is retained to facilitate the model to learn the speed control strategy.

[0086] This embodiment realizes a reliable control action standardization scheme. A linear mapping in the range of [0, 1] is performed on the throttle control amount and the braking control amount, where 0 represents completely released and 1 represents completely depressed. The steering control amount is mapped to the range of [-1, 1], with negative values indicating a left turn and positive values indicating a right turn. To ensure the smoothness of control, an action smoothing processing mechanism is introduced, and the moving average method is used to eliminate sudden changes in the control signal. This processing helps the model learn continuous and stable control strategies.

[0087] This embodiment establishes an innovative time window mechanism. The window length is set to 2 seconds, and the sliding step is 0.1 seconds to ensure the complete capture of continuous driving behaviors. Within each time window, various features are sorted and aligned according to timestamps to handle the timing differences caused by inconsistent sampling frequencies. For sensor data with a higher sampling frequency, linear interpolation is used for resampling to ensure the time consistency of all features.

[0088] This embodiment constructs a complete sequence processing framework. The aligned subsequences contain data in four dimensions: environmental images, navigation instructions, vehicle speed information, and control actions. Dynamic time warping algorithm is used for time series alignment to handle the time delays between different sensors. At the same time, the state transition relationships between adjacent frames are calculated, including position changes, speed changes, and heading angle changes, and this information provides important dynamic features for the model.

[0089] This embodiment designs a reliable state transition analysis method. Based on the aligned sequence data, the state changes between each pair of adjacent frames are calculated. The position change is estimated by the vehicle speed and the time interval, the speed change is directly obtained from the sensor data, and the heading angle change is calculated based on the steering control amount. These state transition information reflect the motion characteristics of the vehicle and help the model learn driving strategies.

[0090] Through the above technological innovations, this embodiment effectively solves several key problems in autonomous driving data processing: unstable data quality, inconsistent feature scales, difficult time series alignment, etc. In practical applications, this scheme can generate high-quality standardized training data, providing a reliable data basis for model training. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through professional data preprocessing and sequence construction processes, the quality and usability of the training data are significantly improved. The systematicness and integrity of this scheme enable it to support the model training needs in different scenarios, and through standardized data processing processes, the efficient organization and management of training data are achieved.

[0091] In an embodiment of the autonomous driving model training method combining GAIL and PPO in this application, it may also specifically include the following content:

[0092] Step S401: Construct a dual-branch structure for the discriminator network. Arrange a spatial attention module and a channel attention module in the frame-level discrimination branch. The spatial attention module calculates the weight coefficients at different spatial positions on the image feature map, and the channel attention module calculates the weight coefficients of different feature channels. The weight coefficients are weighted and combined with the original features to obtain enhanced features. Arrange a long short-term memory network in the trajectory-level discrimination branch to extract temporal features, and input the frame-level features and trajectory-level features into a multi-layer perceptron for fusion;

[0093] Step S402: Construct a policy generation network structure. Use a residual convolutional neural network as the environmental image feature extractor, and use a fully connected layer to process the navigation instruction information and vehicle speed information to obtain a state feature vector. Input the state feature vector into the policy network to generate the probability distribution of vehicle control actions, and use the reparameterization trick to sample from the probability distribution to obtain a deterministic action. Based on the similarity score output by the discriminator network, construct a generative adversarial loss function to optimize the parameters of the policy generation network.

[0094] Optionally, in this embodiment, an innovative dual-branch discriminator structure is first designed. The frame-level discrimination branch uses ResNet-50 as the backbone network, and a spatial attention module and a channel attention module are embedded after each residual block. The spatial attention module generates a spatial weight map through a two-layer convolutional network to highlight the feature expressions of key regions in the image, such as important targets like the vehicle ahead and traffic signs. The channel attention module calculates the importance weights of feature channels through global average pooling and fully connected layers to enhance the response of feature channels related to driving decisions.

[0095] This embodiment realizes an efficient feature enhancement mechanism. The weight calculation of the spatial attention module adopts an adaptive threshold method to dynamically adjust the attention intensity according to the statistical characteristics of the feature map. The channel attention module introduces a residual connection structure to highlight the contribution of important channels while retaining the original features. The outputs of the two attention mechanisms are adaptively fused through a gating unit to ensure the effectiveness of feature enhancement. This design enables the model to accurately capture the key information in the driving scenario.

[0096] This embodiment constructs a reliable temporal feature extraction network. The trajectory-level discrimination branch adopts a two-layer LSTM structure with a hidden layer dimension of 256 to process continuous driving behavior sequences. The gating mechanism of the LSTM can effectively model long-term dependencies and capture the temporal patterns of driving behaviors. To enhance the memory ability of the network, a residual connection is added to the LSTM unit to alleviate the gradient vanishing problem in long sequence training. At the same time, an attention mechanism is introduced to dynamically focus on the key time steps in the sequence.

[0097] This embodiment designs an innovative feature fusion strategy. The frame-level features and trajectory-level features are input into a three-layer perceptron for fusion, and each layer is followed by batch normalization and ReLU activation function. The fusion process adopts an adaptive weight mechanism to dynamically adjust the fusion ratio according to the information content of the two types of features. To enhance the expressive ability of the features, skip connections are added in the fusion layer to ensure the complete transmission of information. The fused features are output as similarity scores through the final discriminant layer.

[0098] This embodiment implements a stable policy generation network. The environmental feature extractor adopts an improved ResNet structure, adding dilated convolution on the basis of the standard residual block to increase the receptive field range. The navigation instruction and vehicle speed information are processed through independent fully connected layers to generate low-dimensional feature vectors. The fusion of state features adopts an attention mechanism to dynamically adjust the importance weights of different features according to the current scene.

[0099] This embodiment constructs a reliable action generation mechanism. The policy network uses a Gaussian mixture model to represent the action space distribution, which contains multiple Gaussian components. Each component is described by a mean vector and a covariance matrix, and the number of components is determined according to the complexity of the control task. The differentiability of action sampling is achieved through the reparameterization trick, that is, using random noise and deterministic transformation to generate action samples. This design not only ensures the diversity of actions but also maintains the stability of training.

[0100] This embodiment establishes an innovative adversarial training framework. An adversarial loss function is constructed based on the similarity scores output by the discriminator, which includes a policy gradient term and an entropy regularization term. The policy gradient guides the network to generate actions closer to the expert behavior, and the entropy regularization prevents the policy from converging prematurely. The training process adopts an alternating optimization strategy, updating the policy network while fixing the discriminator to ensure the stability of training.

[0101] This embodiment designs a complete parameter optimization mechanism. The Adam optimizer is used for network training, and the learning rate is dynamically adjusted using the cosine annealing strategy. To prevent overfitting, dropout and L2 regularization are introduced. During the training process, the discriminant ability of the discriminator is monitored in real time. When the discriminant accuracy is too high, the entropy regularization coefficient is increased to maintain the adversarial balance. At the same time, a gradient clipping mechanism is implemented to avoid the problem of gradient explosion.

[0102] Through the above technological innovations, this embodiment effectively solves multiple key problems in the training of autonomous driving models: insufficient feature extraction, inaccurate temporal modeling, unstable action generation, etc. In practical applications, this solution can accurately imitate expert driving behaviors and generate safe and smooth control actions. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through multi-level feature extraction and adversarial learning, the driving ability of the model is significantly improved. The innovation and systematicness of this solution enable it to handle various driving scenarios, and through continuous adversarial training and optimization, the stable evolution of driving strategies is achieved.

[0103] In an embodiment of the autonomous driving model training method combining GAIL and PPO in this application, the following specific content may also be included:

[0104] Step S501: Input the frame-level features and trajectory-level features into the feature fusion module, and use the weighted summation method with adaptive weights to fuse the two types of features. The adaptive weights are calculated based on the variance and mean of the feature vectors. Input the fused features into the output layer of the discriminator to obtain a similarity score, and perform sigmoid activation processing on the similarity score to obtain a discriminant probability value between 0 and 1.

[0105] Step S502: Optimize the discriminator network and the policy generation network using an alternating training strategy. Fix the parameters of the policy generation network to train the discriminator network to maximize the discrimination accuracy between expert data and generated data. Fix the parameters of the discriminator network to train the policy generation network to minimize the discrimination probability value of the generated data. During the training process, use a gradient penalty term to constrain the gradient norm of the discriminator, and dynamically adjust the training ratio of the two networks based on the discrimination accuracy on the validation set.

[0106] Optionally, this embodiment first designs an innovative feature fusion mechanism. The feature fusion module uses the adaptive weight method to integrate frame-level features and trajectory-level features, and the weight calculation considers the statistical characteristics of the features. Calculate the variance and mean for each feature vector. The variance reflects the information richness of the feature, and the mean reflects the overall strength of the feature. The adaptive weights are normalized through the softmax function to ensure that the weights sum to 1 while maintaining numerical stability. This design enables the model to dynamically adjust the importance of the two types of features according to different scenarios.

[0107] This embodiment realizes the accurate calculation of similarity scores. The fused features are processed through a three-layer fully connected network, followed by batch normalization and ReLU activation functions after each layer, and finally the original similarity score is output. To enhance the discrimination ability of the score, residual connections are introduced in the fully connected layer to ensure the complete transmission of information. The similarity score is mapped to the [0,1] interval through the sigmoid function and converted into a discriminant probability value. This probability representation intuitively reflects the similarity between the generated behavior and the expert behavior.

[0108] In this embodiment, a stable adversarial training framework is constructed. The training objective of the discriminator is to maximize the discrimination accuracy between expert data and generated data, and binary cross-entropy is used as the loss function. At the same time, label smoothing technology is introduced to soften the hard labels [0, 1] to [0.1, 0.9], improving the generalization ability of the model. The training objective of the policy network is to minimize the discrimination probability value of the generated data, and the generation strategy is optimized through gradient backpropagation.

[0109] In this embodiment, a reliable gradient constraint mechanism is designed. A gradient penalty term is added during the discriminator training process to limit the sensitivity of the discriminator to the input. The gradient penalty is achieved by calculating the gradient norm of the discriminator output with respect to the input, and the penalty intensity is increased when the gradient norm exceeds a preset threshold. This design effectively prevents the discriminator from overfitting and maintains the stability of training. At the same time, a gradient clipping mechanism is implemented to avoid the problem of gradient explosion.

[0110] In this embodiment, an innovative training scheduling strategy is implemented. Based on the discrimination accuracy on the validation set, the training ratio of the discriminator and the policy network is dynamically adjusted. When the discriminator performs too strongly, the training frequency of the policy network is increased; when the discriminator performs insufficiently, the training frequency of the discriminator is increased. This adaptive scheduling ensures the balance of adversarial training and prevents any network from gaining an absolute advantage.

[0111] In this embodiment, a complete validation and evaluation mechanism is established. The model performance is regularly evaluated on an independent validation set, and the evaluation metrics include discrimination accuracy, diversity and stability of the generation strategy. The validation set contains a variety of typical driving scenarios to ensure the representativeness of the evaluation results. Based on the validation results, training parameters such as the learning rate and regularization strength are dynamically adjusted to optimize the training process.

[0112] In this embodiment, an innovative early stopping strategy is constructed. Multiple early stopping conditions are set, including the convergence of the discrimination accuracy and the stability of the generation strategy. When the validation metrics do not improve significantly for multiple consecutive epochs, the learning rate is automatically reduced or the training of the corresponding network is stopped. This mechanism avoids performance degradation caused by overtraining and saves computing resources at the same time.

[0113] In this embodiment, an efficient training process monitoring mechanism is designed. Key training metrics are recorded in real time, including loss values, gradient norms, feature distributions, etc. The training process is visualized to facilitate the timely discovery and handling of training anomalies. At the same time, a checkpoint saving mechanism is implemented to save the model parameters regularly, supporting the resumption of training from a breakpoint.

[0114] Through the above technological innovations, this embodiment effectively solves several key problems in generative adversarial training: insufficient feature fusion, unstable adversarial training, difficult model convergence, etc. In practical applications, this solution can stably and effectively train an autonomous driving model to generate control strategies that conform to the driving style of experts. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through a carefully designed adversarial training mechanism, the imitation learning ability of the model is significantly improved. The innovation and systematicness of this solution enable it to handle various driving scenarios and achieve the stable evolution of driving strategies through continuous adversarial training and optimization.

[0115] In an embodiment of the autonomous driving model training method combining GAIL and PPO in this application, the following content may also be specifically included:

[0116] Step S601: Construct a multi-task learning objective function, weight and combine the policy gradient loss of the driving action generation task and the mean square error loss of the trajectory prediction task, calculate the KL divergence between the action distributions of the new policy and the baseline policy, calculate the adaptive confidence interval threshold based on the mean and standard deviation of the KL divergence in the most recent N rounds of iterations, and use the confidence interval threshold as the constraint condition for policy update;

[0117] Step S602: Use the importance sampling method to estimate the expected return of the new policy, calculate the advantage function value for each time step based on the generalized advantage estimation, normalize the advantage function value, multiply the normalized advantage function value by the logarithmic likelihood of the action probability to obtain the policy gradient, and update the parameters in the direction of the policy gradient. When the KL divergence between the new policy and the baseline policy exceeds the confidence interval threshold, terminate this round of update in advance.

[0118] Optionally, this embodiment first designs an innovative multi-task learning framework. The driving action generation task is optimized using the policy gradient method, and the loss function includes an expected return term and an entropy regularization term. The trajectory prediction task guides the model to learn the vehicle kinematic characteristics through the mean square error loss. The losses of the two tasks are combined through an adaptive weight coefficient, and the weight coefficient is dynamically adjusted according to the training difficulty of the task. This design enables the model to simultaneously learn the immediate control decision-making and long-term trajectory planning capabilities.

[0119] This embodiment implements an accurate KL divergence constraint mechanism. By calculating the distribution difference between the new policy and the baseline policy in the action space, the amplitude of policy update is monitored. Based on a sliding window, the KL divergence data in the most recent N rounds of iterations is statistically analyzed to calculate the mean and standard deviation, and a reasonable confidence interval threshold is dynamically determined. When the KL divergence caused by policy update exceeds the threshold, the update process is terminated in a timely manner to ensure the continuity and stability of policy changes.

[0120] This embodiment constructs a reliable importance sampling framework. A trajectory-based policy optimization method is adopted to collect the state-action sequences generated by the interaction between the current policy and the environment. The sampling bias is corrected through importance weights, and the weight calculation takes into account the probability ratio of the old and new policies and the time decay factor. To prevent the variance of the weights from being too large, a truncation technique is introduced to limit the weight range and improve the stability of the sampling estimation.

[0121] This embodiment designs an innovative advantage function estimation method. The Generalized Advantage Estimation (GAE) is used to calculate the advantage value at each time step, comprehensively considering the immediate reward and the future discounted return. The GAE parameter λ controls the trade-off between the immediate reward and the long-term return. A larger value of λ is beneficial for reducing bias but may increase the variance. A baseline function is introduced during the calculation of the advantage function to reduce the variance, and the baseline function predicts the state value function through a value network.

[0122] This embodiment implements a stable normalization mechanism. The calculated advantage function values are normalized at the batch level to eliminate the scale differences between different trajectories. The normalization process considers the distribution characteristics of the advantage values, and outliers are truncated through an adaptive threshold to maintain the numerical stability. The normalized advantage values are more suitable as the weights of the policy gradient to guide the effective update of the policy.

[0123] This embodiment establishes a complete policy update mechanism. The normalized advantage function values are multiplied by the log-likelihood probability of the actions to construct the calculation expression of the policy gradient. An adaptive learning rate method is adopted during the parameter update process, and the update step size is dynamically adjusted according to the gradient norm. At the same time, a gradient clipping mechanism is implemented to avoid the training instability caused by large gradients.

[0124] This embodiment constructs an innovative early termination policy. During the policy update process, the change of the KL divergence is monitored in real time, and when the divergence exceeds the confidence interval threshold, the update is immediately stopped. The early termination mechanism effectively prevents the performance collapse caused by over-updating of the policy, and at the same time realizes the stable optimization of the policy through progressive multi-round updates. To enhance the robustness of the algorithm, a rollback mechanism is introduced to allow recovery to the previous stable state when the update fails.

[0125] This embodiment designs an efficient training management mechanism. Parallel environments are used to collect training data, and multiple environments interact with the policy simultaneously to improve the data collection efficiency. Key metrics, including the expected return, KL divergence, policy entropy, etc., are monitored in real time during the training process, and the training progress is displayed through a visualization tool. At the same time, the function of resuming training from a breakpoint is implemented to support the continuous optimization of long-term training tasks.

[0126] Through the above technological innovations, this embodiment effectively solves several key problems in reinforcement learning training: unstable policy updates, low sample utilization efficiency, difficult training convergence, etc. In practical applications, this solution can stably and efficiently train an autonomous driving model and generate safe and reliable driving strategies. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through multi-task learning and constraint optimization, it significantly improves the driving ability and generalization performance of the model. The innovation and systematicness of this solution enable it to handle various driving scenarios, and through continuous policy optimization and evolution, it realizes the stable evolution of the intelligent driving model.

[0127] In an embodiment of the method for training an autonomous driving model combining GAIL and PPO in this application, it may further specifically include the following content:

[0128] Step S701: Construct an experience replay pool structure. Construct a state transition tuple from the states, actions, rewards, and next states collected in each training episode. Calculate the temporal difference error of each state transition tuple, set a priority weight for the state transition tuple based on the temporal difference error, and store the state transition tuple and its priority weight in the experience replay pool. When the capacity of the experience replay pool reaches the upper limit, remove the historical data with the lowest priority.

[0129] Step S702: Perform importance sampling from the experience replay pool according to the priority weights. Mix the sampled historical data with the new data generated by the current policy to construct a training batch. Calculate the policy loss and value loss based on the mixed training data, update the parameters of the policy generation network by gradient, and determine whether the performance of the policy network converges on the validation set. When there is no significant improvement in the validation performance for multiple consecutive rounds, stop training and output the model parameters.

[0130] Optionally, this embodiment first designs an innovative experience replay pool structure. The state transition tuple includes four elements: the current state, the executed action, the obtained reward, and the next state. Among them, the state information includes multi-modal data such as environmental images, navigation instructions, and vehicle speed. The temporal difference error is obtained by calculating the difference between the actually obtained reward and the expected value, which reflects the degree of correction of the transfer sample to the value estimation. The priority weight is calculated based on the absolute value of the temporal difference error. Samples with larger errors have higher priorities. This design ensures the efficient utilization of key experiences.

[0131] This embodiment realizes an efficient data management mechanism. The experience replay pool adopts a hierarchical storage structure, and divides the data into multiple levels according to the priority weights. Each level organizes the data using a doubly linked list, supporting fast insertion and deletion operations. When the pool capacity reaches the upper limit, the historical data in the low-priority level is preferentially removed. At the same time, a data compression mechanism is implemented to perform lossless compression on large-capacity data such as images to improve the storage efficiency.

[0132] This embodiment constructs a reliable sampling strategy. During importance sampling, randomness and priority weights are combined to avoid over-focusing on high-priority samples. The sampling probability is proportional to the priority weight, but the concentration degree of sampling is adjusted by a temperature parameter. To compensate for the bias brought by priority sampling, importance weights are introduced for correction to ensure the unbiasedness of parameter updates. This design improves the sample utilization efficiency while maintaining the stability of training.

[0133] This embodiment designs an innovative method for constructing training batches. The historical data sampled from the experience pool and the new data generated by the current policy are mixed according to a fixed ratio, and the ratio of new and old data is dynamically adjusted according to the verification performance. The construction of the mixed data takes into account the diversity of scenarios to ensure that each batch contains different types of driving scenarios. At the same time, a data augmentation mechanism is implemented to increase the diversity of training samples through random perturbations.

[0134] This embodiment implements a stable loss calculation framework. The policy loss includes a policy gradient term and an entropy regularization term, and the value loss adopts the mean square error form. The weight coefficients of the loss function are determined by grid search and remain fixed during the training process. To improve the training efficiency, the mini-batch method is used to calculate the gradient, and the batch size is dynamically adjusted according to the video memory capacity. At the same time, the gradient accumulation technique is introduced to support large-batch training.

[0135] This embodiment establishes a complete parameter update mechanism. The Adam optimizer is used for gradient update, and the learning rate is dynamically adjusted by the cosine annealing strategy. Gradient clipping is introduced during the parameter update process to limit the gradient norm and avoid drastic changes in parameters. At the same time, the parameter averaging technique is implemented to improve the stability of the model through moving averages. To prevent overfitting, a dropout layer and L2 regularization are added to the network.

[0136] This embodiment constructs a reliable verification and evaluation system. The model performance is regularly evaluated on an independent validation set, and the evaluation metrics include trajectory tracking accuracy, control smoothness, safety metrics, etc. The validation set contains a variety of typical driving scenarios to ensure the representativeness of the evaluation results. The judgment of performance convergence is based on the moving average of the validation metrics to avoid the influence of short-term fluctuations.

[0137] This embodiment designs an innovative early stopping strategy. When the improvement of the verification performance in N consecutive rounds is lower than the threshold, the early stopping mechanism is triggered. Early stopping not only considers the overall performance but also pays attention to the performance in specific scenarios to prevent the degradation of the performance in some key scenarios. At the same time, the optimal model parameters during the training process are saved to support the rollback operation of the model. This design ensures the efficiency and reliability of the training process.

[0138] Through the above technological innovations, this embodiment effectively solves multiple key problems in the training of autonomous driving models, such as low sample utilization efficiency, unstable training, and slow convergence speed. In practical applications, this solution can efficiently and stably train an intelligent model with good driving capabilities. It is particularly suitable for autonomous driving tasks in complex traffic environments. Through mechanisms such as experience replay and importance sampling, it significantly improves the learning efficiency and generalization ability of the model. The systematicness and integrity of this solution enable it to handle various driving scenarios, and through continuous optimization and verification, it realizes the stable evolution of the intelligent driving model.

[0139] In order to effectively solve the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improve the robustness and generalization ability of autonomous driving models, this application provides an embodiment of an autonomous driving model training device combining GAIL and PPO for implementing all or part of the content of the autonomous driving model training method combining GAIL and PPO. Refer to Figure 2 The autonomous driving model training device combining GAIL and PPO specifically includes the following content:

[0140] A training preprocessing module 10, which is used to collect and preprocess training data. In the autonomous driving simulation environment, multiple test sections are constructed, and continuous driving trajectory sequences of expert drivers in the test sections are collected. The environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process are recorded, and the environmental image information, navigation instruction information, vehicle speed information, and control actions are constructed into a training sample sequence according to the temporal relationship;

[0141] A framework construction module 20, which is used to construct a multi-stage training framework. First, a discriminator network and a policy generation network are trained using generative adversarial imitation learning. The discriminator network includes a frame-level discriminant branch and a trajectory-level discriminant branch. The frame-level discriminant branch extracts single-frame driving behavior features based on the attention mechanism, and the trajectory-level discriminant branch uses a recurrent neural network to process continuous driving behavior sequences. The features output by the frame-level discriminant branch and the trajectory-level discriminant branch are fused and then a similarity score is output. The policy generation network is trained to minimize the similarity score of the discriminator network;

[0142] A model generation module 30, which is used to iteratively optimize the policy generation network using the proximal policy optimization algorithm, construct a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task, set an adaptive confidence interval threshold based on historical training data, optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, store the state transition sequence during the training process in the experience replay pool, sample historical data from the experience replay pool to participate in policy update, and output the trained autonomous driving model through cyclic iteration.

[0143] As can be seen from the above description, the autonomous driving model training device combining GAIL and PPO provided by the embodiments of the present application can collect expert driving trajectory sequences through a simulation environment by designing a multi-stage training framework. A dual-branch discriminator network is constructed. The frame-level branch analyzes single-frame behavior features based on the attention mechanism, and the trajectory-level branch uses a recurrent neural network to process continuous driving sequences to achieve multi-scale feature fusion. The proximal policy optimization algorithm is used for iterative training, a multi-task learning objective function is constructed to optimize both driving action generation and trajectory prediction, and an adaptive confidence interval constraint and an experience replay mechanism are introduced. This method effectively solves the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improves the robustness and generalization ability of the autonomous driving model.

[0144] At the hardware level, in order to effectively solve the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improve the robustness and generalization ability of the autonomous driving model, the present application provides an embodiment of an electronic device for implementing all or part of the content in the autonomous driving model training method combining GAIL and PPO. The electronic device specifically includes the following:

[0145] A processor, a memory, a communications interface, and a bus; wherein, the processor, the memory, and the communications interface complete communication with each other through the bus; the communications interface is used to implement information transmission between the autonomous driving model training device combining GAIL and PPO and related devices such as the core business system, the user terminal, and the relevant database. The logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., and this embodiment is not limited thereto. In this embodiment, the logic controller can be implemented with reference to the embodiments of the autonomous driving model training method combining GAIL and PPO and the embodiments of the autonomous driving model training device combining GAIL and PPO, and the content thereof is incorporated herein, and the repeated parts will not be elaborated.

[0146] It can be understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0147] In practical applications, the part of the method for training an autonomous driving model that combines GAIL and PPO can be executed on the side of the electronic device as described above, or all operations can be completed in the client device. Specifically, it can be selected according to the processing capacity of the client device and the limitations of the user usage scenario, etc. This application does not make any limitations in this regard. If all operations are completed in the client device, the client device may further include a processor.

[0148] The above-mentioned client device may have a communication module (i.e., a communication unit) and can communicate with a remote server to achieve data transmission with the server. The server may include a server on the side of the task scheduling center. In other implementation scenarios, it may also include a server of an intermediate platform, such as a server of a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or may include a server cluster composed of multiple servers, or a server structure of a distributed device.

[0149] Figure 3 It is a schematic block diagram of the system composition of the electronic device 9600 according to an embodiment of the present application. As Figure 3 shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It should be noted that this Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to achieve telecommunication functions or other functions.

[0150] In one embodiment, the function of the method for training an autonomous driving model that combines GAIL and PPO can be integrated into the central processing unit 9100. Among them, the central processing unit 9100 may be configured to perform the following controls:

[0151] Step S101: Collect training data and preprocess it. Construct multiple test sections in the autonomous driving simulation environment, collect the continuous driving trajectory sequences of expert drivers in the test sections, record the environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process, and construct the environmental image information, navigation instruction information, vehicle speed information, and control actions into a training sample sequence according to the time sequence relationship;

[0152] Step S102: Construct a multi-stage training framework. First, use generative adversarial imitation learning to train the discriminator network and the policy generation network. The discriminator network includes a frame-level discriminant branch and a trajectory-level discriminant branch. The frame-level discriminant branch extracts single-frame driving behavior features based on the attention mechanism. The trajectory-level discriminant branch uses a recurrent neural network to process continuous driving behavior sequences, fuses the features output by the frame-level discriminant branch and the trajectory-level discriminant branch, and then outputs a similarity score. Train the policy generation network to minimize the similarity score of the discriminator network.

[0153] Step S103: Use the proximal policy optimization algorithm to iteratively optimize the policy generation network, construct a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task simultaneously, set an adaptive confidence interval threshold based on historical training data, optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, store the state transition sequence during the training process in the experience replay pool, sample historical data from the experience replay pool to participate in policy update, and output the trained autonomous driving model through loop iteration.

[0154] As can be seen from the above description, the electronic device provided in the embodiments of the present application designs a multi-stage training framework, collects expert driving trajectory sequences through a simulation environment, constructs a dual-branch discriminator network, where the frame-level branch analyzes single-frame behavior features based on the attention mechanism, and the trajectory-level branch uses a recurrent neural network to process continuous driving sequences to achieve multi-scale feature fusion. Use the proximal policy optimization algorithm for iterative training, construct a multi-task learning objective function to optimize both driving action generation and trajectory prediction, and introduce an adaptive confidence interval constraint and an experience replay mechanism. This method effectively solves the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improves the robustness and generalization ability of the autonomous driving model.

[0155] In another embodiment, the autonomous driving model training device combining GAIL and PPO can be separately configured from the central processing unit 9100. For example, the autonomous driving model training device combining GAIL and PPO can be configured as a chip connected to the central processing unit 9100, and the functions of the autonomous driving model training method combining GAIL and PPO are realized through the control of the central processing unit.

[0156] As Figure 3 shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily have to include all the components shown in Figure 3 ; in addition, the electronic device 9600 may further include components not shown in Figure 3 , and reference can be made to the prior art.

[0157] As Figure 3 shown, the central processing unit 9100, sometimes also referred to as a controller or operation control, may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives inputs and controls the operations of the various components of the electronic device 9600.

[0158] Among them, the memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. It can store the above-mentioned information related to failures, and can also store programs for executing relevant information. And the central processing unit 9100 can execute the programs stored in the memory 9140 to achieve information storage or processing, etc.

[0159] The input unit 9120 provides inputs to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to supply power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display can be, for example, an LCD display, but is not limited thereto.

[0160] The memory 9140 can be a solid-state memory. For example, it can be a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It can also be a memory that stores information even when power is off, can be selectively erased and has more data. Examples of such a memory are sometimes referred to as EPROMs, etc. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 can include an application / function storage unit 9142, which is used to store application programs and function programs or the processes for operating the electronic device 9600 through the central processing unit 9100.

[0161] The memory 9140 can also include a data storage unit 9143, which is used to store data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 can include various drivers of the electronic device for communication functions and / or for executing other functions of the electronic device (such as a messaging application, an address book application, etc.).

[0162] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0163] Based on different communication technologies, in the same electronic device, multiple communication modules 9110 can be provided, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, so as to implement general telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is also coupled to a central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0164] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps in the above-described autonomous driving model training method that combines GAIL and PPO with the execution subject being a server or a client. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all steps in the above-described autonomous driving model training method that combines GAIL and PPO with the execution subject being a server or a client are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0165] Step S101: Collect training data and preprocess it. Construct multiple test sections in the autonomous driving simulation environment, collect the continuous driving trajectory sequences of expert drivers in the test sections, record the environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process, and construct the environmental image information, navigation instruction information, vehicle speed information, and control actions into a training sample sequence according to the temporal relationship;

[0166] Step S102: Construct a multi-stage training framework. First, use generative adversarial imitation learning to train a discriminator network and a policy generation network. The discriminator network includes a frame-level discriminant branch and a trajectory-level discriminant branch. The frame-level discriminant branch extracts single-frame driving behavior features based on an attention mechanism, and the trajectory-level discriminant branch uses a recurrent neural network to process the continuous driving behavior sequence. The features output by the frame-level discriminant branch and the trajectory-level discriminant branch are fused and then a similarity score is output. Train the policy generation network to minimize the similarity score of the discriminator network;

[0167] Step S103: Use the Proximal Policy Optimization (PPO) algorithm to iteratively optimize the policy generation network, construct a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task simultaneously, set an adaptive confidence interval threshold based on historical training data, optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, store the state transition sequences during the training process in the experience replay pool, sample historical data from the experience replay pool to participate in policy update, and output the trained autonomous driving model through iterative loops.

[0168] As can be seen from the above description, the computer-readable storage medium provided by the embodiments of the present application, by designing a multi-stage training framework, collects expert driving trajectory sequences through a simulation environment. Construct a dual-branch discriminator network. The frame-level branch analyzes single-frame behavior features based on the attention mechanism, and the trajectory-level branch uses a recurrent neural network to process continuous driving sequences, realizing multi-scale feature fusion. Use the Proximal Policy Optimization (PPO) algorithm for iterative training, construct a multi-task learning objective function to optimize both driving action generation and trajectory prediction, and introduce an adaptive confidence interval constraint and an experience replay mechanism. This method effectively solves the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improves the robustness and generalization ability of the autonomous driving model.

[0169] The embodiments of the present application also provide a computer program product that can implement all the steps in the autonomous driving model training method combining GAIL and PPO with the execution subject being a server or a client in the above embodiments. When the computer program / instructions are executed by a processor, the steps of the autonomous driving model training method combining GAIL and PPO are implemented. For example, the computer program / instructions implement the following steps:

[0170] Step S101: Collect and preprocess training data. Construct multiple test sections in the autonomous driving simulation environment, collect the continuous driving trajectory sequences of expert drivers in the test sections, record the environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process, and construct the environmental image information, navigation instruction information, vehicle speed information, and control actions into a training sample sequence according to the time sequence relationship;

[0171] Step S102: Construct a multi-stage training framework. First, use Generative Adversarial Imitation Learning (GAIL) to train the discriminator network and the policy generation network. The discriminator network includes a frame-level discriminant branch and a trajectory-level discriminant branch. The frame-level discriminant branch extracts single-frame driving behavior features based on the attention mechanism, and the trajectory-level discriminant branch uses a recurrent neural network to process continuous driving behavior sequences. The features output by the frame-level discriminant branch and the trajectory-level discriminant branch are fused and then a similarity score is output. Train the policy generation network to minimize the similarity score of the discriminator network;

[0172] Step S103: Use the Proximal Policy Optimization (PPO) algorithm to iteratively optimize the policy generation network, construct a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task simultaneously, set an adaptive confidence interval threshold based on historical training data, optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, store the state transition sequence during the training process in the experience replay pool, sample historical data from the experience replay pool to participate in policy update, and output the trained autonomous driving model through loop iteration.

[0173] As can be seen from the above description, the computer program product provided by the embodiments of this application designs a multi-stage training framework and collects expert driving trajectory sequences through a simulation environment. A dual-branch discriminator network is constructed. The frame-level branch analyzes single-frame behavior features based on the attention mechanism, and the trajectory-level branch uses a recurrent neural network to process continuous driving sequences to achieve multi-scale feature fusion. The Proximal Policy Optimization algorithm is used for iterative training, a multi-task learning objective function is constructed to optimize both driving action generation and trajectory prediction, and an adaptive confidence interval constraint and an experience replay mechanism are introduced. This method effectively solves the deficiencies of traditional technologies in aspects such as driving behavior evaluation and policy optimization, and significantly improves the robustness and generalization ability of the autonomous driving model.

[0174] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (devices), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce an apparatus for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0176] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 specified in one or more of the blocks.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 specified in one or more of the blocks.

[0178] Specific embodiments are used in the present invention to illustrate the principles and implementation manners of the present invention. The description of the above embodiments is only for helping to understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for training an autonomous driving model combining GAIL and PPO, characterized in that The method includes: Collecting training data and preprocessing it, constructing multiple test sections in an autonomous driving simulation environment, collecting the continuous driving trajectory sequences of expert drivers in the test sections, recording the environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process, and constructing the environmental image information, navigation instruction information, vehicle speed information, and control actions into a training sample sequence according to the time sequence relationship; Constructing a multi-stage training framework. First, use generative adversarial imitation learning to train a discriminator network and a policy generation network. The discriminator network includes a frame-level discrimination branch and a trajectory-level discrimination branch. The frame-level discrimination branch extracts single-frame driving behavior features based on an attention mechanism. The trajectory-level discrimination branch uses a recurrent neural network to process continuous driving behavior sequences, fuses the features output by the frame-level discrimination branch and the trajectory-level discrimination branch, and then outputs a similarity score, and trains the policy generation network to minimize the similarity score of the discriminator network; Using the proximal policy optimization algorithm to iteratively optimize the policy generation network, constructing a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task at the same time, setting an adaptive confidence interval threshold based on historical training data, optimizing the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, storing the state transition sequence during the training process in an experience replay pool, sampling historical data from the experience replay pool to participate in policy update, and outputting the trained autonomous driving model through cyclic iteration.

2. The method for training an autonomous driving model combining GAIL and PPO according to claim 1, wherein, The collecting training data and preprocessing it, constructing multiple test sections in an autonomous driving simulation environment, collecting the continuous driving trajectory sequences of expert drivers in the test sections, recording the environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process includes: Constructing a test section map in the autonomous driving simulation environment, dividing the test section map into a training area and a test area, setting multiple sampling start points and end points in the training area, generating a reference trajectory from the start point to the end point based on a path planning algorithm, and inputting the reference trajectory into the simulation environment control module; Collecting the driving data of expert drivers in the training area, deploying an in-vehicle forward camera and a wide-angle camera to collect environmental image information, collecting the navigation instruction information output by the navigation system and the vehicle speed information output by the vehicle speed sensor, recording the throttle control amount, steering control amount, and braking control amount during the expert driving process, and constructing a training sample set for the environmental image information, navigation instruction information, vehicle speed information, and control amount according to the sampling timestamps; 3. The method for training an autonomous driving model combining GAIL and PPO according to claim 1, wherein The constructing the environmental image information, navigation instruction information, vehicle speed information, and control actions into a training sample sequence according to the time sequence relationship includes: Preprocessing the environmental image information, normalizing the RGB image to a preset pixel range, performing data augmentation using random cropping and rotation methods, performing one-hot encoding processing on the navigation instruction information, performing maximum-minimum normalization processing on the vehicle speed information, and linearly mapping the control actions to a standard interval to generate standardized training features; Construct a training sequence based on a sliding time window. Set a time window with a fixed length. Sort the standardized training features within the same time window according to timestamps to construct subsequences. Perform temporal alignment on the subsequences, and construct the aligned subsequences into a training sample sequence. Calculate the state transition relationship between adjacent frames in the training sample sequence.

4. The method for training an autonomous driving model combining GAIL and PPO according to claim 1, characterized in that For the construction of the multi-stage training framework, first use generative adversarial imitation learning to train the discriminator network and the policy generation network. The discriminator network includes a frame-level discriminant branch and a trajectory-level discriminant branch. The frame-level discriminant branch extracts single-frame driving behavior features based on the attention mechanism. The trajectory-level discriminant branch uses a recurrent neural network to process continuous driving behavior sequences, including: Construct a dual-branch structure for the discriminator network. Arrange a spatial attention module and a channel attention module in the frame-level discriminant branch. The spatial attention module calculates the weight coefficients at different spatial positions on the image feature map. The channel attention module calculates the weight coefficients of different feature channels. Combine the weight coefficients with the original features through weighted combination to obtain enhanced features. Arrange a long short-term memory network in the trajectory-level discriminant branch to extract temporal features. Input the frame-level features and the trajectory-level features into a multi-layer perceptron for fusion; Construct the structure of the policy generation network. Use a residual convolutional neural network as the environmental image feature extractor. Use a fully connected layer to process the navigation instruction information and vehicle speed information to obtain a state feature vector. Input the state feature vector into the policy network to generate the probability distribution of vehicle control actions. Use the reparameterization trick to sample a deterministic action from the probability distribution. Construct a generative adversarial loss function based on the similarity score output by the discriminator network to optimize the parameters of the policy generation network.

5. The method for training an autonomous driving model combining GAIL and PPO according to claim 1, characterized in that, After fusing the features output by the frame-level discriminant branch and the trajectory-level discriminant branch and outputting the similarity score, training the policy generation network to minimize the similarity score of the discriminator network includes: Input the frame-level features and the trajectory-level features into the feature fusion module. Use the weighted summation method with adaptive weights to fuse the two types of features. The adaptive weights are calculated based on the variance and mean of the feature vectors. Input the fused features into the output layer of the discriminator to obtain the similarity score. Perform sigmoid activation processing on the similarity score to obtain a discriminant probability value between 0 and 1; Adopt an alternating training strategy to optimize the discriminator network and the policy generation network. Fix the parameters of the policy generation network to train the discriminator network to maximize the discrimination accuracy between expert data and generated data. Fix the parameters of the discriminator network to train the policy generation network to minimize the discrimination probability value of the generated data. Use a gradient penalty term to constrain the gradient norm of the discriminator during the training process. Dynamically adjust the training ratio of the two networks based on the discrimination accuracy on the validation set.

6. The method for training an autonomous driving model combining GAIL and PPO according to claim 1, wherein The proximal policy optimization algorithm is used to iteratively optimize the policy generation network, construct a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task simultaneously, set an adaptive confidence interval threshold based on historical training data, and optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, including: Construct a multi-task learning objective function, perform weighted combination of the policy gradient loss of the driving action generation task and the mean squared error loss of the trajectory prediction task, calculate the KL divergence between the action distributions of the new policy and the baseline policy, calculate the adaptive confidence interval threshold based on the mean and standard deviation of the KL divergence in the most recent N rounds of iterations, and use the confidence interval threshold as the constraint condition for policy update; Use the importance sampling method to estimate the expected return of the new policy, calculate the advantage function value for each time step based on generalized advantage estimation, perform normalization processing on the advantage function value, multiply the normalized advantage function value by the log-likelihood of the action probability to obtain the policy gradient, and update the parameters in the direction of the policy gradient. When the KL divergence between the new policy and the baseline policy exceeds the confidence interval threshold, terminate this round of update in advance.

7. The method for training an autonomous driving model combining GAIL and PPO according to claim 1, characterized in that, The state transition sequences during the training process are stored in the experience replay pool, historical data is sampled from the experience replay pool to participate in policy update, and the trained autonomous driving model is output through loop iteration, including: Construct an experience replay pool structure, construct state transition tuples from the states, actions, rewards, and next states collected in each training episode, calculate the temporal difference error of each state transition tuple, set priority weights for the state transition tuples based on the temporal difference error, store the state transition tuples and their priority weights in the experience replay pool, and remove the historical data with the lowest priority when the capacity of the experience replay pool reaches the upper limit; Perform importance sampling from the experience replay pool according to the priority weights, mix the sampled historical data with the new data generated by the current policy to construct a training batch, calculate the policy loss and value loss based on the mixed training data, perform gradient update on the parameters of the policy generation network, judge whether the performance of the policy network converges on the validation set, and stop training and output the model parameters when there is no significant improvement in the validation performance for multiple consecutive rounds.

8. An automatic driving model training device combining GAIL and PPO, characterized in that, The device includes: A training preprocessing module, which is used to collect and preprocess training data, construct multiple test sections in the autonomous driving simulation environment, collect the continuous driving trajectory sequences of expert drivers in the test sections, record the environmental image information, navigation instruction information, vehicle speed information, and corresponding control actions during the expert driving process, and construct the environmental image information, navigation instruction information, vehicle speed information, and control actions into a training sample sequence according to the temporal relationship; A framework construction module, which is used to construct a multi-stage training framework. First, generative adversarial imitation learning is adopted to train a discriminator network and a policy generation network. The discriminator network includes a frame-level discrimination branch and a trajectory-level discrimination branch. The frame-level discrimination branch extracts single-frame driving behavior features based on an attention mechanism. The trajectory-level discrimination branch uses a recurrent neural network to process continuous driving behavior sequences, fuses the features output by the frame-level discrimination branch and the trajectory-level discrimination branch, and then outputs a similarity score, and trains the policy generation network to minimize the similarity score of the discriminator network. A model generation module, which is used to iteratively optimize the policy generation network by using the proximal policy optimization algorithm, construct a multi-task learning objective function to optimize both the driving action generation task and the trajectory prediction task at the same time, set an adaptive confidence interval threshold based on historical training data, optimize the policy parameters under the constraint that the difference between the new policy and the baseline policy does not exceed the confidence interval threshold, store the state transition sequence during the training process in an experience replay pool, sample historical data from the experience replay pool to participate in policy update, and output the trained autonomous driving model through iterative loops.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the autonomous driving model training method combining GAIL and PPO according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the autonomous driving model training method combining GAIL and PPO according to any one of claims 1 to 7.

Images