Automatic driving digital twin agent system based on reinforcement learning training

Abstract

The invention relates to the technical field of artificial intelligence and automatic driving, and discloses an automatic driving digital twin agent system based on reinforcement learning training, which comprises an agent library construction module, a simulation and configuration module, a distributed simulation execution module, a scheduling and learning module, a strategy training module and a simulation migration module. The method comprises the following steps: constructing a modular digital twin agent library, combining to generate an urban traffic simulation scene, executing large-scale parallel simulation on a distributed computing cluster, implementing dynamic task scheduling and course learning according to an agent learning progress, and training a reinforcement learning strategy and a value network based on simulation data. And verifying the strategy model and implementing migration deployment from simulation to reality. According to the method, the training efficiency can be improved, the diversity of simulation environments is enriched, and the sim-to-real migration difficulty is reduced, so that the robustness of the strategy model is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and autonomous driving technology, and more specifically, to an autonomous driving digital twin intelligent agent system based on reinforcement learning training. Background Technology

[0002] With the rapid development of autonomous driving technology, reinforcement learning-based decision-making and control methods have become an important technical path for achieving high-level autonomous driving. Reinforcement learning continuously optimizes decision-making strategies through the interaction between the agent and the environment, theoretically capable of handling complex and ever-changing traffic scenarios. However, directly training reinforcement learning models in real-world environments presents challenges such as high safety risks, expensive data acquisition, and long training cycles. Therefore, training methods based on digital twin simulation environments have become the mainstream approach.

[0003] Existing simulation training systems generally suffer from the following technical problems: First, the construction of simulation environments lacks modularity and reusability, requiring remodeling each time a new scenario is built, resulting in low development efficiency; second, the diversity of simulation scenarios is insufficient, making it difficult to cover various extreme situations and long-tail scenarios in the real world, resulting in limited generalization ability of the trained policy models; third, the training process lacks an adaptive adjustment mechanism, failing to dynamically adjust the task difficulty according to the learning progress of the agent, leading to low training efficiency or premature entrapment in local optima.

[0004] Furthermore, the transfer from simulation to real-world environments has always been a key challenge in reinforcement learning applications. Due to differences in physical parameters and sensor noise characteristics between simulation and real environments, policy models trained well in simulation often perform poorly in real-world environments. Existing technologies lack systematic methods for domain adaptation and robustness enhancement, resulting in low success rates for actual deployment of policy models and requiring extensive fine-tuning in real-world environments, which increases deployment costs and security risks. Summary of the Invention

[0005] This invention provides an autonomous driving digital twin intelligent agent system based on reinforcement learning training, which solves the technical problems of low training efficiency, single simulation environment, and difficulty in sim-to-real transfer in related technologies.

[0006] This invention provides an autonomous driving digital twin intelligent agent system based on reinforcement learning training, comprising:

[0007] The intelligent entity library construction module is used to construct a modular digital twin intelligent entity library, resulting in a modular digital twin intelligent entity library;

[0008] The simulation and configuration module is used to receive the modular digital twin intelligent agent library, combine and generate urban traffic simulation scenarios, configure environmental parameters, and obtain executable simulation scenarios.

[0009] The distributed simulation execution module is used to receive executable simulation scenarios, execute large-scale parallel simulations on a distributed computing cluster, and obtain optimized training datasets and performance statistics.

[0010] The scheduling and learning module is used to receive the optimized training dataset and performance statistics, implement dynamic task scheduling and course learning according to the learning progress of the agent, and obtain the updated simulation scenario and training monitoring results.

[0011] The policy training module is used to receive updated simulation scenarios and training monitoring results, train reinforcement learning policies and value networks based on simulation data, and obtain the trained policy model and performance curve.

[0012] The simulation migration module is used to verify the strategy model and implement the migration deployment from simulation to reality, and obtain the final deployment strategy model and evaluation report.

[0013] In a preferred embodiment, the intelligent agent library construction module includes:

[0014] A layered decoupling design approach is adopted to decompose the agent into a data layer, a model layer, and an interaction layer; the data layer uses a parametric modeling method to define the morphological data set and a dynamic attribute extraction method to define the state data set.

[0015] The model layer configures a vehicle dynamics model based on Ackermann steering geometry for vehicle-type intelligent agents, a mass dynamics model for pedestrians to obtain a physical model, and a forward-looking camera, a 360-degree LiDAR and a millimeter-wave radar for autonomous vehicles to obtain a perception model.

[0016] The interaction layer defines the state query interface, action execution interface, and environment awareness interface.

[0017] In a preferred embodiment, the simulation and configuration module includes:

[0018] Using high-precision map data as input, the two-dimensional map information is expanded into a three-dimensional spatial model through a three-dimensional scene reconstruction algorithm, and the scene space is divided into hierarchical grid units using a spatial index construction algorithm.

[0019] Select the corresponding type of template from the modular digital twin intelligent agent library and create instances, assign a unique identifier to each instance, and use a spatial distribution algorithm to assign an initial position and initial orientation to each intelligent agent;

[0020] By using an environmental parameter randomization method to set weather conditions, lighting conditions, and road surface condition parameters, and adjusting the friction coefficient, visibility distance, and sensor noise level in the scene, an executable simulation scene is obtained.

[0021] In a preferred embodiment, the distributed simulation execution module includes:

[0022] A deep copy algorithm is used to create K independent scene replicas, and a load balancing scheduling algorithm is used to distribute the K scene replicas to M computing nodes;

[0023] At each time step t, a neighborhood query algorithm is used to find other agents and environmental objects within the perception range of each agent. Observation data is generated using a sensor simulation method. For trainable neural network policies, the observation data is input into the policy network to obtain action output. The motion equation of the agent is solved using a numerical integration method. A collision detection algorithm is used to determine whether a collision has occurred. The immediate reward is calculated and the quadruplet is stored in the experience buffer.

[0024] In a preferred embodiment, the distributed simulation execution module further includes:

[0025] The training data is preprocessed, and the continuous numerical features are standardized. That is, the standardized feature is equal to the original feature value minus the mean of the standardized feature divided by the standard deviation of the standardized feature plus the numerical stability term.

[0026] One-hot encoding is used for categorical variables; dimensional normalization is performed on each component of the reward function, and the scaled forward reward is equal to the reduction in distance relative to the target at that time step divided by the reference distance scale, and the scaled velocity reward is equal to 1 minus the absolute difference between the current velocity and the expected velocity divided by the upper bound of the velocity; the action command is scaled to a range of -1 to 1, and the observation vector is pruned to eliminate the influence of outliers.

[0027] In a preferred embodiment, the scheduling and learning module includes:

[0028] Calculate the task success rate, average cumulative reward, collision rate and average completion time of each agent in the most recent H training segments to obtain a performance index vector;

[0029] Multiple difficulty levels are predefined, each corresponding to different scenario configuration parameters, including traffic density, environmental complexity, and probability of extreme events. An advancement condition judgment method is adopted. When the task success rate is greater than or equal to the success rate threshold and the collision rate is less than or equal to the collision rate threshold, a new difficulty level is set to the current difficulty plus 1. Based on the updated difficulty level, a new scenario configuration instruction is generated, and the parameters of the simulation scenario are dynamically updated to obtain the updated simulation scenario.

[0030] In a preferred embodiment, the policy training module includes:

[0031] A policy network and a value network are constructed. The policy network adopts a multi-branch network structure. The LiDAR branch uses PointNet++ network to extract point cloud features, and the vehicle state branch uses a fully connected network to extract state features. The two feature vectors are concatenated and then passed through a fully connected layer to obtain fused features. The mean and standard deviation of the actions are output through the Actor network structure.

[0032] After fusing features, the value network outputs state value estimates through two fully connected layers. The Xavier initialization method is used to randomly initialize the network parameters, and small batches of training samples are randomly sampled from the experience replay buffer to obtain the training batch.

[0033] In a preferred embodiment, the policy training module further includes:

[0034] The PPO algorithm is used for parameter updates, and the GAE method is used to calculate the advantage function. The advantage estimate is equal to the weighted sum of the temporal difference errors of the offset index from the current time to the future.

[0035] The probability ratio of the current policy to the old policy is calculated. The PPO loss is calculated using the pruning objective function. The loss function of the value network is calculated using the mean squared error loss. The gradient of the loss function with respect to the network parameters is calculated using the backpropagation algorithm. The network parameters are updated using the Adam optimizer. The training is considered to have converged when the relative rate of change of the average reward in the most recent W iteration windows is less than a threshold.

[0036] In a preferred embodiment, the simulation migration module includes:

[0037] A variety of test scenario sets are generated using scenario enumeration and combination methods, including basic test scenario sets, extreme test scenario sets, and adversarial test scenario sets;

[0038] Run all test scenarios in the digital twin environment, calculate the overall task success rate, overall collision rate and average reward, and obtain a simulation environment performance report;

[0039] The robustness of the strategy is enhanced by employing domain randomization techniques. More realistic noise is added to the sensor simulation in the simulation environment. Physical parameter values ​​are randomly sampled during training and testing, vehicle mass is sampled uniformly, and road friction coefficient is sampled.

[0040] In a preferred embodiment, the simulation migration module further includes:

[0041] A shadow mode deployment method is used for incremental verification. The strategy model synchronously receives data from real sensors and calculates the actions output by the model but does not actually execute them. The consistency ratio between the model output and the actual actions is calculated.

[0042] When the consistency ratio is higher than the threshold, the strategy model is loaded into the on-board computing platform of the real vehicle. TensorRT is used to accelerate and model quantization to optimize the model inference speed. Online learning methods are used to fine-tune the strategy model. The safety driver's intervention actions are regarded as expert demonstrations. A gradual decentralization strategy is adopted to gradually increase the proportion of time for the model to autonomously control, so as to obtain the final deployed strategy model.

[0043] The beneficial effects of this invention are as follows:

[0044] By constructing a modular digital twin intelligent agent library and employing a layered decoupling design method to decompose the intelligent agent into a data layer, a model layer, and an interaction layer, parameterized configuration and rapid reuse of the intelligent agent are achieved, significantly improving the efficiency of simulation scenario construction. Through the environment parameter randomization method in the simulation and configuration module, diverse test scenarios can be automatically generated, covering different weather conditions, lighting, traffic density, and other conditions, enhancing the diversity of training data. Through the course learning strategy in the scheduling and learning module, the task difficulty is dynamically adjusted according to the intelligent agent's learning progress, avoiding premature convergence and inefficiency during training, thus improving training efficiency and the performance of the policy model.

[0045] Large-scale parallel simulations are executed on the computing cluster via a distributed simulation execution module, significantly improving data acquisition efficiency. Domain randomization technology in the simulation migration module incorporates realistic sensor noise and physical parameter randomization into the simulation environment, enhancing the robustness of the policy model to environmental changes. A shadow pattern verification method is used for incremental verification on real vehicles, ensuring the safety and reliability of the policy model before actual deployment. Online learning and a progressive decentralization strategy enable a smooth migration from simulation to the real environment, reducing deployment risks and costs and improving the performance of the policy model in real-world scenarios. Attached Figure Description

[0046] Figure 1 This is a block diagram of an autonomous driving digital twin intelligent agent system based on reinforcement learning training according to the present invention;

[0047] Figure 2 This is a flowchart of a digital twin intelligent agent system module for autonomous driving based on reinforcement learning training according to the present invention. Figure 1 ;

[0048] Figure 3 This is a flowchart of a digital twin intelligent agent system module for autonomous driving based on reinforcement learning training according to the present invention. Figure 2 . Detailed Implementation

[0049] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0050] At least one embodiment of the present invention discloses an autonomous driving digital twin intelligent agent system based on reinforcement learning training, such as... Figures 1 to 3 As shown, it includes:

[0051] The intelligent entity library construction module is used to construct a modular digital twin intelligent entity library, resulting in a modular digital twin intelligent entity library;

[0052] Specifically, the following steps are included:

[0053] S11, Intelligent Agent Requirements Analysis and Classification: Based on the application requirements of urban traffic scenarios, intelligent agents are classified into four categories using a traffic participant classification method: autonomous vehicles (training subjects), human-driven vehicles (environmental intelligent agents), pedestrians (environmental intelligent agents), and two-wheeled vehicles including bicycles and motorcycles (environmental intelligent agents), resulting in a set of intelligent agent types. Based on the functional positioning of each type of intelligent agent, a requirement extraction method is used to determine the perception capabilities (such as visual sensors, LiDAR), motion capabilities (such as maximum speed, acceleration), and decision-making capabilities (such as rule systems or neural networks) required by each type of intelligent agent, resulting in a functional requirement specification for each type of intelligent agent. Based on the functional requirement specification, a layered decoupling design method is used to decompose each intelligent agent into three independent layers: a data layer, a model layer, and an interaction layer, resulting in an architectural blueprint for the intelligent agent.

[0054] S12, for the data layer, based on the physical characteristics of the agent, a parametric modeling method is used to define a morphological data set, which includes the agent's length, width, height, mass, and moment of inertia matrix; a dynamic attribute extraction method is used to define a state data set, which includes the agent's three-dimensional position coordinates, velocity vector, acceleration vector, orientation angle, and angular velocity; a task objective definition method is used to construct a target data set, which includes the target position, desired velocity, and reward function, thus obtaining a complete data layer description;

[0055] S13, for the model layer, based on the motion characteristics of the agents, a physics engine selection algorithm is used to configure a vehicle dynamics model based on Ackermann steering geometry for vehicle-type agents and a mass dynamics model for pedestrians, thus obtaining the physical model; based on sensor configuration requirements, a sensor simulation module is used to configure a forward-looking camera, 360-degree LiDAR, and millimeter-wave radar for autonomous vehicles, thus obtaining the perception model. The perception model can generate realistic sensor data using a ray tracing algorithm based on the position and orientation of the agent in the virtual environment; based on the role localization of the agents, a decision model configuration method is used to configure a trainable deep neural network policy model for the training subject, i.e., autonomous vehicles, and a rule-based decision system based on behavior trees for the environmental agents, i.e., human-driven vehicles and pedestrians, thus obtaining the decision model;

[0056] S14, for the interaction layer, based on the communication requirements between agents, a standardized interface design method is adopted. A state query interface is defined to obtain the current state of the agent, an action execution interface is defined to receive and execute control commands, and an environment perception interface is defined to obtain sensor observation data. This results in a complete interaction layer definition, including the state query interface, action execution interface, and environment perception interface.

[0057] S15, based on the definitions of the data layer, model layer, and interaction layer, uses data encapsulation technology to package these layers into a single agent template. Based on the agent template, a parametric design method is used to convert the fixed values ​​in the template into configurable parameters. For example, the length of a vehicle is set to be between 4.0 and 5.5 meters, and its mass to be between 1200 and 2000 kilograms, resulting in a parametric agent template. Based on four agent types, a template instantiation method is used to create several specifically configured agent templates for each type (e.g., different parameter configurations for small cars, SUVs, and large trucks), resulting in an agent template set. Based on the template set, a database storage technology is used to establish an agent library management system, supporting template CRUD operations and version management, resulting in a complete modular digital twin agent library. The output is a reusable modular digital twin agent library containing various parametric agent templates, each with a complete three-layer architecture definition.

[0058] The simulation and configuration module is used to receive the modular digital twin intelligent agent library, combine and generate urban traffic simulation scenarios, configure environmental parameters, and obtain executable simulation scenarios.

[0059] S21, Static Environment Modeling of the Scene: Based on the needs of urban traffic scenes, high-precision map data is used as input. The high-precision map data includes information such as road centerlines, lane boundaries, traffic signs, and traffic lights to obtain the original map data. Based on the original map data, a 3D scene reconstruction algorithm is used to expand the 2D map information into a 3D spatial model, adding textures to the road surface and generating 3D models for buildings to obtain the static 3D model of the scene. Based on the static 3D model, a spatial indexing algorithm (octree) is used to divide the scene space into hierarchical grid units. Each unit records the information of static objects such as roads and obstacles within it, resulting in a scene data structure that supports fast spatial queries.

[0060] S22, Agent Instantiation and Initial State Allocation: Based on a modular digital twin agent library and user-defined scenario configuration parameters, including the number of agents of each type N1, N2, N3, N4, an agent instantiation method is adopted to select the corresponding type of template from the agent library and create an instance. A unique identifier ID is assigned to each instance to obtain a set of agent instances.

[0061] Based on the set of agent instances and the road network information of the scene, a spatial distribution algorithm is used to assign an initial position and initial orientation to each agent, ensuring that the agent is located in a valid road area and does not experience an initial collision, resulting in an initial state configuration set. The initial speed is set according to the agent type and the speed limit of the lane it is in. Based on the target data definition of the agents, a task allocation algorithm is used to generate task objectives for each agent, such as navigation from the starting point to the destination, resulting in a task objective set.

[0062] S23, Environmental Parameter Configuration and Diverse Scene Generation: Based on diverse scene requirements, an environmental parameter randomization method is used to set weather condition parameters (sunny, cloudy, light rain, heavy rain, foggy), lighting condition parameters (daytime, dusk, night), and road surface condition parameters (dry, wet, waterlogged), resulting in the environmental parameter configuration. Based on the environmental parameter configuration, a physical parameter modulation method is used to adjust the friction coefficient in the scene: 0.8 for dry roads and 0.5 for wet roads; visibility distance: 100 meters for sunny days and 20 meters for foggy days; sensor noise level: image noise variance is increased for rainy days, resulting in the modulated scene physical characteristics. Based on the environmental and time parameters, a rendering engine configuration method is used to set visual effects such as light intensity, shadow effects, and rain / snow particle systems, resulting in the scene rendering configuration.

[0063] S24, Simulation Scene Integration and Initialization: Based on the scene static model, spatial index structure, agent instance set, initial state configuration, task objective, physical characteristics, and rendering configuration, a scene integration algorithm is used to integrate all the above elements into a unified simulation environment. This establishes the association between agents and scene elements, initializes the internal states of the physics engine and rendering engine, and yields a complete executable simulation scene. Based on the executable simulation scene, a scene verification method is used to check the legality of the agent's initial position, the consistency of physical parameters, and the correctness of sensor configuration, ensuring the scene can operate normally and yielding a verified simulation scene. The verified executable simulation scene is output, including a complete static environment model, multiple heterogeneous agent instances and their initial states, task objective configuration, and environmental parameter settings.

[0064] The distributed simulation execution module is used to receive executable simulation scenarios, execute large-scale parallel simulations on a distributed computing cluster, and obtain optimized training datasets and performance statistics.

[0065] Specifically, the following steps are included:

[0066] S31, Simulation Scene Replication and Task Allocation: Based on the simulation scene and parallel training configuration, with a target number of parallel instances K, a deep copy algorithm is used to create K independent scene replicas. Each replica has its own independent memory space and state, and they do not interfere with each other. Based on the available computing resource topology, the number of CPU cores, the number of GPUs, and the network bandwidth between nodes, a load balancing scheduling algorithm is used to allocate the K scene replicas to M computing nodes. Each node ni is allocated Ki scene replicas, ensuring that the sum of the number of replicas allocated to all nodes equals K. The optimization objective is to minimize the variance of the computing load on each node, resulting in a task allocation scheme. This scheme includes a list of indexes for each node and its allocated scene replicas. Based on the task allocation scheme, process / thread creation technology is used to start the simulation process on each computing node, completing the deployment of scene replicas and obtaining a set of running simulation instances.

[0067] S32, single-step simulation loop execution; for each simulation instance, at time step t, the following simulation loop is executed: based on the agent's current state and scene space index, a neighborhood query algorithm is used to find other agents and environmental objects within the agent's perception range to obtain a neighborhood object set; based on the neighborhood object set and the agent's perception model, a sensor simulation method is used to generate the agent's observation data; for visual sensors, ray tracing rendering is used to generate RGB images; for lidar, a ray projection algorithm is used to calculate the distance of the reflection point of each laser beam to generate point cloud data;

[0068] Based on observation data and the agent's decision-making model, a policy reasoning method is used to calculate the agent's action commands. For trainable neural network policies, in autonomous vehicles, the observation data is input into the policy network, and after forward propagation, action outputs such as steering wheel angle and acceleration are obtained. For rule-based system policies, in environmental agents, the corresponding decision rules are executed according to behavior tree logic and current observations to obtain action commands.

[0069] Based on action commands, the current state, and the agent's physical model, numerical integration methods, such as the fourth-order Runge-Kutta method, are used to solve the agent's motion equations and calculate its position, velocity, and orientation at the next moment. For a vehicle agent, the motion equations are described as follows: the rate of change of position x equals the velocity multiplied by the cosine of the orientation angle; the rate of change of position y equals the velocity multiplied by the sine of the orientation angle; the rate of change of velocity equals the acceleration control input minus the air resistance and rolling resistance terms, where the air resistance term is calculated as air density multiplied by the drag coefficient multiplied by the frontal area multiplied by half the square of the velocity divided by the mass, and the rolling resistance term is the rolling resistance coefficient multiplied by gravitational acceleration; the rate of change of the orientation angle equals the velocity divided by the vehicle's wheelbase multiplied by the tangent of the front wheel steering angle. Based on the numerical integration results, a state update method is used to obtain the agent's predicted state at time t+1.

[0070] Based on the predicted states and spatial indices of all agents, collision detection algorithms, such as the Separating Axis Theorem (SAT) or the GJK algorithm, are used to determine whether collisions occur between agents and between agents and the environment, resulting in a collision pair set that includes all agent pairs that have collided. Based on this collision pair set, a collision response algorithm, such as the impulse method, is used to calculate the changes in velocity and angular velocity caused by the collision, correcting the agent's state to obtain the final state after considering the collision.

[0071] Based on the agent's final state, task objective, and reward function, a reward calculation method is used to evaluate the agent's performance at the current time step and calculate the immediate reward. The reward function is designed as a multi-objective weighted sum, where the immediate reward equals the weight of the forward reward multiplied by the forward reward, plus the weight of the speed reward multiplied by the speed reward, plus the weight of the collision penalty multiplied by the collision penalty, plus the weight of the lane-keeping reward multiplied by the lane-keeping reward. Here, the immediate reward represents the immediate reward for agent j at time t; the forward reward represents the reward for moving towards the target, calculated as the reduction in distance between the agent and the target position at this time step; the speed reward encourages the agent to maintain a speed close to the desired speed; the collision penalty provides a larger negative reward when a collision occurs; and the lane-keeping reward is calculated based on the agent's deviation from the lane centerline, with learnable weight coefficients assigned to each reward item.

[0072] Based on the updated state and reward, a data recording method is used to store the quadruple of the current time step, namely observation, action, reward and next observation, into the experience buffer to obtain training data samples.

[0073] S33, Simulation Time Advancement and Acceleration Control; Based on the simulation time step, typically set to 0.05 seconds and the current simulation time t, a time advancement method is used to update the simulation clock to t plus the time step; Based on the actual computation time and simulation time step, a time speedup ratio calculation method is used to obtain the speedup ratio, which is equal to the single-step simulation time step divided by the actual computation time of the current simulation step; When the speedup ratio is greater than 1, it indicates that the simulation speed is faster than real-time; When the speedup ratio is less than 1, it indicates that the simulation cannot run in real-time; Based on the computing resource utilization and speedup ratio target, a dynamic adjustment strategy is adopted to optimize simulation parameters, such as reducing rendering quality and reducing the number of agents to maintain the target speedup ratio.

[0074] Once the simulation duration reaches the preset training segment length, such as 1000 time steps or the agent completes the task objective, a segment termination judgment method is used to end the current simulation segment, reset the scene to the initial state or generate a new random initial configuration, and start the simulation of the next segment.

[0075] S34, Training Data Aggregation and Transmission: Based on the training data samples recorded in each simulation instance, a data aggregation method is used to aggregate the data distributed across different computing nodes to a central data storage system via network, resulting in a global training dataset. This dataset includes all state-action-reward-next state quadruplets. Based on the training dataset, data compression and format conversion techniques are used to compress and store the original high-dimensional sensor data, such as images, and to unify the data format to the format required by the training module, resulting in an optimized training dataset.

[0076] S35, Training Data Preprocessing and Feature Normalization: Based on the optimized training dataset and environmental parameter configuration, feature normalization methods are used to preprocess observation, action, and reward-related variables to obtain standardized data suitable for training. For continuous numerical features, including position, velocity, acceleration, orientation radian angle, lane departure, etc., standardization is applied. The standardized feature equals the original feature value minus the mean of the feature divided by the standard deviation of the feature, plus a numerical stability term, such as 10⁻⁸. When features have clear physical upper and lower bounds, normalization is used. The normalized feature equals the original feature value minus the minimum value divided by the maximum value minus the minimum value, plus a numerical stability term. The minimum and maximum values ​​represent the minimum and maximum possible values ​​determined based on the scene and sensor range. For angle-type features, they are uniformly converted to radians and mapped to the -π to π interval before standardization.

[0077] For categorical variables, such as weather, sunlight, and road surface, one-hot encoding is used. That is, the one-hot encoding function maps the categorical variable to a basis vector, where the categorical variable belongs to a category set, such as sunny, cloudy, light rain, heavy rain, and foggy.

[0078] To avoid weighting imbalances caused by inconsistent dimensions, the components of the reward function are normalized and replaced with their normalized forms during calculation: the scaled forward reward equals the reduction in distance relative to the target at that time step divided by a reference distance scale, such as the maximum reasonable forward distance per step; the scaled speed reward equals 1 minus the absolute difference between the current speed and the expected speed divided by the upper speed limit; the scaled collision penalty equals a negative indicator function, 1 if a collision occurs, and 0 otherwise; the scaled lane-keeping reward equals 1 minus the absolute value of the lane centerline deviation divided by the maximum allowable deviation distance. These scaled components replace the original components in the reward calculation to ensure that the weighting coefficients of each reward apply to dimensionless quantities.

[0079] For action commands, a range scaling to -1 to 1 is used. That is, the scaled action is equal to 2 multiplied by (original action minus minimum value divided by maximum value minus minimum value plus a numerical stability term) minus 1, where the minimum and maximum values ​​represent the physical upper and lower bounds of the action, such as the range of acceleration and rotation angle. The scaled action is used as the network input for training, and its inverse transformation is used to restore it to physical dimensions during environment execution.

[0080] The observation vector is pruned to eliminate the influence of outliers; that is, the pruning function restricts x to a lower bound of 'a' and an upper bound of 'b', where 'a' and 'b' are upper and lower bounds determined based on the physical and sensor ranges. Combining the above processing, a preprocessed training dataset is obtained. The optimized training dataset and the preprocessed training dataset are output, containing a large number of state-action-reward-next-state quadruples from the parallel simulation, as well as performance statistics of the simulation execution, such as speedup and throughput.

[0081] The scheduling and learning module is used to receive the optimized training dataset and performance statistics, implement dynamic task scheduling and course learning according to the learning progress of the agent, and obtain the updated simulation scenario and training monitoring results.

[0082] Specifically, the following steps are included:

[0083] S41, based on the output training dataset, a performance metric extraction method is used to calculate the key performance indicators for each agent or group of agents in the most recent H training segments. Specifically, the task success rate is calculated, defined as the proportion of segments that successfully reach the target position out of the total number of segments; the average cumulative reward is calculated, which is equal to the sum of the instantaneous rewards at all time steps in all segments divided by H, where H represents the number of training segments within the statistical window; the collision rate is calculated, defined as the proportion of segments that collide out of the total number of segments; and the average completion time is calculated, representing the average number of time steps for successful segments. A performance metric vector is obtained, which includes the task success rate, average cumulative reward, collision rate, and average completion time. Based on the performance metric vector, statistical analysis methods are used to calculate the confidence interval and variance of each metric, evaluate the stability of agent performance, and obtain the performance stability evaluation results.

[0084] S42, Course Learning Strategy Definition and Difficulty Advancement Judgment: Based on course learning theory, a hierarchical task difficulty design method is adopted, predefining multiple difficulty levels. Each difficulty level corresponds to different scenario configuration parameters, including traffic density, environmental complexity, and the probability of extreme events. Specifically, the beginner difficulty sets a low traffic density (20 vehicles per square kilometer), good weather conditions (sunny days), and fewer interactive opponents (only vehicles in a rule-based driving environment); the intermediate difficulty increases the traffic density to 50 vehicles per square kilometer, introduces uncertain weather (30% probability of light rain and 30% probability of cloudy days) and more diverse environmental agent behaviors (including aggressive driving and conservative driving); the advanced difficulty further increases the traffic density to 100 vehicles per square kilometer, introduces extreme weather (heavy rain, foggy days) and sudden events (pedestrians suddenly crossing the road, vehicles in front braking suddenly). This yields a set of difficulty level configurations.

[0085] Based on the agent's current difficulty level and performance metrics, a promotion condition judgment method is used to check whether the conditions for promotion to the next difficulty level are met. The promotion conditions are set as follows: task success rate greater than or equal to a success rate threshold (default value 0.85), collision rate less than or equal to a collision rate threshold (default value 0.1), and performance stability less than or equal to a stability threshold. Based on the judgment results of the promotion conditions, a decision logic is used: if the promotion conditions are met, the new difficulty level is set to the current difficulty plus 1; if the promotion conditions are not met, the current difficulty is maintained; if performance decreases, i.e., the task success rate is less than 0.5, the difficulty is reduced to the maximum value between the current difficulty minus 1 and the lowest difficulty, resulting in the updated difficulty level.

[0086] S43, Dynamic Adjustment of Scene Parameters: Based on the updated difficulty level and corresponding configuration parameters, a scene parameter mapping method is used to generate new scene configuration instructions. Specifically, based on traffic density parameters, an agent number adjustment algorithm is used to calculate the target number of each type of agent; based on environmental complexity parameters, an environmental parameter setting method is used to update the probability distribution of weather conditions, lighting conditions, and road surface conditions; based on extreme event probability parameters, an event generator configuration method is used to set the trigger probability and type of sudden events. This results in new scene configuration instructions.

[0087] Based on the new scenario configuration instructions and scenario generation method of the simulation and configuration module, scenario reconfiguration technology is adopted to dynamically update the parameters of the simulation scenario. For running simulation instances, a hot update mechanism is used to gradually adjust the number of agents and environmental parameters without interrupting the simulation; for newly started simulation instances, the new configuration is applied directly. This results in an updated simulation scenario.

[0088] S44, Learning Progress Recording and Visual Monitoring: Based on the agent's performance metrics, current difficulty level, and number of training iterations, a log recording method is used to store training progress information in time-series format, resulting in a training log. Based on the training log, data visualization technology is used to generate performance metric change curves over training time, difficulty level advancement timelines, and performance comparison charts under different scenario configurations, resulting in a visual monitoring panel for trainers to monitor the training status in real time. The updated simulation scenario is output and fed back to the distributed simulation execution module for continued simulation, thus recording the agent's learning progress and providing a visual monitoring panel.

[0089] The policy training module is used to receive updated simulation scenarios and training monitoring results, train reinforcement learning policies and value networks based on simulation data, and obtain the trained policy model and performance curve.

[0090] Specifically, the following steps are included:

[0091] S51, Policy Network Architecture Design and Initialization: Based on the observation space of autonomous driving tasks, multimodal sensor data and action space, and continuous control commands, a neural network architecture design method is adopted to construct the policy network and value network.

[0092] For the policy network, based on the input observation data, including LiDAR point clouds and vehicle state information, a multi-branch network structure is adopted: the LiDAR branch uses the PointNet++ network to extract point cloud features, with the input being N x 3 point cloud coordinates, where N is the number of points. After passing through multiple SetAbstraction layers, the point cloud feature vector is obtained; the vehicle state branch uses a fully connected network, with the input being scalar states such as vehicle speed, acceleration, and heading angle. After passing through two fully connected layers, the state feature vector is obtained; based on the two feature vectors, a feature fusion method is used, and the concatenation is passed through a fully connected layer to obtain the fused features; based on the fused features, an Actor network structure is adopted, passing through two fully connected layers, and the mean and standard deviation of the actions are output. The mean and standard deviation are mainly for the two continuous actions of steering wheel angle and acceleration;

[0093] The policy distribution represents the distribution of actions under observation, using a Gaussian distribution, where the mean and standard deviation are output by the policy network. This yields the complete policy network architecture. To ensure the physical rationality and numerical stability of the standard deviation parameter, a positive constraint is imposed on the standard deviation, and a diagonal covariance assumption is adopted, with each action dimension being independent. Specifically, the original network output is taken as the original parameters, and the standard deviation uses a Softplus mapping and a numerical stability term, where the Softplus function is ln(1+ex).

[0094] To achieve low-variance differentiable sampling, a reparameterization technique is employed to sample actions. Specifically, an action is equal to the mean plus the standard deviation, multiplied element-wise by standard normal noise, where the noise follows a standard normal distribution. Before execution in the environment, the sampled actions are restored to their physical dimensions and legal intervals using an inverse transformation defined by S35. For the value network, based on the same observed input and feature extraction branch, a Critic network structure is adopted. After feature fusion, the network passes through two fully connected layers to output a state value estimate, resulting in the value network architecture. Based on this architecture, the Xavier initialization method is used to randomly initialize the policy network parameters and value network parameters, resulting in an initialized network model.

[0095] S52, Experience Replay and Sample Acquisition: Based on the training dataset, an experience replay buffer management method is used to store the training data in a circular buffer with a capacity of C (e.g., C=106). When the buffer is full, a first-in-first-out strategy is used to overwrite the oldest data. In each training iteration, based on the experience replay buffer, a small batch of training samples, such as 256, is extracted using a random sampling method to obtain a training batch. This batch includes observations, actions, rewards, and the next observation. Based on random sampling, data augmentation techniques are used, such as randomly rotating and translating the point cloud, adding small noise to the state information, enhancing the diversity of the training data, reducing the risk of overfitting, and obtaining an augmented training batch.

[0096] S53, Policy Gradient Calculation and Network Parameter Update: Based on the training batch and the current policy and value networks, the Proximal Policy Optimization (PPO) algorithm is used for parameter update. Based on the value network and reward data, the advantage function estimation method is used to calculate the advantage function for each sample. Specifically, for the reward of the time step sequence, the Generalized Advantage Estimation (GAE) method is used to calculate the advantage function, i.e., the advantage estimate is equal to the weighted sum of the temporal difference errors of the offset indices from the current time step forward, where the weight is the discount factor multiplied by the l-th power of the GAE parameter, l represents the offset index from the current time step forward, and the temporal difference error is equal to the immediate reward plus the discount factor multiplied by the value estimate of the next state minus the value estimate of the current state. In the implementation, the above infinite sum is truncated by the sequence length T and end-guided. The advantage function estimate is obtained.

[0097] Based on the dominance function and policy network, a loss function for policy update is constructed using the PPO objective function calculation method. Specifically, the probability ratio of the current policy to the old policy is calculated, i.e., the probability of choosing an action under the current policy divided by the corresponding probability under the old policy. Based on the probability ratio and dominance function, a pruning objective function is used to calculate the PPO loss, which is the average of the minimum values ​​in the negative batch. The minimum value is calculated by multiplying the probability ratio by the dominance and the pruned probability ratio by the dominance. The pruning function restricts the probability ratio to between 1-epsilon and 1+epsilon, with a default epsilon value of 0.2. Based on the value network and actual rewards, a mean squared error loss is used to calculate the loss function of the value network, i.e., the average of the squared differences between the value estimate and the cumulative discounted reward, where the cumulative discounted reward is the sum of the discounted rewards from t=0 to t-1. The loss function value of the value network is obtained. Based on the loss function, the backpropagation algorithm is used to calculate the gradient of the loss function with respect to the network parameters. Based on the gradient information, the Adam optimizer is used to update the network parameters, i.e., the new parameters are equal to the old parameters minus the learning rate multiplied by the Adam-adjusted gradient. The updated policy network parameters and value network parameters are obtained.

[0098] S54, Training Convergence Judgment and Model Saving: Based on the number of training iterations k and performance metrics, average cumulative reward, a convergence judgment method is used to check whether the training has reached the termination condition. Specifically, the relative rate of change of the average reward within the most recent W iteration windows is calculated, i.e., the absolute difference between the current average reward and the average reward before W iterations, divided by the former. Based on the rate of change and the convergence threshold, a judgment logic is used: if the rate of change is less than the threshold and k is greater than the minimum number of iterations, the training is determined to be converged; otherwise, training continues. The convergence judgment result is obtained.

[0099] Based on the convergence judgment result and the current iteration count, a model checkpoint saving strategy is adopted. Periodically (every 100 iterations) or when a new optimal value is reached, the policy network parameters, value network parameters, and optimizer state are saved to disk, resulting in a model checkpoint file. Based on the performance comparison of all checkpoints, an optimal model selection method is used to select the checkpoint with the best validation performance as the final training model, resulting in the trained policy model. The trained policy model is output, including the parameters of the policy network and value network, as well as the performance curves and logs of the training process.

[0100] In some embodiments, a multi-agent reinforcement learning algorithm is used to replace the single-agent PPO algorithm. In some complex scenarios, multiple autonomous vehicles need to make collaborative decisions to optimize global traffic efficiency. In this case, the policy based on single-agent reinforcement learning cannot effectively capture the cooperation and competition relationships between agents. Based on this requirement, the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm is used to replace the standard PPO algorithm. The core innovation of MAPPO lies in the introduction of a centralized value network and a decentralized policy network architecture.

[0101] Specifically, based on observation data from multiple agents, an information aggregation method is used to construct a global state representation. This global state includes the position, velocity, and target information of all trained agents, as well as global environmental features such as traffic flow density and road network topology. Based on the global state, a centralized Critic network is used to estimate the global state value. This network can access information from all agents during training, resulting in a more accurate value estimate.

[0102] For each agent, a decentralized Actor network is used to independently compute actions based on its local observations, ensuring that each agent can make autonomous decisions based on local information during deployment. Based on the value estimated by the centralized Critic and individual rewards, the GAE method is used to compute the advantage function for each agent. Based on the advantage function, the PPO (Plan-Based Objective) is used to prune the objective, updating the policy network parameters for each agent and the shared Critic network parameters respectively.

[0103] To enhance collaboration among agents, an information exchange module is constructed based on an attention mechanism, designed around a communication mechanism. Specifically, each agent, based on its feature vector, employs a multi-head self-attention layer to calculate attention weights for features from other agents. This weight is calculated by dividing the dot product of the query vector and the key vector by the square root of the key vector's dimension, and then normalizing the result using a softmax function.

[0104] Based on attention weights and the value vectors of other agents, a weighted summation method is used to obtain the communication messages of the agents. Based on the communication messages and the characteristics of the agents themselves, a feature fusion method is used to obtain enhanced feature representations, which are then input into the policy network for decision-making, thereby realizing implicit cooperation among agents.

[0105] In one embodiment of the present invention, the purpose of using the MAPPO algorithm is to obtain a more stable value estimate through centralized training, thereby improving training efficiency and the performance of the final policy; to ensure scalability and robustness during deployment through decentralized execution; and to improve traffic efficiency and safety in multi-vehicle cooperative scenarios by realizing effective communication and cooperation among agents through the attention mechanism.

[0106] The simulation migration module is used to verify the strategy model and implement the migration deployment from simulation to reality, and obtain the final deployment strategy model and evaluation report.

[0107] Specifically, the following steps are included:

[0108] S61. Construct a diverse set of test scenarios. Based on testing requirements, cover common scenarios and extreme long-tail scenarios, and use scenario enumeration and combination methods to generate diverse test scenario sets. Specifically, based on scenario dimensions such as weather, lighting, traffic density, and types of emergencies, use orthogonal experimental design to select representative parameter combinations to generate a basic test scenario set. Based on extreme scenario requirements, use boundary condition generation methods to construct long-tail test scenarios, such as dense fog + high traffic density + continuous emergencies, to generate an extreme test scenario set. Based on adversarial example generation theory, use an intelligent adversarial scenario generation algorithm, utilize a trained policy model, and search for scenario configurations that can cause it to fail, such as specific initial distribution and behavior patterns of agents, to generate an adversarial test scenario set. Merge the above scenario sets to obtain a complete test scenario set, which is the union of the basic test scenario set, the extreme test scenario set, and the adversarial test scenario set.

[0109] S62, Simulation Environment Performance Evaluation: Based on the test scenario set and the policy model output by the policy training module, a batch test execution method is used to run all test scenarios in a digital twin environment, recording the complete behavioral trajectory and performance data of the agent. For each test scenario, 10 repeated experiments are run using different random seeds, and indicators such as task success rate, average cumulative reward, number of collisions, average speed, and trajectory smoothness are calculated to obtain single-scenario performance statistics.

[0110] Based on performance statistics across all scenarios, a performance aggregation method is used to calculate overall performance metrics: overall task success rate equals the average task success rate across all test scenarios, which is the inverse of the test scenario set size multiplied by the sum of the task success rates across all test scenarios; overall collision rate; average reward, resulting in a simulation environment performance report; based on the performance report, a bottleneck analysis method is used to identify which scenario types the strategy model performs poorly, such as a success rate below 50% in foggy scenarios, resulting in a performance bottleneck list.

[0111] S63, Domain Randomization and Robustness Enhancement: Based on the differences between the real and simulated environments, domain randomization techniques are used to enhance the robustness of the strategy. Specifically, based on the noise characteristics of real sensors (obtained through statistical analysis of real data), a noise model fitting method is used to incorporate more realistic noise into the sensor simulation of the simulated environment. For LiDAR, noise models such as random point loss, distance measurement error, and reflection intensity fluctuation are used; for vehicle state estimation, noise models such as GPS positioning error and IMU drift are used to obtain a noise-enhanced sensor simulation configuration.

[0112] Due to the uncertainties in physical parameters, such as the variation range of actual vehicle mass and tire friction coefficient, a parameter randomization method is adopted to randomly sample physical parameter values ​​during training and testing. Specifically, for vehicle mass, samples are uniformly taken within ±20% of the nominal value; for road friction coefficient, samples are taken within the range of 0.3 to 0.9 depending on road conditions; and for actuator delay, samples are taken within the range of 0.05 to 0.15 seconds. This results in a randomized configuration of physical parameters.

[0113] Based on enhanced sensors and physical configurations, additional training methods are employed, incorporating domain-randomized samples into the policy training module's training process to enable the policy model to learn robustness to environmental changes. Alternatively, a fine-tuning method is used, based on the already trained model, to perform additional training iterations in a domain-randomized environment to obtain a robustly enhanced policy model.

[0114] S64, Shadow Mode Validation and Online Evaluation: Based on a robust enhanced policy model and a real autonomous vehicle hardware platform, a shadow mode deployment method is used for progressive validation. Specifically, during the operation of the real vehicle, whether driven by a human safety operator or controlled by the existing control system, the policy model synchronously receives data from real sensors. Based on real-time observation, policy network inference is used to calculate the action output by the model, but this action is not actually executed, only recorded.

[0115] Simultaneously, the actual actions of the real system, the operations of the human driver, or the output of the original system are recorded. Based on the model output and the actual actions, a consistency evaluation method is used to calculate the difference between the two, such as the Euclidean distance of the actions and the degree of deviation of the trajectory. For data in continuous time periods, a statistical analysis method is used to calculate the consistency ratio, which is the proportion of time when the model output and the actual actions are within the acceptable error range, to obtain a shadow mode performance evaluation report. Based on the shadow mode evaluation report, a safety judgment method is used. If the consistency ratio is higher than a threshold, such as 90%, and the model output is reasonable in key safety scenarios, the strategy model is determined to enter the real vehicle testing stage. Otherwise, the scenarios where the model output is abnormal are identified, and the model is returned to the strategy training module for targeted retraining or adjustment.

[0116] S65, real-vehicle deployment and online fine-tuning; based on the strategy model verified through shadow mode, model deployment technology is used to load it onto the on-board computing platform of a real vehicle, such as NVIDIA DriveAGX; based on the model inference performance requirements, the real-time control cycle is required to be 0.05 seconds, and model optimization technology, such as TensorRT acceleration and model quantization, is used to optimize the model inference speed to ensure that the real-time requirements are met, resulting in an optimized deployment model;

[0117] In the initial stage of real-vehicle testing, a human-in-the-loop control mode was adopted for safety reasons, allowing a human safety operator to take over vehicle control at any time. Based on real data generated during vehicle operation, including real sensor observations, strategy model outputs, actual vehicle responses, and safety operator intervention signals, an online learning method was used to fine-tune the strategy model. Specifically, the safety operator's intervention actions were treated as expert demonstrations, and behavioral cloning or DAgger algorithms were used in conjunction with reinforcement learning to further optimize the strategy. Based on the fine-tuned model, a gradual decentralization strategy was adopted to gradually increase the proportion of time the model was autonomously controlled, ultimately achieving fully autonomous driving and obtaining the final deployed strategy model.

[0118] S66, Real-world performance comparison and migration effect evaluation; Based on real-world test data and simulation performance data from step S62, a comparative analysis method is used to evaluate the effect of the sim-to-real migration. Specifically, the differences in the same indicators between the simulation environment and the real environment are calculated, such as the success rate difference, i.e., the absolute difference between the simulation success rate and the real success rate, to obtain the migration performance gap. Based on the performance gap analysis, a root cause analysis method is used to identify the main factors leading to performance degradation, such as insufficient simulation accuracy of specific sensors and deviations in the physical model, to obtain a list of improvement suggestions, providing guidance for the iterative optimization of the system. Output: A fully validated and robustly enhanced strategy model that can be deployed to real autonomous vehicles; Simulation environment report, shadow mode evaluation report, real-world performance data, and sim-to-real migration effect evaluation report.

[0119] In some embodiments, model distillation techniques are used to compress the strategy network to meet the computational resource constraints of the vehicle platform;

[0120] In practical deployments, the computing power and memory resources of in-vehicle computing platforms are limited, and the complete policy network (teacher network) trained by the policy training module may be too large to meet real-time requirements. To address this need, knowledge distillation technology is employed to transfer knowledge from a large teacher network to a smaller student network. Specifically, based on the teacher network architecture, such as the PointNet++ backbone, a network simplification method is used to design the student network. This network uses a lighter backbone, such as PointNet or a simplified MLP, reducing the number of parameters to 1 / 5 of the teacher network. Based on knowledge distillation theory, a soft-label distillation method is used to train the student network. Specifically, for each observation in the training dataset, the teacher network outputs the action distribution, i.e., the mean and variance of a Gaussian distribution, while the student network outputs the action distribution. Based on these two distributions, KL divergence is used as the distillation loss, i.e., the expectation of the KL divergence under the observed data distribution, which is the KL divergence between the teacher policy and student policy action distributions.

[0121] Simultaneously, based on the real action labels of the training data, behavioral cloning loss is adopted to ensure that the student network can imitate the expert's actions, i.e., the squared difference between the real action and the mean of the student network's output action under the expectation of the observation and action pair; based on distillation loss and behavioral cloning loss, a weighted combination is used to construct the total loss function of the student network, i.e., the weighted sum of behavioral cloning loss and distillation loss, where the default value of the weight coefficient is usually 0.5 to 1.0; based on the total loss function, the gradient descent method is used to optimize the parameters of the student network to obtain the distilled lightweight policy model;

[0122] Based on the distilled model, a performance comparison test was conducted to compare the performance differences between the student network and the teacher network in both simulation and real environments.

[0123] The purpose of using model distillation technology is to reduce the computational complexity and memory consumption of the model while maintaining policy performance, so that it can be deployed on resource-constrained in-vehicle platforms; and to enable the lightweight model to inherit the decision-making ability of the teacher model through knowledge transfer, thereby avoiding the performance loss caused by training a small model from scratch.

[0124] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. An autonomous driving digital twin intelligent agent system based on reinforcement learning training, characterized in that, include: The intelligent entity library construction module is used to construct a modular digital twin intelligent entity library, resulting in a modular digital twin intelligent entity library; The simulation and configuration module is used to receive the modular digital twin intelligent agent library, combine and generate urban traffic simulation scenarios, configure environmental parameters, and obtain executable simulation scenarios. The distributed simulation execution module is used to receive executable simulation scenarios, execute large-scale parallel simulations on a distributed computing cluster, and obtain optimized training datasets and performance statistics. The scheduling and learning module is used to receive the optimized training dataset and performance statistics, implement dynamic task scheduling and course learning according to the learning progress of the agent, and obtain the updated simulation scenario and training monitoring results. The policy training module is used to receive updated simulation scenarios and training monitoring results, train reinforcement learning policies and value networks based on simulation data, and obtain the trained policy model and performance curve. The simulation migration module is used to verify the strategy model and implement the migration deployment from simulation to reality, and obtain the final deployment strategy model and evaluation report.

2. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The intelligent entity library construction module includes: A layered decoupling design approach is adopted to decompose the agent into a data layer, a model layer, and an interaction layer; the data layer uses a parametric modeling method to define the morphological data set and a dynamic attribute extraction method to define the state data set. The model layer configures a vehicle dynamics model based on Ackermann steering geometry for vehicle-type intelligent agents, a mass dynamics model for pedestrians to obtain a physical model, and a forward-looking camera, a 360-degree LiDAR and a millimeter-wave radar for autonomous vehicles to obtain a perception model. The interaction layer defines the state query interface, action execution interface, and environment awareness interface.

3. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The simulation and configuration module includes: Using high-precision map data as input, the two-dimensional map information is expanded into a three-dimensional spatial model through a three-dimensional scene reconstruction algorithm, and the scene space is divided into hierarchical grid units using a spatial index construction algorithm. Select the corresponding type of template from the modular digital twin intelligent agent library and create instances, assign a unique identifier to each instance, and use a spatial distribution algorithm to assign an initial position and initial orientation to each intelligent agent; By using an environmental parameter randomization method to set weather conditions, lighting conditions, and road surface condition parameters, and adjusting the friction coefficient, visibility distance, and sensor noise level in the scene, an executable simulation scene is obtained.

4. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The distributed simulation execution module includes: A deep copy algorithm is used to create K independent scene replicas, and a load balancing scheduling algorithm is used to distribute the K scene replicas to M computing nodes; At each time step t, a neighborhood query algorithm is used to find other agents and environmental objects within the perception range of each agent. Observation data is generated using a sensor simulation method. For trainable neural network policies, the observation data is input into the policy network to obtain action output. The motion equation of the agent is solved using a numerical integration method. A collision detection algorithm is used to determine whether a collision has occurred. The immediate reward is calculated and the quadruplet is stored in the experience buffer.

5. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The distributed simulation execution module also includes: The training data is preprocessed, and the continuous numerical features are standardized. That is, the standardized feature is equal to the original feature value minus the mean of the standardized feature divided by the standard deviation of the standardized feature plus the numerical stability term. One-hot encoding is used for categorical variables; dimensional normalization is performed on each component of the reward function, and the scaled forward reward is equal to the reduction in distance relative to the target at that time step divided by the reference distance scale, and the scaled velocity reward is equal to 1 minus the absolute difference between the current velocity and the expected velocity divided by the upper bound of the velocity; the action command is scaled to -1 to 1, and the observation vector is pruned to eliminate the influence of outliers.

6. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The scheduling and learning module includes: Calculate the task success rate, average cumulative reward, collision rate and average completion time of each agent in the most recent H training segments to obtain a performance index vector; Multiple difficulty levels are predefined, each corresponding to different scenario configuration parameters, including traffic density, environmental complexity, and probability of extreme events. An advancement condition judgment method is adopted. When the task success rate is greater than or equal to the success rate threshold and the collision rate is less than or equal to the collision rate threshold, a new difficulty level is set to the current difficulty plus 1. Based on the updated difficulty level, a new scenario configuration instruction is generated, and the parameters of the simulation scenario are dynamically updated to obtain the updated simulation scenario.

7. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The policy training module includes: A policy network and a value network are constructed. The policy network adopts a multi-branch network structure. The LiDAR branch uses PointNet++ network to extract point cloud features, and the vehicle state branch uses a fully connected network to extract state features. The two feature vectors are concatenated and then passed through a fully connected layer to obtain fused features. The mean and standard deviation of the actions are output through the Actor network structure. After fusing features, the value network outputs state value estimates through two fully connected layers. The Xavier initialization method is used to randomly initialize the network parameters, and small batches of training samples are randomly sampled from the experience replay buffer to obtain the training batch.

8. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The policy training module also includes: The PPO algorithm is used for parameter updates, and the GAE method is used to calculate the advantage function. The advantage estimate is equal to the weighted sum of the temporal difference errors of the offset index from the current time to the future. The probability ratio of the current policy to the old policy is calculated. The PPO loss is calculated using the pruning objective function. The loss function of the value network is calculated using the mean squared error loss. The gradient of the loss function with respect to the network parameters is calculated using the backpropagation algorithm. The network parameters are updated using the Adam optimizer. The training is considered to have converged when the relative rate of change of the average reward in the most recent W iteration windows is less than a threshold.

9. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The simulation migration module includes: A variety of test scenario sets are generated using scenario enumeration and combination methods, including basic test scenario sets, extreme test scenario sets, and adversarial test scenario sets; Run all test scenarios in the digital twin environment, calculate the overall task success rate, overall collision rate and average reward, and obtain a simulation environment performance report; The robustness of the strategy is enhanced by employing domain randomization techniques. More realistic noise is added to the sensor simulation in the simulation environment. Physical parameter values ​​are randomly sampled during training and testing, vehicle mass is sampled uniformly, and road friction coefficient is sampled.

10. The autonomous driving digital twin intelligent agent system based on reinforcement learning training according to claim 1, characterized in that, The simulation migration module also includes: A shadow mode deployment method is used for incremental verification. The strategy model synchronously receives data from real sensors and calculates the actions output by the model but does not actually execute them. The consistency ratio between the model output and the actual actions is calculated. When the consistency ratio is higher than the threshold, the strategy model is loaded into the on-board computing platform of the real vehicle. TensorRT is used to accelerate and model quantization to optimize the model inference speed. Online learning methods are used to fine-tune the strategy model. The safety driver's intervention actions are regarded as expert demonstrations. A gradual decentralization strategy is adopted to gradually increase the proportion of time for the model to autonomously control, so as to obtain the final deployed strategy model.

Images