A meta-reinforcement learning based simulation-to-reality transfer method for autonomous racing cars
Patent Information
- Application Number
- CN202611345293.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-01
- Publication Date
- 2026-10-30
AI Technical Summary
然而，为了覆盖尽可能多的真实环境变化，需要采用较宽的参数随机化区间，但这直接导致学习到的策略趋于保守——因为策略必须在多种甚至相互冲突的动力学条件下同时表现良好，不得不牺牲极端条件下的激进操控以换取基本的生存能力
1、本发明提供一种基于元强化学习的自动驾驶赛车仿真到现实迁移方法，构建了面向自动驾驶赛车场景的元强化学习整体框架，将元强化学习（RL²框架）首次引入自动驾驶赛车领域，设计了完整的训练与部署流程，解决了该场景下仿真策略难以向现实迁移的固有问题，能够在不依赖测试时在线微调的前提下，使自动驾驶赛车策略既能保持高速竞技性能，又能根据实际部署环境的动力学特性进行快速在线适应，从而实现稳定、高速的仿真到现实零样本迁移。
Smart Images
Figure CN123013695A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of mobile robot and artificial intelligence technology, and in particular relates to a method for transferring the simulation of autonomous driving racing cars to reality based on meta-reinforcement learning. Background Technology
[0002] Autonomous racing cars serve as crucial testing platforms for high-performance robotics technologies, requiring the simultaneous achievement of optimal planning and precise control near the limits of vehicle dynamics. Existing technologies primarily employ a two-stage pipeline approach: first, an optimal reference trajectory is planned, followed by trajectory tracking using controllers such as model predictive control. While this method can achieve reliable performance under well-calibrated model parameters, its effectiveness is highly dependent on the accuracy of the vehicle dynamics model.
[0003] Reinforcement learning, as a policy learning method that does not require explicit modeling of dynamics, has been widely explored for use in autonomous driving in recent years. However, due to considerations of safety and data efficiency, existing reinforcement learning methods are mainly trained in simulation environments. When deployed to real environments, they face the well-known "simulation-to-reality gap" problem—policy performance degrades significantly or even fails completely.
[0004] Domain randomization methods enhance policy robustness by randomizing physical parameters, observation noise, and motion noise during simulation training, and are currently the most widely adopted technique for solving simulation-to-real-world transfer problems. However, to cover as many real-world environmental variations as possible, a wide range of parameter randomization is required, which directly leads to a conservative learning strategy—because the strategy must perform well under multiple, even conflicting, dynamic conditions simultaneously, sacrificing aggressive maneuvers under extreme conditions for basic survivability. Furthermore, domain randomization strategies are essentially "static" general strategies; once deployed to a specific real-world environment, they cannot be adjusted and adapted online based on the actual dynamic response of that environment. If the dynamic characteristics of the real-world environment exceed the randomization coverage during training, the strategy performance will drastically decrease or even completely fail. Summary of the Invention
[0005] To address the aforementioned issues, this invention provides a method for migrating autonomous driving racing car simulations to reality based on meta-reinforcement learning. This method enables autonomous driving racing car strategies to maintain high-speed racing performance while rapidly adapting to the dynamic characteristics of the actual deployment environment without relying on online fine-tuning during testing.
[0006] A method for transferring autonomous driving racing simulations to real-world environments based on meta-reinforcement learning is proposed. After training on the simulation platform, the Actor-Critic network model with fixed network meta-parameters is directly deployed to the physical vehicle environment with zero-shot configuration. The Actor-Critic network model consists of a policy network and a value network with identical network structures, both obtained by cascading a TCN encoder and an MLP network. The training method on the simulation platform is as follows: S1: Sample a task from the vehicle's task distribution and obtain the vehicle dynamics parameters used by the vehicle in the sampled task; S2: Reset the internal state of the TCN encoder to the zero vector, completing the initialization of the inner loop; S3: For each of the K rounds executed in the current sampling task, execute steps S31~S33 until all rounds are completed, thus completing the adaptation process of the inner loop. S31: Reset the vehicle to the starting line position on the track, with an initial speed of 0; S32: Perform the following round-based loop operation: Based on the vehicle dynamics parameters used by the vehicle in the current sampling task, an extended input is constructed for each time t in the current round. , , , , These are the current vehicle status, the vehicle action in the previous moment, the step reward in the previous moment, and the end flag in the previous moment, respectively. The historical sequence is composed of the h extended inputs closest to the current time t. If the cumulative number of steps in the current round is less than h, then fill with zeros up to the length h; Historical sequence Inputting the TCN encoder yields temporal features. ; Time series features Input the MLP network to obtain action instructions; Execute action commands to obtain the vehicle's status, step reward, collision cost, and termination flag at the next moment; Store the interaction data at each moment of the current round into the trajectory buffer of the current sampling task; S33: Determine if the current round has ended. If the termination condition is triggered, the current round ends and step S31 is executed again in the next round. S4: Outer policy update, using the collected trajectory buffer data of the entire sampling task, execute the safety reinforcement learning algorithm PPO-Lagrangian update to update the policy network parameters and value network parameters; S5: Return to step S1, sample the next task, and re-execute steps S2~S4 until the performance of the policy network on the test task converges.
[0007] Furthermore, the vehicle status includes the forward path point, lateral deviation, longitudinal speed, yaw rate, and current steering angle.
[0008] Furthermore, the action commands include speed commands and steering angle commands.
[0009] Furthermore, each task in the vehicle's task distribution corresponds to a Markov decision process with different dynamic parameters.
[0010] Furthermore, in step S33, the internal state of the TCN encoder is retained between round boundaries and is not reset.
[0011] Furthermore, in S33, the termination condition is that the car completes one lap or a collision occurs.
[0012] Furthermore, the reward at each time t The calculation method is as follows:
[0013] in, , These represent the percentage of the vehicle's progress relative to the road centerline at time t and time t-1, respectively. , , These are the vehicle steering angles corresponding to times t, t-1, and t-2, respectively. For time penalty items, , These are the weights corresponding to progress rewards and turning rewards, respectively. and These are the reward weight and lap time for successfully completing a lap, respectively.
[0014] Furthermore, the collision cost at each time t The calculation method is as follows: .
[0015] Furthermore, in step S4, the objective of the PPO-Lagrangian security reinforcement learning algorithm is updated as follows:
[0016] st
[0017] in, To optimize the optimal strategy, γ is the discount factor for future rewards. Let represent the expectation of the trajectory under strategy π and transfer dynamics P. The step reward at time t, Let be the collision cost at time t. This is the safety threshold.
[0018] Beneficial effects: 1. This invention provides a method for transferring simulation to reality of autonomous driving racing cars based on meta-reinforcement learning. It constructs an overall framework of meta-reinforcement learning for autonomous driving racing scenarios, introduces meta-reinforcement learning (RL² framework) into the field of autonomous driving racing for the first time, designs a complete training and deployment process, and solves the inherent problem of the difficulty in transferring simulation strategies to reality in this scenario. It enables autonomous driving racing strategies to maintain high-speed racing performance and adapt quickly online according to the dynamic characteristics of the actual deployment environment without relying on online fine-tuning during testing, thereby achieving stable and high-speed zero-sample transfer from simulation to reality.
[0019] 2. This invention provides a method for transferring autonomous driving racing car simulations to reality based on meta-reinforcement learning. It uses domain randomization as the task generation mechanism of meta-reinforcement learning. By performing meta-training on diverse dynamic task distributions, the policy learns "how to adapt" rather than just "tolerating changes," thereby achieving zero-fine-tuning transfer while maintaining competitive performance. At the same time, this invention does not rely on explicit dynamic parameter estimation or gradient updates. Instead, it implicitly infers the current environmental dynamics by expanding the input sequence and using a temporal convolutional network, achieving online adaptation without parameter updates. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of an autonomous driving racing scenario provided by the present invention. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0022] The technical problem to be solved by this invention is: how to enable autonomous driving racing strategies to maintain high-speed racing performance and adapt quickly online according to the dynamic characteristics of the actual deployment environment without relying on online fine-tuning during testing, thereby achieving stable and high-speed zero-sample transfer from simulation to reality.
[0023] Based on this, the present invention proposes a method based on RL. 2This framework presents a meta-reinforcement learning method for transferring autonomous driving racing simulations to real-world applications. Its core idea is to treat "adaptability" itself as the learning objective—the policy no longer simply tolerates all dynamic changes within a randomized range, but learns to infer the current dynamic environment from its own interaction history and adjusts the control strategy accordingly. This approach is implemented through a two-layer learning architecture of "inner loop adaptation + outer loop optimization": Inner loop (adaptation layer): In multiple episodes of interaction with the environment, the policy encodes the interaction history through a temporal convolutional network (TCN), achieving implicit inference and adaptation of current dynamic parameters. Outer loop (meta-learning layer): The above process is repeated on a large number of different dynamic tasks, enabling the policy to learn "how to adapt," rather than optimizing only for a specific dynamic condition.
[0024] Specifically, this invention provides a method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning. After training on the simulation end, the Actor-Critic network model with fixed network meta-parameters is directly ported and deployed to the physical vehicle environment in a zero-shot manner. The Actor-Critic network model consists of a policy network and a value network with the same network structure, and both the policy network and the value network are obtained by cascading a TCN encoder and an MLP network. The structural parameters of the TCN encoder are shown in Table 1.
[0025] Table 1
[0026] The structural parameters of the MLP network are shown in Table 2.
[0027] Table 2
[0028] Furthermore, the training method for the simulation end is as follows: S1: Sample a task from the vehicle's task distribution and obtain the vehicle dynamics parameters used by the vehicle in the sampled task; The task distribution of the vehicles is shown in Table 3.
[0029] Table 3
[0030] It should be noted that the autonomous driving racing problem can be modeled as a Markov Decision Process (MDP), defined by the tuple (S, A, P, R, γ): S: State space; A: Action space; P: S × A × S → [0,1]: State transition probability distribution; R: S × A → Reward function; γ ∈ [0,1]: Discount factor for future rewards.
[0031] The agent at each time step According to the current state Select Action The environment then transitions to the next state. And return the reward The goal of strategy π: S → A is to maximize the expected cumulative discount reward.
[0032] The problem of autonomous racing extends to task distribution Each of the tasks This corresponds to an MDP with different dynamic parameters. The goal of meta-learning is to train a meta-learner. This enables it to quickly perform a given task Instantiate an adapted policy .
[0033] The meta-training phase employs a two-layer optimization structure: Inner loop: Adapts to the current sampling task to obtain task-specific policy parameters. ; Outer loop: Optimize meta parameters The goal is to maximize the expected performance of the strategy across the task distribution.
[0034] S2: Reset the internal state of the TCN encoder to the zero vector, completing the initialization of the inner loop; S3: For each of the K rounds executed in the current sampling task, execute steps S31~S33 until all rounds are completed, thus completing the adaptation process of the inner loop. S31: Reset the vehicle to the starting line position on the track, with an initial speed of 0; S32: Perform the following round-based loop operation: S32a: Construct extended inputs for each time t in the current round based on the vehicle dynamics parameters used by the vehicle in the current sampling task. , , , , These are the current vehicle status, the vehicle action in the previous moment, the step reward in the previous moment, and the end flag in the previous moment, respectively. It should be noted that this invention employs the RL² framework, designing the inner loop adaptation as an implicit process—the policy directly adjusts its output based on the historical interaction sequence without requiring any parameter updates. At each time step t, the policy input is expanded to: Let represent the current state, the action of the previous moment, the reward of the previous moment, and the end flag of the previous moment, respectively. The outer optimization objective is to maximize the expected cumulative discount reward over the entire trial (K rounds).
[0035] The vehicle status includes the forward path point, lateral deviation, longitudinal speed, yaw rate, and current steering angle, as shown in Table 4.
[0036] Table 4
[0037] S32b: Compile the h extended inputs closest to the current time t into a historical sequence. If the cumulative number of steps in the current round is less than h, then fill with zeros up to the length h; S32c: Transfer historical sequence Inputting the TCN encoder yields temporal features. ; S32d: Time-series features Inputting the MLP network yields action commands, including speed and steering angle commands; the policy network outputs normalized actions (each dimension is within the range of [-1, 1]), which are denormalized based on their respective maximum and minimum values before execution; S32e: Execute action commands to obtain the vehicle's status, step reward, collision cost, and termination flag at the next moment; Specifically, the reward at each time step t The calculation method is as follows:
[0038] in, , These represent the percentage of the vehicle's progress relative to the road centerline at time t and time t-1, respectively. , , These are the vehicle steering angles corresponding to times t, t-1, and t-2, respectively. For time penalty items, , These are the weights corresponding to progress rewards and turning rewards, respectively. and These are the reward weight and lap time for successfully completing a lap, respectively; It should be noted that the reward for successfully completing a lap is the termination reward, while the rewards in other cases are step rewards. The step reward consists of three parts: the first part represents the progress reward obtained by using the difference in the percentage of progress between the two steps relative to the center line of the road; the second constant part is used to penalize time loss; and the third part is used to penalize turning oscillations. Collision cost at each time t The calculation method is as follows:
[0039] S32f: Store the interaction data of the current round at each time point into the trajectory buffer of the current sampling task; S33: Determine if the current round has ended. If the termination condition is triggered, the current round ends and step S31 is executed again in the next round. It should be noted that the internal state of the TCN encoder is persisted across turn boundaries and is not reset. This is the key mechanism by which the RL² framework achieves cross-turn adaptation.
[0040] S4: Outer policy update, using the collected trajectory buffer data of the entire sampling task, execute the safety reinforcement learning algorithm PPO-Lagrangian update to update the policy network parameters and value network parameters; It should be noted that the goal of the PPO-Lagrangian security reinforcement learning algorithm is:
[0041] st
[0042] in, To optimize the optimal strategy, γ is the discount factor for future rewards. Let represent the expectation of the trajectory under strategy π and transfer dynamics P. The step reward at time t, Let be the collision cost at time t. This is the safety threshold.
[0043] S5: Return to step S1, sample the next task, and re-execute steps S2~S4 until the performance of the policy network on the test task converges.
[0044] Therefore, compared to the traditional DR strategy, which requires compromise under multiple or even conflicting dynamic conditions and is forced to adopt conservative strategies (such as excessively low cornering speeds) to ensure "no errors" in all tasks, severely sacrificing competitive performance, this invention adopts the RL² meta-reinforcement learning framework. The strategy continuously accumulates interaction history through the TCN encoder, gradually "perceiving" the actual dynamics of the current task within K=3 rounds and adjusting the control strategy accordingly. It does not require reserving a safety margin and solves the conservatism problem of the traditional DR strategy by transforming "static tolerance" into "dynamic adaptation".
[0045] In the simulation test results, domain randomization resulted in direct collisions under low friction conditions, making it impossible to complete a full lap test. In contrast, as shown in Table 5, the present invention can complete tests under all parameter settings, and the lap time is shorter than that of domain randomization.
[0046] Table 5
[0047] The zero-sample transfer experiment on a 1 / 10 scale physical platform also demonstrated the same effect, with a track length of 106 meters and a maximum vehicle control speed of 3 meters per second. As shown in Table 6, the present invention not only has a shorter lap time but also a smaller variance, indicating more stable performance.
[0048] Table 6
[0049] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.
Claims
1. A method for transferring the simulation of autonomous driving racing cars to reality based on meta-reinforcement learning, characterized in that, After training on the simulation platform, the Actor-Critic network model with fixed network meta-parameters is directly ported and deployed to the physical vehicle environment with zero samples. The Actor-Critic network model consists of a policy network and a value network with the same network structure, both of which are obtained by cascading a TCN encoder and an MLP network. The training method on the simulation platform is as follows: S1: Sample a task from the vehicle's task distribution and obtain the vehicle dynamics parameters used by the vehicle in the sampled task; S2: Reset the internal state of the TCN encoder to the zero vector, completing the initialization of the inner loop; S3: For each of the K rounds executed in the current sampling task, execute steps S31~S33 until all rounds are completed, thus completing the adaptation process of the inner loop. S31: Reset the vehicle to the starting line position on the track, with an initial speed of 0; S32: Perform the following round-based loop operation: Based on the vehicle dynamics parameters used by the vehicle in the current sampling task, an extended input is constructed for each time t in the current round. , , , , These are the current vehicle status, the vehicle action in the previous moment, the step reward in the previous moment, and the end flag in the previous moment, respectively. The historical sequence is composed of the h extended inputs closest to the current time t. If the cumulative number of steps in the current round is less than h, then fill with zeros up to the length h; Historical sequence Inputting the TCN encoder yields temporal features. ; Time series features Input the MLP network to obtain action instructions; Execute action commands to obtain the vehicle's status, step reward, collision cost, and termination flag at the next moment; Store the interaction data at each moment of the current round into the trajectory buffer of the current sampling task; S33: Determine if the current round has ended. If the termination condition is triggered, the current round ends and step S31 is executed again in the next round. S4: Outer policy update, using the collected trajectory buffer data of the entire sampling task, execute the safety reinforcement learning algorithm PPO-Lagrangian update to update the policy network parameters and value network parameters; S5: Return to step S1, sample the next task, and re-execute steps S2~S4 until the performance of the policy network on the test task converges.
2. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, Vehicle status includes forward path point, lateral deviation, longitudinal speed, yaw rate, and current steering angle.
3. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, Action commands include speed commands and steering angle commands.
4. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, Each task in the vehicle's task distribution corresponds to a Markov decision process with different dynamic parameters.
5. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, In step S33, the internal state of the TCN encoder is retained between round boundaries and is not reset.
6. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, In S33, the termination condition is that the car completes one lap or a collision occurs.
7. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, Reward at each time t The calculation method is as follows: in, , These represent the percentage of the vehicle's progress relative to the road centerline at time t and time t-1, respectively. , , These are the vehicle steering angles corresponding to times t, t-1, and t-2, respectively. For time penalty items, , These are the weights corresponding to progress rewards and turning rewards, respectively. and These are the reward weight and lap time for successfully completing a lap, respectively.
8. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, Collision cost at each time t The calculation method is as follows: 。 9. The method for transferring autonomous driving racing car simulation to reality based on meta-reinforcement learning as described in claim 1, characterized in that, In step S4, the target of the PPO-Lagrangian security reinforcement learning algorithm is updated as follows: s.t. in, To optimize the optimal strategy, γ is the discount factor for future rewards. Let represent the expectation of the trajectory under strategy π and transfer dynamics P. The step reward at time t, Let be the collision cost at time t. This is the safety threshold.