A simulation environment dynamic updating robot training method

Abstract

The application discloses a simulation environment dynamic updating robot training method. The application relates to the technical field of robot simulation training, and particularly relates to the technical field of a simulation environment dynamic updating robot training method. According to a simulation environment contribution evaluation method and a simulation environment dynamic updating algorithm, a simulation environment pool is dynamically updated, and the simulation environment pool after dynamic updating is used for robot simulation training. The method comprises the following steps: setting a simulation engine set; initializing a simulation environment probability distribution; sampling an environment; in order to evaluate an environment contribution characteristic value of an initial simulation engine test environment, environment contribution evaluation parameters are selected, including a training environment cumulative reward function average value, a training environment average survival time, a training environment average running time, a cross-simulation environment cumulative reward function average value and a cross-simulation environment average survival time; calculating an environment contribution score; constructing an environment distribution updating function; and calculating an environment probability distribution.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of robot simulation training, and in particular to a simulation environment dynamic updating robot training method. BACKGROUND

[0002] Robot simulation environment methods have developed rapidly, significantly promoting robot research and application. Thanks to the improvement of computing power and graphics processing, simulation platforms such as Gazebo, Webots, MuJoCo and IsaacLab have become more realistic, supporting complex physical modeling and multi-robot collaborative simulation. The rise of deep learning and reinforcement learning has prompted the simulation environment to become a core tool for training robot control strategies, especially in the fields of autonomous driving, unmanned aerial vehicles and service robots. The digital twin technology of virtual and real combination further enhances the interaction ability between simulation and reality, which is used for real-time monitoring and optimization of robot systems. At the same time, the growth of the open source community (such as the ROS ecosystem) has reduced the development threshold, promoting algorithm sharing and cross-field application. However, training robots quickly and efficiently in a simulation environment is still a major challenge.

[0003] Chinese invention patent "A method for accelerating simulation robot deep reinforcement learning" (CN110728368B) proposes a method for accelerating simulation robot deep reinforcement learning, which selects a learning node to cooperate with multiple environment nodes, and optimizes the learning process in a distributed framework to reduce the time overhead of research, debugging and deployment.

[0004] The 2022 master's degree thesis of National University of Defense Technology "Research on Reinforcement Learning Training Acceleration Method for Robot Tasks" (Zhao Yong) addresses the problem of long reinforcement learning training time and complex simulation environment in robot tasks, and proposes an acceleration method that combines different precision simulation environment sampling and distributed computing technology to improve training efficiency.

[0005] arXiv repository: 2504.18904 (2025) "RoboVerse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning." (Geng Haoran) proposes a comprehensive robot learning framework RoboVerse, aiming to solve the problems of data expansion and evaluation standardization. It integrates multiple simulators into a unified simulation platform through MetaSim, abstracts them into a general interface, and realizes seamless integration and expansion. RoboVerse provides a high-fidelity synthetic dataset, combined with public dataset transfer, policy generation and data augmentation methods, covering a variety of scenarios and tasks, and has realistic rendering effects.

[0006] However, the above technical solutions still have the following defects:

[0007] (1) Existing simulation environments (platforms) (such as the distributed framework or MetaSim of RoboVerse) usually adopt preset resource allocation strategies, such as fixed allocation of the number of simulation nodes, simulator types (such as Gazebo, MuJoCo), or simulation step length. This static allocation cannot adapt to changes in task requirements during the training process (such as the need for rapid exploration at the beginning and fine-tuning with high precision at the end).

[0008] When the complexity of the task or the quality of the data requirements change, static allocation may cause some simulator resources to be idle or overloaded, reducing computational efficiency. For example, a low-precision simulator may generate invalid data during the fine-tuning stage, while a high-precision simulator may occupy too many computational resources during the initial exploration stage.

[0009] (2) Existing platforms lack evaluation mechanisms to monitor the quality of data generated by different simulators or simulation step lengths (such as reward convergence speed, model generalization performance). Therefore, the system cannot dynamically adjust resource allocation based on data effectiveness, such as preferentially allocating high-precision simulators to high-reward tasks or reducing the resource occupation of inefficient simulation nodes.

[0010] The lack of feedback causes resource allocation to deviate from training goals, which may waste computational resources on low-quality data generation, prolong training time, or reduce model performance. SUMMARY

[0011] To address the problem that existing simulation environments cannot adapt to changes in task requirements during the training process and lack evaluation feedback mechanisms, causing resource allocation to deviate from training goals, the purpose of the present invention is to propose a simulation environment dynamic updating method for robot training. According to the simulation environment contribution evaluation method and the simulation environment dynamic updating algorithm, the simulation environment pool is dynamically updated, and the simulation environment pool after dynamic updating is used for robot simulation training. The method described in the present invention can dynamically allocate computational resources and can adjust the simulation environment in a timely manner based on partial simulation results.

[0012] The method comprises the following steps:

[0013] S1, set a set of simulation engines , wherein, represents the th simulation engine, represents the index value of different simulation engines;

[0014] Initialize the simulation environment probability distribution , wherein, represents the simulation step length, The value range of This represents the minimum simulation step size. Indicates the maximum simulation step size. Indicates the current iteration number. , This indicates the total number of simulation environment iterations;

[0015] S2. Conduct environmental sampling: according to Random initialization Configure a training simulation environment;

[0016] Set the current sampling simulation step size set ,in, Indicates the first One simulation step size, Index values ​​representing different simulation step sizes;

[0017] S3. To evaluate the environmental contribution feature value of the initial simulation engine test environment, the following environmental contribution evaluation parameter is selected: the average value of the cumulative reward function of the training environment. Average survival time in the training environment Average runtime of training environment Average value of cumulative reward function across simulation environments and average survival time across simulation environments

[0018] S4. For the environmental contribution assessment parameters selected in step S3, construct their interpolation models using the K-nearest neighbor algorithm, and normalize them to between 0 and 1 using the softmax function.

[0019] S5. Calculate the environmental contribution score based on the results of step S4. ;

[0020] S6, according to Constructing an environment distribution update function

[0021]

[0022] Indicates the first One simulated step size;

[0023] S7, will Perform normalization and calculate the environmental probability distribution. ,

[0024] in, Represents the normalization coefficient;

[0025] S8, according to Update the next round of environmental sampling.

[0026] Furthermore, the average value of the cumulative reward function in the training environment The calculation formula is:

[0027] ,in, This represents the total number of agents in each simulation environment. Indicates the first An intelligent agent. Indicates the first One survival step, express The total number of survival steps for each agent. Indicates the first Wheel of Life The agent in the th... The reward function value of the step. , This represents the total number of training rounds for each agent. Represents the discount factor, in Internal value, Indicates the first Discount factor for survival steps;

[0028] The average survival time of the training environment The calculation formula is: ;

[0029] The average runtime of the training environment The calculation formula is: ,in, express Training of individual agents The system runtime of the wheel;

[0030] The average value of the cumulative reward function across simulation environments The calculation formula is:

[0031] in, and These represent the training engine and the training simulation step size, respectively. and These represent the test engine and the test simulation step size, respectively.

[0032] The average survival time across simulation environments The calculation formula is:

[0033]

[0034] Further, in step S4, the interpolation model normalizes the results between 0 and 1 using a softmax function, including: normalizing the training cumulative average reward feature Normalizing the training average survival time feature Normalizing the training running time feature Normalizing the cross-platform cumulative average reward feature And normalizing the cross-platform average survival time feature

[0035] Further, the calculation formula of the environment contribution score Wherein, The environment contribution value is represented.

[0036] Further, the calculation formula of the environment contribution value

[0037]

[0038] Wherein, , , , And Respectively represent the weighted coefficients of And

[0039] Further, the calculation formula of the normalization coefficient Wherein, The integral between the minimum simulation step To the maximum simulation step

[0040] Further, the calculation formula of the The integral function is represented.

[0041] The method has the following advantages:

[0042] (1) The method can dynamically update the simulation environment pool according to the simulation environment contribution evaluation method and the simulation environment dynamic updating algorithm, can dynamically allocate resources according to part of the simulation results, dynamically adjust the simulation environment most beneficial to robot training, and thus improve the training efficiency and save the training time.

[0043] ​​​​​​​(2) The existing precise simulation engine (for example, Mujoco) can achieve very precise simulation effect, but its computing resource consumption is huge, and the system running time is long, and the method disclosed in the application can balance various factors such as training effect, computing resource and system running time, and dynamically select a better simulation environment, which can save computing resource and system running time under the premise of meeting the training effect, and can achieve the training effect of the precise simulation engine.

[0044] (3) Since the simulation environment dynamic updating method disclosed in the application directly selects different simulation engine training environments in the training process, and has been tested in different simulation environments in the process of judging the training effect, the adaptability of the agent in different simulation environments is improved, so the agent trained by the method disclosed in the application can reduce or even not need the sim2sim verification process, and further improve the simulation training efficiency.

[0045] (4) The method disclosed in the application selects an environment contribution evaluation parameter to quantify the influence of the current simulation environment configuration on the reinforcement learning effect, and constructs an interpolation model of the environment contribution feature to estimate the environment features of the unsampled simulation steps. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The flowchart of the method disclosed in the application;

[0047] Figure 2 The normalized training average reward comparison diagram of the application;

[0048] Figure 3 The normalized training survival time comparison diagram of the application;

[0049] Figure 4 The training total time table of the application;

[0050] Figure 5 The normalized cross-platform reward diagram of the method disclosed in the application after training;

[0051] Figure 6 The normalized cross-platform average reward diagram of the Mujoco method disclosed in the application after training;

[0052] Figure 7 The normalized cross-platform average reward diagram of the Genesis method disclosed in the application after training;

[0053] Figure 8 The normalized cross-platform average reward diagram of the IsaacLab method disclosed in the application after training;

[0054] Figure 9This is a schematic diagram of the normalized cross-platform survival time after training using the method described in this invention.

[0055] Figure 10 This is a schematic diagram of the normalized cross-platform survival time after the Mujoco method described in this invention has been trained.

[0056] Figure 11 This is a schematic diagram of the normalized cross-platform survival time after the Genesis method described in this invention has been trained.

[0057] Figure 12 This is a schematic diagram of the normalized cross-platform survival time after the IsaacLab method described in this invention has been trained. Detailed Implementation

[0058] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1

[0060] This embodiment provides a method for dynamically updating robot training in a simulation environment. The flowchart of the method is as follows: Figure 1 As shown, the method includes the following steps:

[0061] initialization:

[0062] Set up a collection of simulation engines , Indicates the first A simulation engine, setting the simulation step size. The range of values ​​for: in, and These represent the minimum and maximum values ​​that the simulation step size can take, respectively.

[0063] Initialize the simulation environment To ensure a uniform distribution, the superscript 0 indicates the 0th iteration, meaning training has not yet begun. In the 0th iteration, we can assume... and If they are mutually independent, then , Can be , express It follows a uniform probability distribution, that is, exist The uniform distribution of discrete random variables represents a physical simulation engine, and the value (simulation engine) can be {IsaacLab, IsaacGym, Mujoco, Genesis, PyBullet}. The probability can be a uniform distribution.

[0064] Building a simulation engine test environment , test environments , wherein , represents a specific test environment index, represents the total number of test environments; for example: when , the set of simulation step lengths is , the set of test environments is represented as:

[0065]

[0066] Update iteration start: (from to ), represents the current iteration number, represents the total number of iterations.

[0067] First step environment sampling:

[0068] Randomly initialize training simulation environment configurations using the random distribution , wherein represents the training simulation environment index.

[0069] Second step reinforcement learning training:

[0070] Initialize agents in each simulation environment for rounds of reinforcement learning training, , represents the round of training.

[0071] Third step environment contribution feature calculation:

[0072] In order to evaluate the environment contribution feature value of the initial simulation engine test environment, reflecting the running effect of the agent in the environment, the following five environment contribution evaluation parameters are selected:

[0073] (1) Average value of cumulative reward function of training environment :

[0074] The calculation formula of the average value of the cumulative reward function of the training environment is:​ wherein, represents the total number of agents under each simulation environment, represents the th agent, represents the th survival step, represents the total number of survival steps of the th agent, represents the reward function value of the th agent in the th round at the th step, , represents the total number of training rounds, represents a discount factor, which takes a value within , represents the discount factor of the th survival step; is used to quantify the influence of the current simulation environment configuration on the reinforcement learning effect, and the greater the value, the better the effect.

[0075] (2) Training environment average survival time :

[0076] The calculation formula of the training environment average survival time is: ; is used to quantify the influence of the current simulation environment configuration on the reinforcement learning effect, and the value reflects the reinforcement learning effect. It is not always the longer the better for all tasks, and should be considered comprehensively in combination with the average value of the cumulative reward function.

[0077] (3) Training environment average running time :

[0078] The calculation formula of the training environment average running time is: , wherein, represents the system running time of the th agent training rounds; is used to quantify the influence of the current simulation environment configuration on the reinforcement learning effect. The value indirectly reflects the reinforcement learning effect, which should be considered comprehensively in combination with the training average survival time. The training average survival time is positively correlated with the training average environment running time.

[0079] (4) Cross-simulation environment cumulative reward function average value

[0080] The calculation formula of the cross-simulation environment cumulative reward function average value is:

[0081]

[0082] wherein, and represent training engine and training simulation step length respectively, and represent testing engine and testing simulation step length respectively; for quantifying the influence of the current simulation environment configuration on the reinforcement learning effect, the value directly reflects the reinforcement learning effect, and the larger the value, the better the effect.

[0083] (5) Cross-simulation environment average survival time

[0084] The cross-simulation environment average survival time The calculation formula is:

[0085] for quantifying the influence of the current simulation environment configuration on the reinforcement learning effect, the value directly reflects the reinforcement learning effect, and the larger the value, the better the effect.

[0086] In order to better preserve the local characteristics of the feature function, the method of the application uses K-nearest neighbor algorithm to construct interpolation models of the above-mentioned five environment contribution evaluation parameters respectively: , , , , Since the training environment is randomly generated, the method of the application constructs the corresponding interpolation model to estimate the environment contribution characteristics of a specific simulation step length. Taking the training cumulative average reward interpolation function as an example, it is assumed that each simulation engine has simulation step length simulation environment, and the simulation step length set can be represented as , and the corresponding cumulative reward function average value set can be represented as By using the nearest neighbor algorithm, the training cumulative average reward interpolation function is estimated. First, the distance feature set is calculated as , the distance feature set is sorted from small to large, and the smallest simulation step length is taken, and the average value can obtain the cumulative average reward interpolation function: , represent the simulation step length. Similarly, the training average survival time interpolation function , the running time , the cross-platform testing cumulative average reward , the cross-platform average survival time interpolation function , can be estimated by the above method.

[0087] The interpolation model can estimate the environment characteristics of the simulation step that is not sampled.

[0088] Environment contribution feature normalization:

[0089] Considering that different environment contribution features have different dimensions, the invention selects a softmax function to normalize them between 0 and 1, making the score estimation more uniform and smooth, and excluding the influence of different dimensions.

[0090] The results of the above normalization are: normalized training cumulative average reward feature , normalized training average survival time feature , normalized training running time feature , normalized cross-platform cumulative average reward feature , and normalized cross-platform average survival time feature .

[0091] Environment contribution score calculation:

[0092] First, calculate the environment contribution value ,

[0093]

[0094] Among them, , , , and represent the weighted coefficients of .

[0095] Since the environment contribution score can be used to represent discrete probability, but it needs to ensure that the overall sum is 1, the environment contribution value needs to be normalized to obtain the environment contribution score ,

[0096] The environment contribution score method uses a linear model to ensure interpretability and facilitate user weight adjustment.

[0097] Environment distribution update:

[0098] According to the environment contribution score calculated from the current training result, the environment contribution score is used to estimate the probability distribution of the next round of environment sampling: first, for each , use linear interpolation to generate : between to , construct Since the normalized score is an estimate of discrete points, but a continuous function is needed to represent the probability density, we use linear interpolation to obtain the corresponding continuous function.

[0099] because It is a continuous function. To obtain a probability density function, we need to... Perform normalization to ensure that the function is within range. arrive The integral between them is 1.

[0100] Then to To normalize the distribution: First, calculate the normalization coefficient. :

[0101] in, express Minimum simulation step size To the maximum simulation step size The integral between them. This represents an integral function.

[0102] Finally, according to Calculate the updated environment probability distribution ,

[0103] The probability distribution for the next round of environmental sampling is obtained through the current environmental contribution score. Therefore, in the next round of environmental sampling, it will be easier to sample the vicinity of the environment with a high current environmental contribution score. The sampling used in the next round will make the probability distribution update of each round of environment sampling more inclined to the environment that is favorable to training.

[0104] Example 2

[0105] This embodiment is a further limitation of Embodiment 1.

[0106] This embodiment utilizes Roboverse to construct a simulation environment for training a robot dog's speed tracking reinforcement learning.

[0107] This embodiment sets up a simulation engine collection. Set the simulation step size interval The test environment is set up as follows: Set the initialization simulation environment probability distribution It is a uniform random distribution.

[0108] The simulation environment was updated 5 times, and each environment underwent 5000 training rounds in each iteration.

[0109] Initialize 20 simulation environments;

[0110] Each simulation environment initializes 64 agents for training, and each environment training uses rl_rs (a library of algorithms for reinforcement learning) for 1000 rounds of reinforcement learning training.

[0111] This embodiment compares the method and single simulation engine (simulation step length is 0.02) described in the application: Mujoco, Genesis and IsaacLab. In the comparative experiment, each environment is set with one agent, without switching any simulation environment and simulation step length, and is trained for 5000 rounds.

[0112] In the comparative experiment:

[0113] The normalized training average reward comparison result is shown in Figure 2 ;

[0114] The normalized training survival time comparison result is shown in Figure 3 ;

[0115] The total training time comparison result is shown in Figure 4 ;

[0116] The normalized cross-platform reward comparison results of the results of the method and the Mujoco, Genesis and IsaacLab simulation environment training in the Mujoco, Genesis and IsaacLab simulation environment are shown in Figure 5 , 6 , 7 and 8, respectively;

[0117] The normalized cross-platform survival time comparison results of the results of the method and the Mujoco, Genesis and IsaacLab simulation environment training in the Mujoco, Genesis and IsaacLab simulation environment are shown in Figure 9 , 10 , 11 and 12, respectively;

[0118] According to the comparison experiment results, it can be known that the method described in the application (the method) can significantly improve the training efficiency and save the training time. Under the same training conditions (the same reinforcement learning toolkit, training parameter setting, the same problem, and the same simulation environment configuration (the same simulation step length and the same simulation engine)), the method described in the application can achieve similar effects as the mujoco (the most accurate) simulation environment, but the training time is reduced by more than 50%.

[0119] And because the simulation environment dynamic updating method of the application directly selects different simulation engine training environments during the training process, and the training effect has been tested in different simulation environments during the evaluation process, the adaptability of the agent in different simulation environments is improved, so the agent trained by the method of the application can reduce or even eliminate the need for a sim2sim verification process.

Claims

1. A dynamic updating robot training method in a simulation environment, characterized in that, The method comprises the following steps: S1, setting a set of simulation engines wherein, denotes a simulation engine, denotes an index value for a different simulation engine; Initializing a simulation environment probability distribution wherein, denotes a simulation step size, the value interval of is: denotes a minimum simulation step size, denotes a maximum simulation step size, denotes a current iteration number, , denotes a total number of simulation environment iteration numbers; S2, environmental sampling: according to , randomly initialize training simulation environment configurations; Setting current round sampling simulation step set wherein, denotes the simulation step, denotes the index value of different simulation steps; each simulation environment is initialized each agent is trained for one round of reinforcement learning training, , denotes the round of training; S3, select environment contribution evaluation parameters to evaluate the environment contribution characteristic value of the initial simulation engine test environment, the selected environment contribution evaluation parameters include: training environment cumulative reward function average , training environment average survival time , training environment average running time , cross-simulation environment cumulative reward function average , and cross-simulation environment average survival time ; The average value of the cumulative reward function in the training environment The calculation formula is: ,in, This represents the total number of agents in each simulation environment. Indicates the first An intelligent agent. Indicates the first One survival step, express The total number of survival steps for each agent. Indicates the first Wheel of Life The agent in the th... The reward function value of the step. Represents the discount factor, in Internal value, Indicates the first Discount factor for survival steps; The training environment average survival time The calculation formula is: ; The training environment average run time The formula is: Wherein, Indicates The system run time of the agent training Round The cross-simulation environment cumulative reward function average value The calculation formula is: wherein, and denote the training engine and the training simulation step size, respectively, and denote the testing engine and the testing simulation step size, respectively, denotes the test reward function value of the th agent in the th episode at the th step. The cross-simulation environment average survival time The formula for calculating is: ； S4, for the environmental contribution evaluation parameters selected in step S3, an interpolation model of each selected evaluation parameter is constructed respectively by using a K-nearest neighbor algorithm, and each constructed interpolation model is normalized to between 0 and 1 by using a softmax function; S5. Calculate the environmental contribution score based on the normalized results of step S4 ; The environmental contribution score The calculation formula is: in, Indicates the first The simulation engine in the first Environmental contribution value per simulation step; The first The simulation engine in the first Environmental contribution value per simulation step The calculation formula is: wherein , , , and respectively represent a weighting factor of the first channel S6、According to Constructing an environmental distribution update function represents the first simulation step; S7, to normalizing, computing the environmental probability distribution , wherein, denotes a normalization coefficient; The normalization coefficient The calculation formula is: wherein, represents the integral from At the minimum simulation step to the maximum simulation step between. The The calculation formula is: denotes the integral function; S8. The method of any of S1-S7, Update the next round of environmental sampling.

2. The dynamic updating robot training method in a simulated environment according to claim 1, wherein, In step S4, the interpolation model normalizes the results to between 0 and 1 using a softmax function includes: normalized training cumulative average reward features Normalized training average survival time features Normalized training run time features Normalized cross-platform cumulative average reward features And normalized cross-platform average survival time features .

Images