Robot reinforcement learning training method and device, and storage medium

Abstract

The application relates to the technical field of robots, and discloses a robot reinforcement learning training method and device and a storage medium, the method comprising the following steps: obtaining a current observation value, determining a target joint position according to a policy model and the current observation value; calculating a joint control torque through the target joint position and the current observation value, applying the joint control torque to a simulation robot model, and collecting an updated observation value; and generating a reinforcement training policy according to a reward function and the updated observation value. In the manner, the application simulates the cooperation mode of a real robot control system by performing policy reasoning at a first preset frequency and performing physical simulation at a second preset frequency. By introducing the reward function, dense and accurate learning signals are provided for the simulation robot model from two dimensions of overall posture balance and local joint stress, the robot is actively guided to maintain the center of gravity within a support polygon in the static and motion states, and the reliability of robot training is improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, apparatus and storage medium for robot reinforcement learning training. Background Technology

[0002] Humanoid robots, due to their high compatibility with human environments, have become a research frontier in the field of robotics. To enable robots to perform complex dynamic tasks such as walking, running, and carrying loads, reinforcement learning has become a mainstream method for training control strategies. Training in a simulated environment can avoid damage to the physical robot, greatly improve training efficiency, and reduce trial-and-error costs.

[0003] Currently, building training environments based on physics simulation engines such as Isaac Sim and MuJoCo is a common practice. However, different simulation engines have different focuses in their design and optimization goals, leading to inherent performance differences and limitations. Specifically, some simulation platforms are based on high-performance GPU rendering and physics engines, possessing industrial-grade multi-sensor rendering capabilities, and can efficiently handle complex visual information and a large number of parallel instances, resulting in fast training speeds and significant effects. However, they are relatively weak in the modeling accuracy of rigid body dynamics, especially physical details such as contact mechanics and friction coefficients. This often leads to a "sim2real" gap problem when policy models trained in Isaac Sim are directly deployed to physical robots due to inaccurate contact mechanics simulations. For example, unpredictable scenarios such as robot joint tremors, foot slippage, or instability under slight external forces may occur. Therefore, how to improve the reliability of robot training by refining the design of robot training functions has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a robot reinforcement learning training method, apparatus, and storage medium, which improves the reliability of robot training by finely designing the robot's training functions.

[0005] Firstly, this application provides a robot reinforcement learning training method, the method comprising:

[0006] In a preset simulation scenario, the current observation value of the simulation robot model is obtained according to a first preset frequency. Based on the preset strategy model and the current observation value, the target joint position of the simulation robot model is determined.

[0007] The joint control torque is calculated based on the target joint position and the current observation value according to the second preset frequency, and the joint control torque is applied to the simulated robot model, and the observation value is collected and updated.

[0008] Based on the preset reward function and the updated observations, a reinforcement training strategy is generated to perform reinforcement learning training on the simulated robot model.

[0009] Secondly, this application also provides a robot reinforcement learning training device, the device comprising:

[0010] The target joint position determination module is used to obtain the current observation value of the simulation robot model according to a first preset frequency in a preset simulation scenario, and determine the target joint position of the simulation robot model according to a preset strategy model and the current observation value.

[0011] The observation acquisition module is used to calculate the joint control torque based on the target joint position and the current observation value according to a second preset frequency, apply the joint control torque to the simulated robot model, and acquire and update the observation value.

[0012] The reinforcement learning training module is used to generate a reinforcement training strategy based on a preset reward function and the updated observations, so as to perform reinforcement learning training on the simulated robot model through the reinforcement training strategy.

[0013] Thirdly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the robot reinforcement learning training method described above.

[0014] This application discloses a robot reinforcement learning training method, apparatus, and storage medium. The robot reinforcement learning training method includes acquiring the current observation values ​​of a simulated robot model in a preset simulation scenario according to a first preset frequency; determining the target joint position of the simulated robot model according to a preset policy model and the current observation values; calculating joint control torques according to the target joint position and the current observation values ​​at a second preset frequency, applying the joint control torques to the simulated robot model, and collecting updated observation values; generating a reinforcement training strategy according to a preset reward function and the updated observation values ​​to perform reinforcement learning training on the simulated robot model. Through the above method, this application simulates the cooperative mode of a real robot control system by using a "first preset frequency" for policy reasoning and a "second preset frequency" for physical simulation. By introducing a reward function, dense and accurate learning signals are provided to the simulated robot model from two dimensions: overall posture balance and local joint forces. This actively guides the robot to maintain its center of gravity within the supporting polygon in both static and dynamic states, improving the reliability of robot training. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic flowchart of a robot reinforcement learning training method provided in an embodiment of this application;

[0017] Figure 2 A schematic block diagram of a robot reinforcement learning training device provided for embodiments of this application;

[0018] Figure 3 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0021] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] This application provides a robot reinforcement learning training method, apparatus, and storage medium. The robot reinforcement learning training method can be applied to a server, using a "first preset frequency" for policy reasoning and a "second preset frequency" for physical simulation, simulating the cooperative mode of a real robot control system. By introducing a reward function, dense and precise learning signals are provided to the simulated robot model from two dimensions: overall posture balance and local joint forces. This actively guides the robot to maintain its center of gravity within the supporting polygon in both static and dynamic states, improving the reliability of robot training.

[0024] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a robot reinforcement learning training method provided in an embodiment of this application. This robot reinforcement learning training method can be applied to a server to improve the reliability of robot training.

[0026] like Figure 1 As shown, the robot reinforcement learning training method specifically includes steps S10 to S30.

[0027] Step S10: In a preset simulation scenario, obtain the current observation value of the simulation robot model according to the first preset frequency, and determine the target joint position of the simulation robot model according to the preset strategy model and the current observation value;

[0028] Specifically, in a preset simulation scenario, the current observation value of the simulation robot model is periodically acquired at a first preset frequency (e.g., one control step is executed every two control cycles). The observation value includes joint position, velocity, base state and foot contact information. Continuous historical data containing the current observation value is input into a preset neural network strategy model for forward calculation, and the target joint position of each joint of the simulation robot model is output and determined.

[0029] Step S20: Calculate the joint control torque based on the target joint position and the current observation value according to the second preset frequency, apply the joint control torque to the simulation robot model, and collect and update the observation value;

[0030] Specifically, the physical simulation loop is run at a higher second preset frequency (e.g., one physical simulation step per control cycle): based on the target joint position and the actual joint position and velocity in the current observation, the precise joint control torque is calculated by the proportional-derivative (PD) controller; this torque is applied to the corresponding joint of the simulated robot model, and the physical engine is driven forward by one simulation step, thereby updating the dynamic state of the entire system and collecting the latest observation after the physical simulation step update.

[0031] Step S30: Generate a reinforcement training strategy based on the preset reward function and the updated observation value to perform reinforcement learning training on the simulation robot model through the reinforcement training strategy.

[0032] Specifically, the updated observations are calculated based on the preset reward functions (including the base stability reward function and the joint force balance reward function) to generate a comprehensive reward value. The comprehensive reward value is fed back to the reinforcement learning algorithm to update the parameters of the policy network, thereby generating an optimized reinforcement training policy. The training policy is used for subsequent control decisions, thus forming a complete closed loop from simulation to policy update, continuously iterating and optimizing, and finally training a humanoid robot control policy that can move stably and efficiently.

[0033] This embodiment discloses a robot reinforcement learning training method, apparatus, and storage medium. The robot reinforcement learning training method includes acquiring the current observation values ​​of a simulated robot model in a preset simulation scenario according to a first preset frequency; determining the target joint position of the simulated robot model according to a preset strategy model and the current observation values; calculating joint control torques according to the target joint position and the current observation values ​​at a second preset frequency, applying the joint control torques to the simulated robot model, and collecting updated observation values; generating a reinforcement training strategy according to a preset reward function and the updated observation values ​​to perform reinforcement learning training on the simulated robot model. Through the above method, this application simulates the cooperative mode of a real robot control system by using a "first preset frequency" for strategy reasoning and a "second preset frequency" for physical simulation. By introducing a reward function, dense and accurate learning signals are provided to the simulated robot model from two dimensions: overall posture balance and local joint forces. This actively guides the robot to maintain its center of gravity within the supporting polygon in both static and dynamic states, improving the reliability of robot training.

[0034] based on Figure 1 In the illustrated embodiment, step S10 includes:

[0035] In the preset simulation environment, the current joint position of the simulated robot model in the current cycle is collected according to the first preset frequency, and the current joint position is determined as the current observation value;

[0036] The target action is determined based on the preset strategy model and the current joint position;

[0037] The target joint position is determined based on the current joint position and the target motion.

[0038] Specifically, in a preset simulation environment, the operation loops according to a first preset frequency. At the beginning of each control cycle, the actual angle data of each driven joint of the simulated robot model at the current moment is collected from the physical simulation engine, i.e., the current joint position. This data is determined as the main current observation value for that control cycle. The current joint position data is concatenated and stacked with the previous frame data in the observation history buffer to form a stacked observation vector containing temporal information. The stacked observation vector is input into a preset neural network strategy model. After forward computation, the neural network strategy model outputs an action vector, which is the target action for this control cycle. Calculations are performed based on the current joint position and the target action:

[0039] If the strategy model outputs an absolute position command, then the target action itself is the target joint position;

[0040] If the output is a relative increment, the current joint position is added to the target motion to obtain the final target joint position.

[0041] In a specific embodiment, step S20 includes:

[0042] According to the second preset frequency, the joint control torque is calculated based on the target joint position and the current joint position;

[0043] Based on the target action, generate control commands;

[0044] In response to the control command, the joint control torque is applied to the simulated robot model, and the updated joint position of the simulated robot model is acquired.

[0045] Specifically, the inner physical simulation loop is run at a higher second preset frequency (e.g., one physical simulation step per control cycle). Based on the target joint position obtained from shared variables and the current joint position and velocity read in real-time from the simulation environment, a proportional-derivative (PD) controller calculates the joint control torque. This calculated joint control torque is encapsulated as a control command. Responding to this command, the simulation engine's program interface is invoked to apply the joint control torque to the corresponding joint actuator of the simulated robot model, driving the simulation engine forward by a minimum physical time step. Once the simulation calculation for this step is complete, updated state data for all joints is immediately collected from the simulation environment, i.e., the joint positions are updated, thus completing one physical simulation cycle and providing the latest state information for the decision-making in the next control cycle.

[0046] In a specific embodiment, step S30 includes:

[0047] The base stability reward value is generated using the base stability reward function and the updated observation value;

[0048] The joint force balance reward value is generated using the joint force balance reward function and the updated observation value;

[0049] The reinforcement training strategy is generated by weighting and summing the base stability reward value and the joint force balance reward value based on a preset ratio.

[0050] Specifically, the robot's base stability reward value is calculated using updated observations. The method involves acquiring the coordinates of the projection points of both feet in real time, calculating the center point of the line connecting them as a dynamic reference point, calculating the current Euclidean distance between the robot's base center projection point and this dynamic reference point, and generating a corresponding negative reward value when the current Euclidean distance exceeds a set threshold. The final base stability reward value is then output. Simultaneously, a joint force balance reward value is generated based on the joint torque data in the updated observations: if the robot's command speed is lower than a set threshold, the torque values ​​of the left and right symmetrical joints are acquired, the ratio of the absolute difference of the torque of each symmetrical joint to the sum of the torques is calculated, and the absolute value is taken. The ratios of all symmetrical joints are summed to obtain the total penalty value, which is then negatively evaluated and used as the joint force balance reward value.

[0051] The system performs a weighted summation of the base stability reward value and the joint force balance reward value according to a pre-set weight ratio (e.g., 7:3), and feeds the resulting comprehensive reward value back into the reinforcement learning algorithm. The parameters of the policy network are optimized through the policy gradient update mechanism, thereby generating a performance-optimized reinforcement training policy.

[0052] In a specific embodiment, the base stability reward value is generated through the base stability reward function and the updated observation value, including:

[0053] Obtain the projection point of the simulated robot model on the horizontal plane;

[0054] Calculate the current Euclidean distance between the projection point and the preset dynamic reference point, wherein the preset dynamic reference point is the center point of the projection of the left and right feet of the simulation robot model;

[0055] The base stability bonus value is determined based on the relationship between the current Euclidean distance and the preset Euclidean distance threshold.

[0056] If the current Euclidean distance is less than the preset Euclidean distance threshold, then the base stability bonus value is determined to be the first bonus value;

[0057] If the current Euclidean distance is greater than or equal to the preset Euclidean distance threshold, then the base stability reward value is determined to be the second reward value.

[0058] Specifically, in the implementation of this step, the system first obtains the coordinates P of the projection point of the center of the simulation robot model base on the horizontal plane from the simulation engine. base (x b ,y b Simultaneously, the coordinates of the center points of the robot's left and right feet projected onto the horizontal plane are obtained through foot sensors, denoted as P. left (x l ,y l ) and P right (x r ,y r Subsequently, the center point of the left and right foot projection center points is calculated as the preset dynamic reference point P. ref Its coordinates are ((x l +x r ) / 2,(y l +y r Next, calculate the projection point P of the base. base With dynamic reference point P ref The current Euclidean distance d between them is calculated using the following formula:

[0059] The system compares the current Euclidean distance d with the preset Euclidean distance threshold d. threshold Comparison:

[0060] If d <d threshold Then, the base stability reward value is determined to be the first reward value, which is either zero or a positive reward constant;

[0061] If d≥d thresholdThen the base stability bonus value is determined to be the second bonus value, which is a negative number of the current Euclidean distance d or a negative bonus value proportional to d, that is, bonus value = -k×d (where k is the proportionality coefficient).

[0062] The final output is the calculated base stability reward value, which is used for subsequent reward weighted summation and strategy optimization.

[0063] In a specific embodiment, the joint force balance reward value is generated using the joint force balance reward function and the updated observation value, including:

[0064] If the command speed value is greater than a preset speed threshold, the joint force balance reward value is determined as the first balance reward value according to the joint force balance reward function;

[0065] If the commanded speed value is less than or equal to the preset speed threshold, the current torque value of each group of symmetrical joints is extracted from the updated observation value;

[0066] The joint force balance reward value is generated based on the current torque value and the joint force balance reward function.

[0067] Specifically, it is determined whether the command velocity value of the simulated robot model obtained from the updated observations is greater than a preset velocity threshold. If it is greater, the joint force balance reward value is directly determined as the first balance reward value (usually set to zero), and torque calculation is not required at this time;

[0068] If the commanded speed value is less than or equal to a preset speed threshold, it indicates that the robot is in a stationary or low-speed stable state. Then, the current torque values ​​of the left and right symmetrical joint groups (such as the left and right knee joints, or the left and right hip joints) are extracted from the updated observations. Based on these torque values, the torque imbalance of each symmetrical joint group is calculated and summed to finally generate a joint force balance reward value. Specifically, for each symmetrical joint group, the ratio of the difference in its absolute torque value to the sum is calculated, and this ratio is summed across all symmetrical joints and then negative to obtain the final joint force balance reward value.

[0069] In a specific embodiment, the joint force balance reward value is generated based on each of the current torque values ​​and the joint force balance reward function, including:

[0070] The single-group joint balance reward value of each group of symmetrical joints is calculated using the joint force balance reward function and the current torque value of each group of symmetrical joints.

[0071] The joint force balance reward value is obtained by summing all the single-group joint balance reward values.

[0072] Specifically, for each group of symmetrical joints (such as left hip-right hip, left knee-right knee, left ankle-right ankle), the balance reward value for a single group of joints is calculated using the joint force balance reward function and the current torque value of that group of symmetrical joints. The specific calculation method involves obtaining the absolute values ​​of the torques of the left and right joints in that group, and calculating the ratio of the absolute value of the difference to the sum of the absolute values ​​of the torques in that group.

[0073]

[0074] Where ε is a minimal constant used to prevent division by zero errors, T L,i For the torque of the left joint, |T R,i | represents the torque at the right joint. The single-group joint balance reward value U is calculated for all symmetrical joint groups. i Summing yields the total imbalance:

[0075] Finally, this sum is multiplied by a negative weighting coefficient -λ to generate the final joint force balance reward value R. balance =-λ×U total As a feedback signal to the strategy model, it encourages the robot to learn to maintain a stable posture with balanced torques at both leg joints.

[0076] Based on any of the above embodiments, in this embodiment, the steps preceding step S10 include:

[0077] Configure environment parameters and agent parameters;

[0078] Configure the preset simulation scenario according to the environmental parameters, and construct the simulation robot model according to the agent parameters.

[0079] Specifically, the system reads and parses a predefined configuration file to obtain environmental and agent parameters. Environmental parameters include the simulation engine type, simulation time step, rendering mode, gravitational acceleration, and ground friction coefficient. Agent parameters include the robot model's file path, number of joints, joint range of motion, initial posture, and the proportional and differential gain coefficients of the PD controller.

[0080] The system initializes a preset simulation scenario based on environmental parameters, calls the API of the selected simulation engine to create a simulation environment instance, configures physical parameters, and loads scene elements. Simultaneously, it constructs a simulation robot model based on agent parameters, parses the robot model file, instantiates the model in the simulation environment, sets its initial joint positions and velocities, and configures corresponding controller parameters for each joint, thus completing the creation and initialization of the simulation robot model.

[0081] Please see Figure 2 , Figure 2This is a schematic block diagram of a robot reinforcement learning training device provided in an embodiment of this application. This robot reinforcement learning training device is used to execute the aforementioned robot reinforcement learning training method. The robot reinforcement learning training device can be configured on a server.

[0082] like Figure 2 As shown, the robot reinforcement learning training device 400 includes:

[0083] The target joint position determination module 410 is used to obtain the current observation value of the simulation robot model according to a first preset frequency in a preset simulation scenario, and determine the target joint position of the simulation robot model according to the preset strategy model and the current observation value.

[0084] The update observation acquisition module 420 is used to calculate the joint control torque according to the second preset frequency based on the target joint position and the current observation value, apply the joint control torque to the simulation robot model, and acquire and update the observation value;

[0085] The reinforcement learning training module 430 is used to generate a reinforcement training strategy based on a preset reward function and the updated observation value, so as to perform reinforcement learning training on the simulation robot model through the reinforcement training strategy.

[0086] Furthermore, the target joint position determination module 410 includes:

[0087] The current observation value determination submodule is used to collect the current joint position of the simulation robot model in the current cycle according to the first preset frequency in the preset simulation environment, and determine the current joint position as the current observation value;

[0088] The target motion determination submodule is used to determine the target motion based on the preset strategy model and the current joint position.

[0089] The target joint position determination submodule is used to determine the target joint position based on the current joint position and the target action.

[0090] Furthermore, the updated observation acquisition module 420 includes:

[0091] The joint control torque calculation submodule is used to calculate the joint control torque according to the target joint position and the current joint position at the second preset frequency.

[0092] The control instruction generation submodule is used to generate control instructions based on the target action;

[0093] The updated joint position acquisition submodule is used to apply the joint control torque to the simulation robot model in response to the control command and acquire the updated joint position of the simulation robot model.

[0094] Furthermore, the reinforcement learning training module 430 includes:

[0095] The base stability reward value generation submodule is used to generate a base stability reward value through the base stability reward function and the updated observation value;

[0096] The joint force balance reward value generation submodule is used to generate a joint force balance reward value using the joint force balance reward function and the updated observation value;

[0097] The reinforcement training strategy generation submodule is used to generate the reinforcement training strategy by weighted summation of the base stability reward value and the joint force balance reward value based on a preset ratio.

[0098] Furthermore, the base stability reward value generation submodule includes:

[0099] The projection point acquisition unit is used to acquire the projection point of the simulation robot model on the horizontal plane;

[0100] The current Euclidean distance calculation unit is used to calculate the current Euclidean distance between the projection point and the preset dynamic reference point, wherein the preset dynamic reference point is the center point of the left and right foot projections of the simulation robot model;

[0101] The base stability reward value calculation unit is used to determine the base stability reward value based on the relationship between the current Euclidean distance and the preset Euclidean distance threshold.

[0102] The first reward value determination unit is used to determine the base stability reward value as the first reward value when the current Euclidean distance is less than the preset Euclidean distance threshold.

[0103] The second reward value determination unit is used to determine the base stability reward value as the second reward value when the current Euclidean distance is greater than or equal to the preset Euclidean distance threshold.

[0104] Furthermore, the base stability reward value generation submodule includes:

[0105] The first balance reward value determination unit is used to determine the joint force balance reward value as the first balance reward value according to the joint force balance reward function when the command speed value is greater than the preset speed threshold.

[0106] The current torque value extraction unit is used to extract the current torque value of each group of symmetrical joints from the updated observation value when the command speed value is less than or equal to the preset speed threshold.

[0107] The joint force balance reward value generation unit is used to generate the joint force balance reward value based on each of the current torque values ​​and the joint force balance reward function.

[0108] Furthermore, the joint force balance reward value generation unit includes:

[0109] The single-group joint balance reward value calculation subunit is used to calculate the single-group joint balance reward value of each group of symmetrical joints using the joint force balance reward function and the current torque value of each group of symmetrical joints;

[0110] The joint force balance reward value generation subunit is used to sum all the single-group joint balance reward values ​​to obtain the joint force balance reward value.

[0111] Furthermore, the robot reinforcement learning training device 400 includes:

[0112] The parameter configuration module is used to configure environment parameters and agent parameters;

[0113] The simulation scene and model generation module is used to configure the preset simulation scene according to the environmental parameters and to construct the simulation robot model according to the agent parameters.

[0114] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 3 It runs on the computer device shown.

[0116] Please see Figure 3 , Figure 3 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0117] See Figure 3 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0118] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any robot reinforcement learning training method.

[0119] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0120] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any kind of robot reinforcement learning training method.

[0121] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0122] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0123] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0124] In a preset simulation scenario, the current observation value of the simulation robot model is obtained according to a first preset frequency. Based on the preset strategy model and the current observation value, the target joint position of the simulation robot model is determined.

[0125] The joint control torque is calculated based on the target joint position and the current observation value according to the second preset frequency, and the joint control torque is applied to the simulated robot model, and the observation value is collected and updated.

[0126] Based on the preset reward function and the updated observations, a reinforcement training strategy is generated to perform reinforcement learning training on the simulated robot model.

[0127] In one embodiment, the current observation value of the simulated robot model is obtained according to a first preset frequency in a preset simulation scenario, and the target joint position of the simulated robot model is determined according to a preset strategy model and the current observation value, for the purpose of:

[0128] In the preset simulation environment, the current joint position of the simulated robot model in the current cycle is collected according to the first preset frequency, and the current joint position is determined as the current observation value;

[0129] The target action is determined based on the preset strategy model and the current joint position;

[0130] The target joint position is determined based on the current joint position and the target motion.

[0131] In one embodiment, a joint control torque is calculated based on the target joint position and the current observation value according to a second preset frequency, and the joint control torque is applied to the simulated robot model. Updated observation values ​​are also collected to achieve the following:

[0132] According to the second preset frequency, the joint control torque is calculated based on the target joint position and the current joint position;

[0133] Based on the target action, generate control commands;

[0134] In response to the control command, the joint control torque is applied to the simulated robot model, and the updated joint position of the simulated robot model is acquired.

[0135] In one embodiment, a reinforcement training strategy is generated based on a preset reward function and the updated observations to perform reinforcement learning training on the simulated robot model, for the purpose of:

[0136] The base stability reward value is generated using the base stability reward function and the updated observation value;

[0137] The joint force balance reward value is generated using the joint force balance reward function and the updated observation value;

[0138] The reinforcement training strategy is generated by weighting and summing the base stability reward value and the joint force balance reward value based on a preset ratio.

[0139] In one embodiment, a base stability reward value is generated using the base stability reward function and the updated observation value, for the purpose of:

[0140] Obtain the projection point of the simulated robot model on the horizontal plane;

[0141] Calculate the current Euclidean distance between the projection point and the preset dynamic reference point, wherein the preset dynamic reference point is the center point of the projection of the left and right feet of the simulation robot model;

[0142] The base stability bonus value is determined based on the relationship between the current Euclidean distance and the preset Euclidean distance threshold.

[0143] If the current Euclidean distance is less than the preset Euclidean distance threshold, then the base stability bonus value is determined to be the first bonus value;

[0144] If the current Euclidean distance is greater than or equal to the preset Euclidean distance threshold, then the base stability reward value is determined to be the second reward value.

[0145] In one embodiment, a joint force balance reward value is generated using the joint force balance reward function and the updated observation value, for the purpose of:

[0146] If the command speed value is greater than a preset speed threshold, the joint force balance reward value is determined as the first balance reward value according to the joint force balance reward function;

[0147] If the commanded speed value is less than or equal to the preset speed threshold, the current torque value of each group of symmetrical joints is extracted from the updated observation value;

[0148] The joint force balance reward value is generated based on the current torque value and the joint force balance reward function.

[0149] In one embodiment, the joint force balance reward value is generated based on each of the current torque values ​​and the joint force balance reward function, for the purpose of:

[0150] The single-group joint balance reward value of each group of symmetrical joints is calculated using the joint force balance reward function and the current torque value of each group of symmetrical joints.

[0151] The joint force balance reward value is obtained by summing all the single-group joint balance reward values.

[0152] In one embodiment, before obtaining the current observation value of the simulated robot model according to a first preset frequency in a preset simulation scenario, and determining the target joint position of the simulated robot model according to a preset strategy model and the current observation value, the following is performed:

[0153] Configure environment parameters and agent parameters;

[0154] Configure the preset simulation scenario according to the environmental parameters, and construct the simulation robot model according to the agent parameters.

[0155] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the robot reinforcement learning training methods provided in the embodiments of this application.

[0156] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot reinforcement learning training method, characterized in that, include: In a preset simulation scenario, the current observation value of the simulation robot model is obtained according to a first preset frequency. Based on the preset strategy model and the current observation value, the target joint position of the simulation robot model is determined. The joint control torque is calculated based on the target joint position and the current observation value according to the second preset frequency, and the joint control torque is applied to the simulated robot model, and the observation value is collected and updated. Based on the preset reward function and the updated observation, a reinforcement training strategy is generated to perform reinforcement learning training on the simulated robot model through the reinforcement training strategy. The step of obtaining the current observation value of the simulation robot model according to a first preset frequency in a preset simulation scenario, and determining the target joint position of the simulation robot model according to a preset strategy model and the current observation value, includes: In the preset simulation environment, the current joint position of the simulated robot model in the current cycle is collected according to the first preset frequency, and the current joint position is determined as the current observation value; The target action is determined based on the preset strategy model and the current joint position; The target joint position is determined based on the current joint position and the target motion; The preset reward function includes a base stability reward function and a joint force balance reward function. The step of generating a reinforcement training strategy based on the preset reward function and the updated observations to perform reinforcement learning training on the simulated robot model includes: The base stability reward value is generated using the base stability reward function and the updated observation value; The joint force balance reward value is generated using the joint force balance reward function and the updated observation value; The reinforcement training strategy is generated by weighted summation of the base stability reward value and the joint force balance reward value based on a preset ratio. The step of generating the base stability reward value through the base stability reward function and the updated observation value includes: Obtain the projection point of the simulated robot model on the horizontal plane; Calculate the current Euclidean distance between the projection point and the preset dynamic reference point, wherein the preset dynamic reference point is the center point of the projection of the left and right feet of the simulation robot model; The base stability bonus value is determined based on the relationship between the current Euclidean distance and the preset Euclidean distance threshold. If the current Euclidean distance is less than the preset Euclidean distance threshold, then the base stability bonus value is determined to be the first bonus value; If the current Euclidean distance is greater than or equal to the preset Euclidean distance threshold, then the base stability reward value is determined to be the second reward value.

2. The robot reinforcement learning training method according to claim 1, characterized in that, The updated observations include updating joint positions. The process involves calculating joint control torque based on the target joint position and the current observations using a second preset frequency, applying the joint control torque to the simulated robot model, and collecting updated observations, including: According to the second preset frequency, the joint control torque is calculated based on the target joint position and the current joint position; Based on the target action, generate control commands; In response to the control command, the joint control torque is applied to the simulated robot model, and the updated joint position of the simulated robot model is acquired.

3. The robot reinforcement learning training method according to claim 1, characterized in that, The updated observations include commanded velocity values. Generating joint force balance reward values ​​using the joint force balance reward function and the updated observations includes: If the command speed value is greater than a preset speed threshold, the joint force balance reward value is determined as the first balance reward value according to the joint force balance reward function; If the commanded speed value is less than or equal to the preset speed threshold, the current torque value of each group of symmetrical joints is extracted from the updated observation value; The joint force balance reward value is generated based on the current torque value and the joint force balance reward function.

4. The robot reinforcement learning training method according to claim 3, characterized in that, The step of generating the joint force balance reward value based on each of the current torque values ​​and the joint force balance reward function includes: The single-group joint balance reward value of each group of symmetrical joints is calculated using the joint force balance reward function and the current torque value of each group of symmetrical joints. The joint force balance reward value is obtained by summing all the single-group joint balance reward values.

5. The robot reinforcement learning training method according to any one of claims 1 to 4, characterized in that, Before obtaining the current observation value of the simulation robot model according to a first preset frequency in a preset simulation scenario, and determining the target joint position of the simulation robot model according to a preset strategy model and the current observation value, the process includes: Configure environment parameters and agent parameters; Configure the preset simulation scenario according to the environmental parameters, and construct the simulation robot model according to the agent parameters.

6. A robot reinforcement learning training device, characterized in that, include: The target joint position determination module is used to obtain the current observation value of the simulation robot model according to a first preset frequency in a preset simulation scenario, and determine the target joint position of the simulation robot model according to a preset strategy model and the current observation value. The observation acquisition module is used to calculate the joint control torque based on the target joint position and the current observation value according to a second preset frequency, apply the joint control torque to the simulated robot model, and acquire and update the observation value. The reinforcement learning training module is used to generate a reinforcement training strategy based on a preset reward function and the updated observation value, so as to perform reinforcement learning training on the simulation robot model through the reinforcement training strategy. The target joint position determination module includes: The current observation value determination submodule is used to collect the current joint position of the simulation robot model in the current cycle according to the first preset frequency in the preset simulation environment, and determine the current joint position as the current observation value; The target motion determination submodule is used to determine the target motion based on the preset strategy model and the current joint position. The target joint position determination submodule is used to determine the target joint position based on the current joint position and the target action; The reinforcement learning training module includes: The base stability reward value generation submodule is used to generate a base stability reward value through the base stability reward function and the updated observation value; The joint force balance reward value generation submodule is used to generate a joint force balance reward value using the joint force balance reward function and the updated observation value; The reinforcement training strategy generation submodule is used to generate the reinforcement training strategy by weighted summing of the base stability reward value and the joint force balance reward value based on a preset ratio. The base stability reward value generation submodule includes: The projection point acquisition unit is used to acquire the projection point of the simulation robot model on the horizontal plane; The current Euclidean distance calculation unit is used to calculate the current Euclidean distance between the projection point and the preset dynamic reference point, wherein the preset dynamic reference point is the center point of the left and right foot projections of the simulation robot model; The base stability reward value calculation unit is used to determine the base stability reward value based on the relationship between the current Euclidean distance and the preset Euclidean distance threshold. The first reward value determination unit is used to determine the base stability reward value as the first reward value when the current Euclidean distance is less than the preset Euclidean distance threshold. The second reward value determination unit is used to determine the base stability reward value as the second reward value when the current Euclidean distance is greater than or equal to the preset Euclidean distance threshold.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the robot reinforcement learning training method as described in any one of claims 1 to 5.

Images