Method and system for controlling embodied agent based on reward function of optimized reinforcement learning

Abstract

The application belongs to the technical field of robots, and discloses a method and system for controlling embodied agents by using a reward function based on optimized reinforcement learning. An interaction trajectory is formed by using the state and control instruction of the embodied agent, a plurality of random interaction trajectories are obtained by randomly sampling the interaction trajectories of a plurality of gaits of the embodied robot, the gradient of the reward function is calculated by using the plurality of random interaction trajectories, and the reward value of the reward function is updated by using the gradient; the updated reward value and the current state of the embodied agent are input into a policy network of robot actions to obtain the current control instruction of the robot. Through the application, the problem of low control precision of the agent caused by the fact that the reward function cannot be automatically updated is solved.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, and more specifically, relates to a method and system for controlling an embodied intelligent agent based on a reward function of optimized reinforcement learning. Background Technology

[0002] In the field of robotics, particularly for humanoid robots with high-dimensional state spaces and nonlinear dynamics, achieving stable and efficient motion control (such as walking and running) is a core challenge. Reinforcement learning provides a powerful tool for solving such problems by allowing robots to learn through extensive trial and error in a physical simulation environment. However, the success of reinforcement learning is heavily reliant on a carefully designed reward function to guide the robot in learning the desired behavior. Currently, the design of reward functions is primarily manual. For example, a reward function for training a robot to walk typically requires combining multiple sub-items such as forward speed, trunk stability, energy consumption, and gait symmetry, and manually tuning their weights. This process is not only extremely time-consuming, but the resulting reward function is often suboptimal, leading to a trained robot with stiff gait, high energy consumption, and poor disturbance resistance.

[0003] To address this issue, while methods such as inverse reinforcement learning attempt to learn reward functions from expert demonstrations, obtaining high-quality expert demonstration data covering various scenarios (e.g., human-telescopic robot walking) is extremely costly and impractical in the robotics field. Existing technologies generally employ a fixed, manually set reward function for policy training, which severely limits the robot's autonomous learning ability and performance ceiling when facing complex terrain or variable tasks. Therefore, a method that can solve these problems is urgently needed. Summary of the Invention

[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for controlling embodied agents based on reward functions derived from optimized reinforcement learning, thereby solving the problem of low agent control accuracy caused by the inability to automatically update reward functions.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for controlling an embodied agent based on a reward function of optimized reinforcement learning is provided, the method comprising the following steps:

[0006] The interaction trajectory is formed by using the state and control commands of the embodied intelligent agent. Random sampling is performed on the interaction trajectory of multiple gaits of the embodied robot to obtain multiple random interaction trajectories. The gradient of the reward function is calculated using these multiple random interaction trajectories, and the reward value of the reward function is updated using the gradient.

[0007] The updated reward value and the current state of the embodied agent are input into the policy network for the robot's actions to obtain the robot's current control commands.

[0008] More preferably, the steps for calculating the gradient of the reward function are as follows:

[0009] (1) Randomly sample a portion of the trajectory from multiple random interactive trajectories;

[0010] (2) Based on the sampled partial trajectory simulation strategy distribution And calculate the advantage function. and The value;

[0011] (3) Calculate the reward gradient based on the simulated policy distribution and the value of the advantage function.

[0012] More preferably, the gradient of the reward function is calculated using the following formula:

[0013]

[0014] in, To optimize the objective function of the upper layer, It is a policy distribution Expectations This indicates that the value within [] is calculated. The derivative, These are the parameters of the reward function. Given a pre-defined base reward function and policy network, the advantage function is... Given the current reward function and policy network to be optimized, the advantage function is... It is the gradient of the reward function.

[0015] More preferably, the reward value of the updated reward function is calculated according to the following formula:

[0016] ω=ω+

[0017] Where ω is the reward function value, It is the gradient of the reward function.

[0018] More preferably, the reward function is a deep neural network, the input of which is the state and control instructions of the embodied agent, and the output is the reward function value.

[0019] More preferably, the reward function includes a state encoder, an action encoder, and a forward model, wherein the input of the state encoder is the state of the embodied agent, and the output is a state embedding vector; the input of the action encoder is a control command, and the output is an action embedding vector; the input of the forward model is the state embedding vector and the action embedding vector, and the output is the reward function value.

[0020] More preferably, the method is repeated multiple times until the reward function value converges.

[0021] More preferably, the state of the embodied intelligent agent includes the joint angles, angular velocities, and torso posture of the embodied intelligent agent.

[0022] According to another aspect of the present invention, a system for controlling an embodied agent based on a reward function of optimized reinforcement learning is provided. The system includes an actuator for performing the method for controlling an embodied agent based on a reward function of optimized reinforcement learning as described above.

[0023] According to another aspect of the present invention, a computer storage medium is provided having a computer program stored thereon, which, when executed, is used to implement the above-described method for controlling an embodied agent based on a reward function of optimized reinforcement learning.

[0024] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0025] 1. In this invention, the gradient function is updated in real time by using the gradient of the reward function, thereby realizing the real-time update of the reward function and the control instructions of the embodied agent. This eliminates the dependence on domain experts to manually design and debug the robot reward function, greatly reducing the technical threshold and R&D cost of applying reinforcement learning to complex robot control tasks, and solving the problem of low control accuracy of the agent caused by the inability of the reward function to be automatically updated.

[0026] 2. The policy network of the present invention is used to control the interaction between the robot and the working environment in the lower-level policy optimization. Its function is to control the robot to interact with the environment to generate a trajectory, so as to calculate the gradient of the reward function and update the reward function parameters, while ensuring the implementation of the lower-level policy optimization.

[0027] 3. The reward function of this invention adopts a deep neural network structure with adjustable parameters, replacing the original structure that directly sets the function form and parameters, ensuring that the form and parameters of the reward function can be updated in real time by updating the gradient function.

[0028] 4. This invention replaces tedious and inefficient manual debugging by automatically discovering the optimal reward function that is highly matched with the walking task. This can significantly improve the robot's policy learning speed and reduce the number of falls during walking, thereby enhancing the robot's autonomous control capability and generalization performance in diverse scenarios. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall framework of the automatic discovery method for the optimal reward function of embodied intelligent agents constructed according to a preferred embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the reward function neural network structure constructed according to a preferred embodiment of the present invention, wherein (a) is the overall architecture diagram of the reward function, including a state encoder, an action encoder and a forward model, (b) is the specific structure of the state encoder and the action encoder, which is a deep neural network, and (c) is the specific structure of the forward model, which is a deep neural network.

[0031] Figure 3 This is a performance comparison chart constructed according to a preferred embodiment of the present invention in different simulation environments, wherein (a) is the one used in the embodiment. humanoid-v1 The robot diagram is shown in (b), which is a line graph of the reward function of the scheme tested in a simulated environment when the PPO method is used as the lower-level policy network, and (c) is the performance evaluation result of the scheme transferred to a real robot. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0033] like Figure 1 As shown, an automatic discovery method for the optimal reward function in robot motion control is presented. This method constructs a two-layer optimization control framework to automatically optimize the reward function while the robot learns control strategies for performing physical tasks. This embodiment focuses on controlling a humanoid robot in… MuJoCo Taking learning stable walking in a physical simulation environment as an example, this method mainly includes the following:

[0034] Upper-level reward optimization: The task execution trajectory generated by lower-level interactions will be optimized. Store in trajectory buffer It samples small batches of historical state-action data from the buffer and estimates the upper-level optimization objective based on this data. The gradient of the network parameter ω in the reward function This gradient is based on the advantage function. An approximate calculation is performed to evaluate the performance relative to the average when a certain joint torque is applied in a given robot state. The formula for updating the reward function using gradients is as follows:

[0035]

[0036]

[0037] Lower-level strategy optimization: In a system with parameters The current reward function network is determined. Under the guidance of [unclear], the robot interacts with its simulated working environment to generate an interactive trajectory that includes continuous sensor readings such as joint angles, angular velocities, and torso posture, as well as target torque control commands for each joint. .

[0038] Based on the updated reward function network The calculated reward signal, along with the robot's current state, is input into the robot's action policy network. In this process, the current control commands for the robot are obtained to optimize its walking gait. This includes the robot's action policy network. The inputs are sensor readings and the output values ​​of the reward function network, and the output is the control command.

[0039] Lower-level policy optimization employs reinforcement learning algorithms (such as PPO), aiming to optimize the dynamic reward function generated by the upper-level modules. Under the guidance of [the relevant authorities], by continuously adjusting the strategy network The parameters are set to maximize the cumulative reward obtained by the robot within a task cycle, thereby learning the optimal motion control strategy.

[0040] Alternating Updates: This is achieved by alternately executing lower-level policy optimization and upper-level reward optimization in the policy network. and reward function network The robot undergoes co-evolution until its walking performance converges. Ultimately, the optimal control strategy and reward function for this walking task are obtained.

[0041] From the moment the robot stands up until it falls or reaches its maximum distance, a round is completed. During the interaction, the lower-level control strategy is continuously optimized (lower-level loop). After the round ends, the system performs an upper-level reward function optimization (upper-level loop) based on the overall performance of the round (such as total walking distance, average speed, posture stability, etc.) in order to provide better learning guidance in the next task round.

[0042] This embodiment fundamentally solves the problem of relying on manually designed reward functions when reinforcement learning is applied to complex physical systems such as robots by constructing a two-layer optimization framework. Its core value lies in formalizing the problem of finding the optimal reward function into a solvable optimization problem aimed at maximizing the robot's task performance. This framework dynamically and adaptively generates reward signals that match the robot's current learning stage and task difficulty. Especially in scenarios with sparse rewards, such as the initial learning stage of the robot, or complex task scenarios, it can significantly accelerate the learning process and discover optimal strategies that surpass human intuition.

[0043] In some embodiments, the reward function This is a deep neural network model whose input is the robot's state. and actions Among them, state It is a high-dimensional vector containing information such as the angles and angular velocities of all the robot's joints, the posture (quaternion) and linear velocity of the torso, and the readings of the plantar pressure sensors; motion. These are the 17 torque vectors that drive the robot. Through its internal encoder and forward propagation structure, the network ultimately outputs a scalar reward value to evaluate the quality of this "state-action" relationship. .

[0044] Specifically, please refer to Figure 2 In the diagram, state St represents the sensor reading, and action at represents the control command, such as... Figure 2 As shown in (a), the specific structure of the reward function includes a state encoder, an action encoder, and a forward model, as detailed below:

[0045] State encoder: such as Figure 2 (b) is an independent neural network that receives state information from the robot’s simulated environment (angles and angular velocities of 22 joints and position, posture and velocity of the torso) as input and encodes it into a high-dimensional state embedding vector through a multi-layer fully connected network or other network structure.

[0046] Motion encoder: such as Figure 2 (b) in the diagram is a separate neural network, the receiver policy network. The output torque values ​​applied to the robot's 17 movable joints are used as input and encoded into high-dimensional motion embedding vectors.

[0047] Forward model: such as Figure 2 (c) is a fusion network that concatenates the state embedding vector and action embedding vector mentioned above, and then processes them through subsequent neural network layers to finally output a scalar value, which is the reward for the current state-action pair.

[0048] In some embodiments, to achieve stable training, the entire process is performed in an alternating manner. In a complete walking task cycle, the humanoid robot starts walking from its starting position until it falls or reaches a preset time limit; the trajectory of the entire cycle is then stored in a trajectory buffer. Subsequently, from Using mid-sampled data, perform one upper-level reward optimization on the reward function network. Update.

[0049] In other embodiments, an automatic discovery system for optimal reward functions for robot control is also provided. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the robot control method described above. For example, the system can be a robot controller integrating a GPU or a dedicated AI acceleration chip, a high-performance computing server for robot simulation training, or an autonomous driving domain controller deploying the algorithm of this invention. The memory contains program modules implementing a two-layer optimization framework, including a policy network module, a reward function network module, and a control logic module for scheduling lower and upper-layer optimizations. During runtime, the processor loads these modules and executes the method by interacting with the robot simulation environment (whether real or simulated), ultimately outputting control commands to drive robot movement or obtaining a trained policy and reward model.

[0050] The present invention will be further described below with reference to specific embodiments.

[0051] This embodiment deploys the automatic discovery method for the optimal reward function of the present invention in a high-dimensional and complex humanoid robot walking task, and verifies it on both physical simulation and real physical prototypes.

[0052] 1. Simulation Environment Verification

[0053] This embodiment firstly... MuJoCo In a physical simulation environment Humanoid-v1 Robots (such as) Figure 3 The model is a humanoid robot with 17 driven joints, and its mission objective is to learn to walk forward quickly and stably.

[0054] The two-layer optimization framework described in this invention is constructed. In the lower-layer policy optimization, the robot's policy network uses the policy network from the PPO algorithm, with both its policy and value function networks being three-layer fully connected structures (256-128-64 neurons). The input state to the policy network is a 376-dimensional vector, containing the linear and angular velocities of the robot's root, the relative postures of different body parts, the velocities of different body parts, and sensor observation information. The output action of the policy network is the torque values ​​of 17 controllable joints. The reward function network structure in the upper-layer reward optimization is as follows... Figure 2 As shown, both the state encoder and action encoder use a 3-layer deep neural network with an output dimension of 256 and a hidden layer dimension of 128. The feedforward model uses a 4-layer deep neural network with an input dimension of 512, an output dimension of 1, and hidden layer dimensions of 256, 128, and 64, respectively. In this embodiment, the original reward function designed by experts and provided by the environment is not used. Instead, the upper-layer optimization module automatically discovers the optimal reward function based solely on the sparsest final signal (the total forward distance at the end of the round). Specifically, in the lower-layer policy optimization, the embodied agent interacts with the working environment 2048 times, and in the upper-layer reward optimization, 75% of the trajectories are randomly sampled to simulate the policy distribution and calculate the value of the advantage function. The upper-layer reward optimization and lower-layer policy optimization alternate for a total of 1000 iterations.

[0055] During training, the upper-level optimization module automatically learned and generated a dynamic, dense reward function. Analysis revealed that this function assigns high rewards to states consistent with efficient walking dynamics and penalizes unstable or ineffective actions (such as jumping or excessive body tilting). Figure 3 As shown in Figure (b), the robot using the method of this invention (red curve) significantly outperforms the baseline PPO algorithm (blue curve) using expert design rewards in terms of learning speed, final walking speed, and gait stability.

[0056] 2. Physical prototype migration verification

[0057] To verify the effectiveness of the invention in the real world, the model trained in the simulation was deployed on a self-developed, 1.6-meter-tall, 17-DOF humanoid robot physical prototype.

[0058] Considering the differences between simulation and reality, an online fine-tuning strategy was adopted: on the physical prototype, the optimal reward function network already learned in the simulation was frozen, and only the lower-level policy network was subjected to short-term online learning. The robot walks on a platform equipped with a safety gantry, and its real-time status is acquired through an onboard IMU and joint encoders.

[0059] like Figure 3Experiment (c) shows that after approximately 2 hours of online fine-tuning, the robot can detach from the safety harness and achieve continuous and stable walking of more than 50 steps on a flat surface. In contrast, the model trained using the direct baseline transfer method still cannot walk stably after the same amount of fine-tuning time. This demonstrates that the reward function learned in this invention has stronger robustness and generalization ability, effectively guiding the robot to quickly adapt to and complete tasks in the real physical world, showcasing significant practical application value.

[0060] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for controlling a body-equipped agent based on an optimized reinforcement learning reward function, characterized by, The method includes the following steps: The interaction trajectory is formed by using the state and control commands of the embodied agent. Random sampling is performed on the interaction trajectory of multiple gaits of the embodied agent to obtain multiple random interaction trajectories. The gradient of the reward function is calculated using these multiple random interaction trajectories, and the reward value of the reward function is updated using the gradient. The updated reward value and the current state of the embodied agent are input into the policy network of the robot's actions to obtain the robot's current control commands; The steps for calculating the gradient of the reward function are as follows: (1) Randomly sample a portion of the trajectory from multiple random interactive trajectories; (2) Simulate the policy distribution from the sampled trajectories and compute the value of the advantage function and . (3) Calculate the reward gradient based on the simulated policy distribution and the value of the advantage function; The formula for calculating the gradient of the reward function is as follows: in, To optimize the objective function of the upper layer, It is a policy distribution Expectations This indicates that the value within [] is calculated. The derivative of It is the reward value of the reward function. Given a pre-defined base reward function and policy network, the advantage function is... Given the current reward function and policy network to be optimized, the advantage function is... It is the gradient of the reward function.

2. The method of claim 1, wherein the reward function is defined as: R = ∑i=1n (Ri - Ravg)2, wherein Ravg = 1 T ∑i=1n Ri, and wherein Ravg is an average reward value of the reward function. The reward value of the updated reward function is calculated according to the following formula: ω = ω+ where ω is the reward value of the reward function, is the gradient of the reward function.

3. The method for controlling an embodied agent based on a reward function derived from optimization reinforcement learning as described in claim 1, characterized in that, The reward function is a deep neural network. The input to the reward function is the state and control instructions of the embodied agent, and the output is the reward value of the reward function.

4. The method for controlling an embodied agent based on a reward function of optimized reinforcement learning as described in claim 3, characterized in that, The reward function includes a state encoder, an action encoder, and a forward model. The state encoder takes the state of the embodied agent as input and outputs a state embedding vector. The action encoder takes control commands as input and outputs an action embedding vector. The forward model takes the state embedding vector and the action embedding vector as input and outputs the reward value of the reward function.

5. The method for controlling an embodied agent based on a reward function derived from optimization reinforcement learning as described in claim 1, characterized in that, Repeat the above method multiple times until the reward value of the reward function converges.

6. The method for controlling an embodied agent based on a reward function of optimized reinforcement learning as described in claim 1, characterized in that, The state of the embodied intelligent agent includes the joint angles, angular velocities, and torso posture of the embodied intelligent agent.

7. A system for controlling an embodied intelligent agent based on a reward function of optimized reinforcement learning, characterized in that, The system includes an actuator for performing a method for controlling an embodied agent based on a reward function of optimized reinforcement learning, as described in any one of claims 1-6.

8. A computer storage medium having stored thereon a computer program, characterized in that When the computer program is executed, it is used to implement the method for controlling an embodied agent based on a reward function of optimized reinforcement learning as described in any one of claims 1-6.

Images