Humanoid robot motion control method based on structure topology perception and decoupling value estimation

Abstract

This invention belongs to the field of intelligent motion control technology for humanoid robots, specifically involving a motion control method for humanoid robots based on structural topology perception and decoupled value estimation. By constructing a kinematic adjacency matrix that conforms to the robot's specific physical shape and symmetry characteristics, and transforming it into a topology-aware attention mask for a Transformer backbone network, the invention guides features to propagate and interact only among neighboring joint nodes with explicit kinematic coupling and mechanical linkages, thus achieving explicit modeling of spatial structure and kinematic chain constraints. Simultaneously, this invention reorganizes the multi-objective optimization problem of dynamically stable humanoid robot motion into four semantically independent heterogeneous reward categories: command tracking, whole-body stability, motion regularization, and gait shaping. Four independent value network branches are configured for decoupled value fitting. By independently calculating the sub-dominance functions of each channel and weighted fusion, the gradient conflict and backpropagation interference of heterogeneous signals in multi-objective optimization are mitigated.

Description

Technical Field

[0001] This invention belongs to the field of intelligent motion control technology for humanoid robots, specifically relating to a motion control method for humanoid robots based on structural topology perception and decoupling value estimation. Background Technology

[0002] With the development of embodied intelligence, humanoid robots have become a key industrial focus. Due to their hybrid dynamic characteristics, such as high-dimensional degrees of freedom, strong nonlinearity, and severe underactuation, the autonomous motion control of humanoid robots presents significant challenges. Traditional model predictive control methods heavily rely on accurate analytical physics models. When faced with unknown disturbances or complex unstructured terrain, these methods are highly susceptible to model mismatch, leading to system instability and failure. Consequently, the robustness and cross-environment generalization ability of these algorithms are severely limited.

[0003] While model-free deep reinforcement learning alleviates the reliance on analytical models through simulation data-driven approaches, existing control policy networks typically flatten joint states and sensor commands into a one-dimensional vector input, completely severing the robot's inherent body topology and kinematic chain hierarchical constraints, resulting in low feature extraction efficiency. Furthermore, humanoid robot walking is a typical example of coupled multi-objective optimization. Traditional reinforcement learning uses manually adjusted weights to forcibly scalarize and merge heterogeneous reward terms such as command tracking, overall stability, and energy economy into a single scalar reward. This leads to severe gradient conflicts during backpropagation of learning signals of different polarities, making the trained policy prone to violent motion oscillations and difficult to converge smoothly. Therefore, this invention provides a humanoid robot motion control method based on structural topology perception and decoupled value estimation. Summary of the Invention

[0004] The purpose of this invention is to provide a motion control method for humanoid robots based on structural topology perception and decoupling value estimation, which can significantly improve the control convergence speed, command tracking accuracy and motion output smoothness of bipedal humanoid robots, and has excellent cross-physics engine transfer and generalization robustness.

[0005] The specific technical solution adopted by this invention is as follows: A humanoid robot motion control method based on structural topology perception and decoupling value estimation is implemented using a hierarchical reinforcement learning framework. This framework includes a high-level motion planner based on reinforcement learning and a low-level joint proportional differential control feedback loop. The specific control process includes the following steps: Step 1: Define a partially observable Markov decision process model for the humanoid robot in a reinforcement learning environment, and construct the direct observation space of the onboard sensors corresponding to the Actor policy network. And the simulation privileged observation space corresponding to the Critic value network, which includes the actual linear velocity of the base. The high-level motion planner receives the observed state at a set first frequency and generates the target angle increments for the 12 drive joints of the lower limbs. It then hands them over to the low-level joint proportional differential controller, which operates at a higher second frequency, to output the final joint physical torque to drive the humanoid robot to walk. Step 2: By abstracting the lower limb joints and torso center of the humanoid robot as nodes in a graph structure, the physical properties and spatial constraints of each kinematic chain of the robot are explicitly transformed into a morphological adjacency matrix containing self-loops. And use this morphological adjacency matrix to build a binary hard topological attention mask. ; the hard topology attention mask The Transformer network self-attention computation layer injected into the high-level planner uses negative infinite weights to suppress and block the disordered diffusion of information between non-mechanically connected joint nodes, prompting the network to explicitly extract the dynamic spatial coupling features between lower limb joint links. Step 3: Decompose the overall control objective of the humanoid robot's walking into four semantically independent heterogeneous reward category sets, namely: the instruction tracking reward set for following the external translational turning velocity objective. A collection of rewards for maintaining trunk uprightness and preventing falls. Motion regularization reward set used to save energy and ensure smooth joint output. , and the gait shaping reward set used to regulate the desired gait phase ; Step 4: During the update and training phase of the reinforcement learning network, four completely independent Critic value networks are configured to specifically fit the expected state reward values ​​of the four semantic reward categories defined in Step 3; when calculating the policy dominance function, the sub-dominance values ​​of the four channels are calculated based on the independent evaluation results of each value network. The total advantage estimate is obtained by summing and accumulating the sub-advantage values ​​of each channel to guide the policy network parameters for near-end policy optimization (PPO) updates.

[0006] Preferably, the humanoid robot morphological adjacency matrix defined in step 2 In the Transformer network, the hip, knee, and ankle joint nodes on one side of the humanoid robot are configured to interact fully within a group, and the central node of the torso is bidirectionally connected to the hip joint groups of both thighs, thereby preserving physical symmetry constraints in the Transformer network.

[0007] Preferably, in step 4, when updating the parameters of the policy network and the value network, a learning rate control strategy based on adaptive update magnitude is adopted to solve the Kullback-Leibler divergence value between the old and new policies in real time. If the divergence value exceeds the set upper limit safety threshold, the current global parameter iteration optimization step size is automatically reduced proportionally.

[0008] The technical effects achieved by this invention are as follows: The control architecture of this invention consists of a high-level reinforcement learning motion planner and a high-frequency, low-level proportional-differential (PD) joint tracking controller. In the high-level planner, the invention abstracts the lower limb joints and torso center of the humanoid robot as graph-structured nodes. By constructing a kinematic adjacency matrix that conforms to its specific physical morphology and symmetry characteristics, and transforming it into a topology-aware attention mask for the Transformer backbone network, features are guided to propagate and interact only among neighboring joint nodes with explicit kinematic coupling and mechanical linkages, thus achieving explicit modeling of spatial structure and kinematic chain constraints. Simultaneously, this invention reorganizes the multi-objective optimization problem of dynamically stable humanoid robot motion into four semantically independent heterogeneous reward categories: command tracking, whole-body stability, motion regularization, and gait shaping. Four independent value network branches are configured for decoupled value fitting. By independently calculating the sub-dominance functions of each channel and weighted fusion, the gradient conflict and backpropagation interference of heterogeneous signals in multi-objective optimization are effectively mitigated. Experiments show that this method can significantly improve the control convergence speed, command tracking accuracy, and motion output smoothness of bipedal humanoid robots, and has excellent robustness in cross-physics engine transfer and generalization. Attached Figure Description

[0009] Figure 1 This is the average reward curve of the reinforcement learning training process of this invention; Figure 2 This is a speed tracking curve of the base during the reasoning process of this invention. Detailed Implementation

[0010] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0011] like Figure 1 As shown, a humanoid robot motion control method based on structural topology perception and decoupled value estimation is implemented using a hierarchical reinforcement learning framework. This framework includes a high-level motion planner based on reinforcement learning and a low-level joint proportional differential control feedback loop. The specific control process includes the following steps: Step 1: Define a partially observable Markov decision process model for the humanoid robot in a reinforcement learning environment, and construct the direct observation space of the onboard sensors corresponding to the Actor policy network. And the simulation privileged observation space corresponding to the Critic value network, which includes the actual linear velocity of the base. The high-level motion planner receives the observed state at a set first frequency and generates the target angle increments for the 12 drive joints of the lower limbs. It then hands them over to the low-level joint proportional differential controller, which operates at a higher second frequency, to output the final joint physical torque to drive the humanoid robot to walk. Step 2: By abstracting the lower limb joints and torso center of the humanoid robot as nodes in a graph structure, the physical properties and spatial constraints of each kinematic chain of the robot are explicitly transformed into a morphological adjacency matrix containing self-loops. And use this morphological adjacency matrix to build a binary hard topological attention mask. ; the hard topology attention mask The Transformer network self-attention computation layer injected into the high-level planner uses negative infinite weights to suppress and block the disordered diffusion of information between non-mechanically connected joint nodes, prompting the network to explicitly extract the dynamic spatial coupling features between lower limb joint links. Step 3: Decompose the overall control objective of the humanoid robot's walking into four semantically independent heterogeneous reward category sets, namely: the instruction tracking reward set for following the external translational turning velocity objective. A collection of rewards for maintaining trunk uprightness and preventing falls. Motion regularization reward set used to save energy and ensure smooth joint output. , and the gait shaping reward set used to regulate the desired gait phase ; Step 4: During the update and training phase of the reinforcement learning network, four completely independent Critic value networks are configured to specifically fit the expected state reward values ​​of the four semantic reward categories defined in Step 3; when calculating the policy dominance function, the sub-dominance values ​​of the four channels are calculated based on the independent evaluation results of each value network. The total advantage estimate is obtained by summing and accumulating the sub-advantage values ​​of each channel to guide the policy network parameters for near-end policy optimization (PPO) updates.

[0012] Preferably, the humanoid robot morphological adjacency matrix defined in step 2 In the Transformer network, the hip, knee, and ankle joint nodes on one side of the humanoid robot are configured to interact fully within a group, and the central node of the torso is bidirectionally connected to the hip joint groups of both thighs, thereby preserving physical symmetry constraints in the Transformer network.

[0013] Preferably, in step 4, when updating the parameters of the policy network and the value network, a learning rate control strategy based on adaptive update magnitude is adopted to solve the Kullback-Leibler divergence value between the old and new policies in real time. If the divergence value exceeds the set upper limit safety threshold, the current global parameter iteration optimization step size is automatically reduced proportionally.

[0014] In its specific implementation, this invention includes the following steps: Step 1: State space definition and hierarchical control architecture design.

[0015] The motion control problem of humanoid robots is modeled as a partially observable Markov decision process (POMDP), denoted as a six-tuple. ,in For the system's hidden state space, The observation space available to the strategy. For the action space, Here is the state transition matrix. For the reward function, This is the discount factor.

[0016] An asymmetric Actor-Critic architecture is adopted, meaning the policy network only receives ordinary observations that can be directly acquired by the sensors of the real robot payload. The Critic value network, during the training phase, has additional privileged access to simulator-side observations. Action space This corresponds to the 12 lower limb drive joints of the bipedal humanoid robot. The specific observation space and privileged observation space of the system are shown in Table 1 below.

[0017]

[0018] The entire control system operates on a two-layer hierarchical control architecture, consisting of a low-frequency high-level motion planner and a high-frequency low-level joint controller cascaded together. The high-level motion planner, operating as a reinforcement learning policy network at 50Hz, is responsible for periodically receiving the current ordinary observation vector and, in each planning cycle, executing the following formula... The system generates and outputs target angle position reference signals for the 12 physical joints of the lower limbs. The low-level joint controller, operating as a proportional-differential (PD) controller at 500Hz, directly takes the target joint positions from the high-level planner and the robot's current actual joint angle states as input. It tracks joint positions in real time through a high-speed physical feedback loop, calculates and outputs the final joint-level driving torque command. This design utilizes a low-level, high-frequency feedback loop to alleviate the direct torque output pressure on the reinforcement learning network, significantly improving the overall walking stability of the humanoid robot.

[0019] Step 2: Construction of body structure topology graph and topology-aware attention masking mechanism.

[0020] To explicitly capture the spatial kinematic coupling and bilateral symmetrical coordination between linkages in a humanoid robot, this invention abstracts the robot's lower limb joints and torso into a graph structure with explicit topological relationships. In this graph structure, the system constructs a token sequence containing one global node and twelve local joint nodes. The global node... Located at the center of the torso. For the Actor, its feature encoding is a combined vector containing the base angular velocity, projected gravity, velocity command, and periodic gait descriptor. ; This represents the angular velocity of the base in three directions. Represents the gravity projection vector. This indicates a speed tracking command in three directions. It is a two-dimensional vector representing the gait cycle; for Critic, its feature encoding additionally incorporates the true linear velocity of the base, i.e., written as . Represents the linear velocity of the base in three directions; local nodes to Each of the robot's 12 lower limb drive joints is represented by a unique feature encoding for each local joint node, which is a vector combining the current joint position, current joint velocity, and historical motion. Before extracting the backbone from the input features, the original features of all nodes are first projected onto a latent space of uniform dimension through a locally learnable linear mapping layer, thereby forming dimension-aligned initial input sequences of tokens on both the Actor and Critic sides, as shown in the following mathematical expressions: ; in, This represents the input at step t of the i-th node in the strategy branch. This represents the hidden state of the i-th node at step t in the policy branch. and Represents the weight matrix and bias vector mapped to the i-th node of the policy branch; This represents the input at step t of the i-th node in the value branch. Let represent the hidden state of the i-th node at step t in the value branch. and This represents the weight matrix and bias vector mapped to the i-th node of the value branch.

[0021] The resulting input sequence as well as .

[0022] in, Let represent the hidden states of each node in the t-th step of the strategy branch. This represents the hidden state after the policy branch mapping; Let represent the hidden states of each node in the t-th step of the value branch. This represents the hidden state after the value branch mapping.

[0023] In the feature fusion stage, in order to overcome the neglect of the robot's physical structure by traditional one-dimensional flat splicing input, the system explicitly defines a self-loop adjacency matrix that fits the robot's kinematic constraints, based on the specific multi-joint linkage shape of the humanoid robot. Here, A is the adjacency matrix constructed based on the joint connections of the humanoid robot, and I is the identity matrix. In this adjacency matrix, the hip joint groups, knee joint nodes, and ankle joint nodes of a single leg are set to have fully connected interactions within the group, and the center node of the torso is bidirectionally connected to the hip joint groups of both thighs, thus explicitly preserving physical symmetry in the graph structure. Using this adjacency matrix, the system further constructs a binarized topological attention mask. If the corresponding elements of the adjacency matrix Then mask element If the corresponding elements of the adjacency matrix Then mask element Subsequently, this hard topology mask is directly injected into the Transformer self-attention mechanism computation layer of the high-level planner. The mathematical form of its mask attention mechanism is as follows: ; Where Q, K, and V represent the query, key, and value feature matrices, respectively; d is the feature dimension used to scale the attention score; M is the attention bias or mask matrix for injecting joint topological relationships; and softmax is used to obtain normalized attention weights and perform weighted fusion on V.

[0024] By introducing negative infinity weights into the Softmax calculation term This invention forcibly severs the attention weight scores between joint nodes that are not physically connected in the mechanical structure, and forces the feature flow to propagate features and exchange information only between neighboring joint nodes that have physical connections or strong kinematic coupling, thereby realizing explicit modeling of multi-link spatial structures and kinematic chain hierarchical constraints.

[0025] Step 3: Design of the semantic decoupling four-class reward function.

[0026] To fundamentally eliminate optimization interference between heterogeneous sub-objectives during humanoid robot walking, this invention completely abandons the traditional approach of linearly combining all reward terms into a single scalar reward. Instead, it reorganizes the dense rewards during the reinforcement learning training phase and uses the formula... The rewards are divided into four semantically independent heterogeneous reward categories. The specific subcategories and mathematical expressions are shown in Table 2 below:

[0027] The reward functions of these four categories are independently input into their respective evaluation channels during training, which fundamentally isolates the mutual interference of optimization directions caused by the different dimensions and polarities of different control terms, and provides a solid joint data foundation for the subsequent multi-crit independent decoupled evaluation architecture.

[0028] Step 4: Iterative update of the multi-critic decoupling optimization system and strategy network.

[0029] Based on the four independent reward categories defined in step three, the training system of this invention is configured with four structurally independent value networks, each with different parameters. This is expressed as follows: Each value network is specifically responsible for fitting the expected cumulative return value of a particular return category, and its value function mapping can be uniformly represented as: ; During the parameter update phase of the policy network, the system employs the Proximal Policy Optimization (PPO) algorithm. The policy network uses a shearing loss function. Dependence on total advantage estimate The system updates the overall advantage estimate, which is then completely decomposed into four independent channels. For each value network channel, the system first calculates the category-specific temporal difference error based on the real-time reward of the current category, the next state, and the value assessment of the current state. Subsequently, this error is used to independently generate sub-dominance estimates for each channel through the Generalized Dominance Estimation (GAE) mechanism. The specific mathematical recursive formula is as follows: ; in, This represents the advantage estimate of the k-th evaluation channel at time t, used to measure the superiority or inferiority of the current action relative to the average level of that channel; k represents the evaluation channel number. In this invention, k = 1, 2, 3, 4, which correspond to four types of reward functions or four independent evaluation dimensions, respectively. This represents the discount factor, used to measure the impact of future rewards on current decisions; its value typically ranges from [value range missing]. ; represents the decay coefficient in generalized dominance estimation, used to weigh the bias and variance; l represents the cumulative time step offset from the current time t. This represents the time-series difference error of the k-th evaluation channel at time t+l.

[0030] Finally, the overall advantage estimate for guiding the policy network to update its parameters using gradients is obtained. The method of directly summing and accumulating the four channel sub-dominance values ​​is used to obtain the result, which satisfies... This allows the policy network to simultaneously receive conflict-free gradient direction guidance from four independent dimensions.

[0031] When collaboratively updating network parameters, the regression objectives of the four Critic value network branches are set as the sum of the sub-dominance value of each channel and the current state value estimate, i.e., satisfying... .in This represents the target reward estimate for the k-th Critic branch at time t, which is obtained by adding the advantage estimate and the current state value estimate. This represents the advantage estimate of the k-th evaluation channel, used to measure the relative merit of the current action relative to the value function of that channel; This represents the observation of the k-th Critic network on the current policy. The estimated state value function, where The parameters of the k-th Critic network. The total value loss function of the entire system. Instead of using the traditional fixed scalar weighting, it is directly defined as the arithmetic mean of the mean square errors of the four Critic branches, and its formula is expressed as: ; Combined with the shearing loss term of the policy network And policy entropy constraints used to encourage environmental exploration and prevent policies from prematurely converging to local optima. The system constructs a unified joint optimization objective total loss function, whose joint expression is as follows: .

[0032] To further ensure training stability in highly dynamic environments, the system uses the following formula during update iterations: ; The system monitors the Kullback-Leibler (KL) divergence between the old and new strategies in real time. It employs an adaptive learning rate adjustment strategy, automatically and proportionally reducing or increasing the global update step size of the current optimizer when the divergence value exceeds or falls below a set safety threshold. It is important to note that this multi-Critic value estimation architecture and privileged observation vector only exist during the cloud or ground simulation training phase. When deploying and rehearsing the actual humanoid robot's onboard microprocessor, all Critic value networks are directly discarded, and only the simplified Actor policy network is retained for low-latency motion instruction generation. This significantly improves network convergence efficiency and control quality while ensuring extremely low inference computation overhead on the onboard device.

[0033] like Figure 1 As shown, during the reinforcement learning training process, as the number of training iterations increases, the average reward value gradually increases and eventually stabilizes, indicating that the method proposed in this invention can achieve stable convergence and obtain an effective motion control strategy.

[0034] like Figure 2 As shown, during the reasoning process, the actual speed of the robot base can track the target speed command well, and the actual speed curve and the target speed curve maintain a high degree of consistency, which verifies the effectiveness and tracking performance of the control method proposed in this invention.

[0035] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A motion control method for humanoid robots based on structural topology perception and decoupling value estimation, characterized in that: The humanoid robot motion control method is implemented based on a hierarchical reinforcement learning framework. This framework includes a high-level motion planner based on reinforcement learning and a low-level joint proportional differential control feedback loop. The specific control process includes the following steps: Step 1: Define a partially observable Markov decision process model for the humanoid robot in a reinforcement learning environment, and construct the direct observation space of the onboard sensors corresponding to the Actor policy network. And the simulation privileged observation space corresponding to the Critic value network, which includes the actual linear velocity of the base. ; The high-level motion planner receives the observed state at a set first frequency and generates the target angle increments of the 12 drive joints of the lower limb. It then hands them over to the low-level joint proportional differential controller, which operates at a higher second frequency, to output the final joint physical torque to drive the humanoid robot to walk. Step 2: By abstracting the lower limb joints and torso center of the humanoid robot as nodes in a graph structure, the physical properties and spatial constraints of each kinematic chain of the robot are explicitly transformed into a morphological adjacency matrix containing self-loops. And use this morphological adjacency matrix to build a binary hard topological attention mask. ; the hard topology attention mask The Transformer network self-attention computation layer injected into the high-level planner uses negative infinite weights to suppress and block the disordered diffusion of information between non-mechanically connected joint nodes, prompting the network to explicitly extract the dynamic spatial coupling features between lower limb joint links. Step 3: Decompose the overall control objective of the humanoid robot's walking into four semantically independent heterogeneous reward categories, namely: the instruction tracking reward set for following the external translational turning velocity objective. A collection of rewards for maintaining trunk uprightness and preventing falls. Motion regularization reward set used to save energy and ensure smooth joint output. , and the gait shaping reward set used to regulate the desired gait phase ; Step 4: During the update and training phase of the reinforcement learning network, four completely independent Critic value networks are configured to specifically fit the expected state reward values ​​of the four semantic reward categories defined in Step 3; when calculating the policy dominance function, the sub-dominance values ​​of the four channels are calculated based on the independent evaluation results of each value network. The total advantage estimate is obtained by summing and accumulating the sub-advantage values ​​of each channel to guide the policy network parameters for near-end policy optimization (PPO) updates.

2. The humanoid robot motion control method based on structural topology perception and decoupling value estimation according to claim 1, characterized in that: The humanoid robot morphological adjacency matrix defined in step 2 In the Transformer network, the hip, knee, and ankle joint nodes on one side of the humanoid robot are configured to interact fully within a group, and the central node of the torso is bidirectionally connected to the hip joint groups of both thighs, thereby preserving physical symmetry constraints in the Transformer network.

3. The humanoid robot motion control method based on structural topology perception and decoupling value estimation according to claim 1, characterized in that: In step 4, when updating the parameters of the policy network and the value network, a learning rate control strategy based on adaptive update magnitude is adopted to solve the Kullback-Leibler divergence value between the old and new policies in real time. If the divergence value exceeds the set upper limit safety threshold, the current global parameter iteration optimization step size is automatically reduced proportionally.

Images