Humanoid robot control method and system based on reinforcement learning

Abstract

The invention relates to the technical field of intelligent robot control, in particular to a humanoid robot control method and system based on reinforcement learning. The method comprises the steps of constructing and dynamically updating a structure experience type reachable state space by fusing a multi-dimensional real-time state and long-term execution feedback, and providing a safety constraint basis for the humanoid robot; based on the space, generating a basic control track which preferentially guarantees the continuity and the restorability of the structure; then action correction of reinforcement learning is strictly limited in a safe neighborhood of a reachable state space, and a learning process is guided through a reward function fusing a structural stability margin and a correction amplitude, so that local performance optimization is realized; by means of multi-source sensing closed-loop feedback, online track correction and empirical weight and strategy updating are carried out, and long-term self-adaptive optimization is achieved; according to the invention, experience sharing, cooperative training and remote strategy updating among multiple robots are supported through an internet platform, and the overall learning efficiency and deployment safety of the system in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control technology, specifically to a humanoid robot control method and system based on reinforcement learning. Background Technology

[0002] As an advanced form of robot capable of adapting to human work and living environments, humanoid robots rely on motion control technology to support the completion of diverse tasks. With the development of artificial intelligence technology, data-driven methods such as reinforcement learning have shown significant potential in the field of robot motion planning and control, providing new ways to achieve highly adaptive and intelligent robot behavior.

[0003] Chinese invention patent CN120428573B discloses a humanoid robot control system and method based on reinforcement learning. The humanoid robot control system includes a first control subsystem and a second control subsystem. The first control subsystem includes a strategy inference module, a state transition module, and a robot control module. The second control subsystem includes a data acquisition module and a drive control module. The data acquisition module is used to perform timestamp alignment and outlier filtering of sensor data and transmit it to the state transition module. The state transition module is used to fuse multi-source sensor data and construct a temporal state feature containing real-time measurement values ​​and historical time-series information. The strategy inference module is used to generate multi-joint angle target values ​​of the robot based on the temporal state feature provided by the state transition module. The robot control module is used to parse the multi-joint angle target values ​​output by the strategy inference module, select the control mode, and generate control commands including parameter adjustment instructions.

[0004] It is worth noting that the development of cloud computing and large-scale Internet platforms has provided a crucial data foundation, distributed simulation environment, and efficient model iteration capabilities for reinforcement learning training, making the training and verification of complex strategies possible. Current research and practice are dedicated to combining the autonomous decision-making capabilities of reinforcement learning with the inherent high-dimensional, nonlinear, and unstable dynamic characteristics of humanoid robots, in order to improve the robot's task performance in unstructured environments. In this process of technological integration, how to ensure that the learning process and the generated control strategies always conform to the physical structural safety constraints of the robot, and achieve a smooth transition and reliable deployment from virtual training to physical execution, has become a key research topic for promoting the mature application of this field. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the background technology by proposing a humanoid robot control method and system based on reinforcement learning.

[0006] The technical solution of this invention: a humanoid robot control method based on reinforcement learning, comprising the following specific implementation steps: S1. Construct and dynamically update a structural experience-based reachable state space. By integrating multi-dimensional real-time states and long-term execution feedback of joint configuration, driving load, and posture stability, introduce structural stability margin and structural experience weight to hierarchically characterize the reachability of robot actions, forming a state space that is continuously updated during operation, serving as the basis for safety constraints in the control and reinforcement learning process. S2. Generate a basic control trajectory based on the structural experience-based reachable state space, prioritizing the structural safety, motion continuity, and state recoverability of the humanoid robot's actions. By selecting task-related postures, constructing a state-level trajectory skeleton, introducing a time parameterization mechanism for structural margin modulation, and verifying recoverability, a long-term executable basic control trajectory is formed. S3. Based on the basic control trajectory, reinforcement learning local action correction is performed. The action correction of reinforcement learning is restricted to the safe neighborhood defined by the structural empirical reachable state space. By integrating the structural stability margin and the action correction magnitude, a reward function is designed to guide the learning process. This ensures that the optimization exploration of reinforcement learning is constrained within the safe neighborhood, and the output control trajectory with optimized local performance does not destroy the overall stability. S4. By using multi-source sensors to perceive the robot's execution status and environmental information in real time, and combining task error assessment and environmental constraints, the control trajectory corrected by reinforcement learning is corrected online through closed-loop feedback. At the same time, the data collected during the execution process is used to update the structural experience weights and reinforcement learning strategies to achieve long-term adaptive optimization of the system.

[0007] Preferably, in step S1, constructing and dynamically updating the structural empirically reachable state space specifically includes: Construct a structural state vector that uniformly describes the robot's current joint angles, joint angular velocities, joint drive loads, joint coupling relationships, and overall attitude state. Based on the structural state vector, a unified structural stability margin function is constructed from three dimensions: joint stroke safety distance, driving load safety margin, and robot overall posture safety distance. The safety margin of the current state from the structural instability boundary is continuously quantified by the value of this function. For each structural state, a time-evolving structural empirical weight is introduced. Through an empirical weight update function, the reliability of the structural state is accumulated based on long-term execution feedback, thereby distinguishing between theoretically calculated reachable states and actual long-term verified reliable states. The structural states are screened by combining the preset minimum stability margin threshold and the minimum empirical weight threshold to form an empirically reachable state space, and the boundaries of this space are dynamically corrected during robot operation.

[0008] Preferably, in step S2, generating the basic control trajectory specifically includes: The current task objective is associated and matched with the structural empirical reachable state space. A task deviation metric function is used to filter the attitude states from the structural empirical reachable state space that satisfy both structural reachability and task functional requirements, forming a task-related attitude candidate set. Based on the task-related attitude candidate set, and according to the constraints of joint configuration change amplitude and driving load continuity, several key structural states are selected and connected in sequence to construct a state-level trajectory skeleton. A time modulation mechanism driven by structural stability margin is introduced into the state-level trajectory skeleton. The execution speed of the trajectory in different segments is dynamically adjusted by a time modulation factor. The execution pace is automatically slowed down in segments with low structural margin, and a time-parameterized basic control trajectory is generated. The structural stability margin of the generated basic control trajectory is evaluated as a whole over its complete execution cycle. Its long-term execution reliability is verified by a recoverability evaluation index, and the verified trajectory is solidified as a reference benchmark for subsequent reinforcement learning correction.

[0009] Preferably, in step S3, reinforcement learning local action correction is performed, specifically including: Define a locally correctable action space, constrain the action correction vector output by the reinforcement learning algorithm at each step within this space, and limit its correction magnitude to no more than the maximum allowable value determined by the joint capability and actuator performance, ensuring that the correction action is within the safe neighborhood of the structural empirical reachable state space; Construct a reward function that integrates a task reward term that directly reflects the task completion degree, a structure stability margin term that encourages state safety, and an action smoothing penalty term that suppresses excessive corrections. By adjusting the weight coefficients of each term, the optimization direction of reinforcement learning can be guided. Under the constraints of the locally modifiable action space and the guidance of the reward function, a reinforcement learning policy is trained. This policy outputs local action corrections based on the current state and the base trajectory, and structural experience weights are incorporated during training to amplify the influence of long-term reliable states. The action correction output by the reinforcement learning strategy is superimposed and integrated with the basic control trajectory to form a locally corrected control trajectory. An online gradient update strategy is designed to dynamically fine-tune the correction to ensure that the entire correction process always meets the structural safety boundary.

[0010] Preferably, in step S4, closed-loop feedback and long-term optimization specifically include: During task execution, information such as joint angles, joint angular velocities, joint torques, center of mass position, center of mass velocity, contact forces, task target position, and environmental status is collected and fused in real time to form a multi-dimensional real-time execution state vector. Based on the real-time execution state vector, a comprehensive task error function is calculated. This function quantifies the deviation between the current state and the expected state in many aspects, such as centroid position, centroid velocity, end target position, contact force, and environmental constraints. Based on the gradient of the comprehensive task error function, the environmental constraint gradient, and the locally corrected trajectory obtained in step S3, a closed-loop correction control quantity is calculated in real time, and this correction quantity is fused with the reinforcement learning correction action by a certain coefficient to generate the final executable control trajectory. By utilizing all the collected data, the structural experience weights and reinforcement learning strategies are updated synchronously through an update function, thus achieving a closed loop of experience accumulation and strategy optimization.

[0011] Preferably, the structural stability margin function is specifically defined as the minimum value among the calculation results of the joint travel safety distance function, the drive load safety margin function, and the overall attitude safety distance function.

[0012] Preferably, the empirical weight update function is the sum of the empirical weight value of the previous control cycle, an empirical update coefficient, and the current execution feedback score; The execution feedback score is calculated by comprehensively considering multiple practical engineering indicators, including posture recovery time, joint vibration amplitude, driving temperature rise trend, and energy consumption changes.

[0013] Preferably, when constructing the state-level trajectory skeleton, the norm of the difference between the joint angle vectors between two adjacent key states shall not exceed a preset joint angle continuity threshold, and the norm of the difference between the joint load vectors between two adjacent key states shall not exceed a preset joint load continuity threshold.

[0014] Preferably, the weight coefficient of the structural stability margin term in the reward function is set according to the priority requirements of safety for the specific task, and the weight coefficient of the motion smoothness penalty term is set according to the maximum allowable instantaneous speed of the robot joint and the load response capability of the actuator.

[0015] The technical solution of the present invention: a humanoid robot control system based on reinforcement learning, which is used to execute the above-mentioned humanoid robot control method based on reinforcement learning, comprising: The structural experience reachable space construction module is used to collect information on the joint angles, joint velocities, loads, and center of mass of the humanoid robot, and combine it with historical task execution experience to construct and dynamically update the structural experience-based reachable state space. The basic trajectory generation module is used to generate basic control trajectories that meet the requirements of structural safety, continuity and recoverability based on the structural empirical reachable state space and task objectives. The reinforcement learning local correction module is used to optimize the action locally within the framework of the basic control trajectory, combining the constraints of the structural empirical reachable state space and the safety reward function, and output the corrected action using the reinforcement learning algorithm. The task closed-loop and feedback correction module is used to evaluate task execution errors and environmental constraints based on real-time sensing information from multiple sources of sensors, generate closed-loop correction control variables, and fuse them with correction actions to form the final execution trajectory. At the same time, it updates the experience weights in the structural experience reachable space construction module and the strategies in the reinforcement learning local correction module.

[0016] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects: This invention designs a humanoid robot control method and system based on reinforcement learning. First, by constructing a structurally empirically reachable state space, it deeply integrates the real-time states of the robot's joints, load, and posture with long-term execution experience, forming dynamically updated safe action boundaries. This significantly improves the structural stability and motion reliability of the humanoid robot during complex task execution, effectively avoiding instability, overload, or mechanical damage caused by blind exploration. Second, by constraining the action corrections learned through reinforcement learning within an empirically reachable safe neighborhood and designing a reward function that integrates structural margin and task objectives, it can achieve autonomous optimization of local actions while ensuring system safety. This invention enhances the robot's adaptability to dynamic environments and multi-task conditions. Furthermore, it possesses excellent system compatibility and scalability, enabling not only integration into existing industrial control systems for collaborative operation with production line equipment and scheduling systems, but also remote strategy updates, experience sharing, and collaborative learning via an internet platform. This supports knowledge transfer and continuous optimization for multiple robots in different scenarios. Ultimately, through real-time perception, online correction, and long-term experience accumulation, this closed-loop control system achieves an effective balance between task execution efficiency, motion safety, and system adaptability, providing crucial technical support for the reliable deployment of humanoid robots in intelligent manufacturing, service collaboration, and other fields. Attached Figure Description

[0017] Figure 1 This is a flowchart of a humanoid robot control method based on reinforcement learning proposed in this invention; Figure 2 This is a system architecture diagram of a humanoid robot control system based on reinforcement learning proposed in this invention. Detailed Implementation

[0018] Example 1, as Figure 1As shown, the present invention proposes a humanoid robot control method based on reinforcement learning, which includes the following specific implementation steps: S1. By constructing a structural state representation that integrates joint configuration, driving load, posture stability, and long-term execution feedback, stability margin and empirical weights are introduced to hierarchically characterize the robot's motion reachability, forming a structural empirical reachability state space that is continuously updated during operation. This provides a unified and reliable basis for motion constraints for subsequent control and reinforcement learning. The specific implementation process is as follows: S11. By uniformly constructing a structural state vector from real-perceptible structural information such as joint angles, joint velocities, drive loads, joint coupling relationships, and overall posture, the robot's configuration, forces, and stability at any given moment are synchronously described. This provides a state foundation that closely reflects actual execution conditions for reachability assessment. Specifically, starting from actual robot operation, a structural state expression method oriented towards execution reliability is constructed, representing the robot's structural state at time t as: ; This structural state vector maps the robot's structurally realizable state at any given moment to the same state space. in, It represents the comprehensive structural state of the humanoid robot at time t, and is used to uniformly describe the robot's current posture, motion trend, force situation and overall stability state. This represents the set of current angle values ​​of each joint of the robot at time t, used to reflect the overall configuration and posture of the robot, and is obtained directly by the joint position encoder; This represents the rate at which the angles of each joint change over time, used to describe the dynamic trend of robot movements. It is achieved by analyzing the joint angles... Perform time difference calculation; It represents the equivalent driving torque or load level of each joint of the robot at the current moment, used to characterize the stress state of the structure, and is calculated by the actuator current feedback, torque sensor or equivalent torque estimation model; This represents the joint coupling state vector, used to describe the cooperative or coupled relationship exhibited by multiple joints during actual movement, reflecting the structural linkage characteristics. It is obtained through statistical analysis of joint angle, angular velocity, and load changes in historical movement data. It represents the robot's overall posture state in space, including the position of the center of mass, posture angles, and support contact information, which is calculated by the inertial measurement unit, foot sensors, and posture fusion algorithm. S12. Based on the structural state vector, a unified structural stability margin index is constructed from three dimensions: joint travel, driving load capacity, and overall attitude stability. The weakest structural factor is used as the criterion to continuously quantify whether the state meets the conditions for safe execution. Specifically: After obtaining the structural state vector, a structural stability margin function is introduced to measure the "safety margin" between the current state and the structural instability boundary. The structural stability margin function is defined as follows: ; when When the value exceeds a preset threshold, the state is marked as a structurally stable and reachable state. in, The structural stability margin function is used to quantitatively describe the minimum safety margin between the current structural state and instability or structural limit. This represents the minimum safe distance between the current joint angle and its mechanical travel limit, determined by joint design parameters, assembly tolerances, and the current joint angle. Calculated; This indicates the remaining proportion or safety margin of the current joint load relative to its rated safe load, calculated by the driver's rated parameters and the current drive load. The results were obtained through comparison and calculation; The safe distance between the robot's center of mass projection and the boundary of the support area is represented by the overall attitude parameters. Calculated by combining plantar contact information; S13. By introducing structural empirical weights that evolve over time, the long-term execution reliability of states that have passed the stability margin screening is accumulated, thereby giving higher credibility to states that have been repeatedly verified to be stable and feasible. This distinguishes between theoretically attainable states and practically credible states. Specifically: Considering that theoretical reachability does not equate to long-term executableness, we introduce structural empirical weights to describe the degree to which a given state is "repeatedly verified as feasible" in actual execution. For each structural state, we introduce an empirical weight update function: ; When a state continuously receives positive feedback in multiple executions, its empirical weight increases and it is regarded as a highly reliable reachable state. in, It represents the structural experience weight, that is, the degree to which a certain structural state has been verified as reliable and repeatable in long-term actual execution; This represents the experience update coefficient, which is used to adjust the relative influence of historical experience and the latest execution feedback on the experience weight update. The execution feedback score is used to comprehensively reflect the performance quality of the current structural state in actual execution. It is calculated by a combination of engineering indicators such as attitude recovery time, joint vibration amplitude, driving temperature rise trend, and energy consumption change. Representing structural state The structural empirical weight value corresponding to the previous control moment or the previous execution cycle is used to characterize the degree to which the structural state has been verified as stable, reliable and repeatable during historical execution. S14. The structural state is filtered and updated by combining the stability margin threshold and the empirical weight threshold to form an empirically driven reachable state space. This space boundary is dynamically corrected during robot operation, providing real-time constraints for subsequent control trajectory generation and reinforcement learning action pruning. Specifically: By combining structural states, stability margins, and empirical weights, a final reachable state space for the humanoid robot used for control and learning constraints is formed. The reachable state space is defined as follows: ; in, The reachable state space represents the set of states that a humanoid robot is allowed to enter and perform actions under the dual constraints of structural stability and long-term execution experience. This represents the minimum stability margin threshold, used to limit the minimum safety margin that a structural state must possess, and is determined by the robot's structural safety design specifications and actual operational safety requirements. This represents the minimum empirical weight threshold, used to define the minimum empirical level at which a state is considered "reliable in the long term," and is set based on the robot's long-term operational statistics and fault tolerance requirements. It should be noted that the reachable state space It is continuously updated during robot operation and serves as a unified constraint boundary for subsequent basic control trajectory generation and reinforcement learning action pruning.

[0019] S2. Based on the structurally empirical reachable state space constructed in step S1, without pursuing optimal action, priority is given to ensuring the structural safety, continuity, and recoverability of the humanoid robot's actions. This is achieved by selecting task-related poses from empirically reachable states, constructing a state-level trajectory skeleton, introducing a time parameterization mechanism for structural margin modulation, and verifying the recoverability of the generated trajectory. This forms a long-term executable basic control trajectory, providing a stable and reliable reference path for subsequent reinforcement learning-based constrained corrections. The specific implementation process is as follows: S21. By associating and matching the current task objective with the structurally empirical reachable state space, a set of attitude states that conform to the task functional requirements is selected from the reachable states that have been verified through long-term execution. Specifically: Based on the reachable state space output in step S1 For the current task objective G, construct a set of task-related attitude candidates: ; This screening process retains only attitude states that are structurally proven to be reachable over a long period of time and functionally meet the task requirements. Where G represents the current mission objective, such as the end pose or a specific operational objective; This represents the set of task-related pose candidates, that is, the set of states that satisfy task matching within the reachable state space; It represents a single state vector in the reachable state space, containing information such as robot joint angles, angular velocities, loads, joint coupling states, and overall posture; This represents the task deviation measurement function, describing the state. The degree of matching with the mission objective G, including end-effector pose error, attitude error, or support state error defined according to mission requirements; This represents the task deviation threshold, used to filter states that match the task objective. It is set based on task accuracy requirements and robot capability experience. S22. Based on the candidate set of task postures, and according to the constraints of joint configuration change amplitude and driving load continuity, several structural states are selected to construct a state-level trajectory skeleton. This ensures that the robot's movements maintain smooth joint changes and controllable load changes during state transitions, thereby avoiding implicit instability problems caused by state jumps. Specifically: Construct a state-level trajectory skeleton consisting of stable states from the candidate set. Select several key states to form a state sequence: ; And it satisfies the following structural continuity constraints: ; in, This represents the trajectory skeleton, which is a sequence of states composed of several key poses strung together. This represents the j-th key state in the trajectory skeleton; The joint angle vector representing the j-th critical state; Represents the joint load vector for the j-th critical state; This represents the joint angle continuity threshold, which is the maximum allowable change in joint angle between adjacent states. It is set based on the maximum angular velocity of the joint, the control cycle, and mechanical characteristics. This represents the joint load continuity threshold, which is the maximum allowable change in joint load between adjacent states. It is set based on the rated load and thermal response characteristics of the drive. S23. For the constructed state-level trajectory skeleton, a time parameterization mechanism driven by structural stability margin is introduced. This automatically slows down the trajectory execution pace in sections with low structural margin and maintains normal execution speed in sections with sufficient margin. This improves the overall stability and execution reliability of the action without altering the trajectory structure. Specifically: A time modulation mechanism driven by structural margin is introduced to automatically slow down the basic trajectory in "tight" structural sections. The basic control trajectory is defined as follows: ; ; When the structural margin decreases, the trajectory automatically decelerates, thereby reducing the risk of instability; in, This represents the basic control trajectory output (joint target or equivalent control command) at time t. This represents a trajectory interpolation function used to generate smooth, continuous trajectories between critical states. Linear or spline interpolation is commonly used, but the time modulation factor... Modulated by structural margin; This represents the time modulation factor, used to dynamically adjust the trajectory execution speed; This represents the structural stability margin function, used to evaluate the state safety at time t; This represents the empirical reference margin, used to normalize the time modulation factor, and is determined by robot experience and task safety requirements. S24. By comprehensively evaluating the structural stability margin of the basic control trajectory during execution, the recoverability of the trajectory within the complete execution cycle is verified. Only the trajectory whose stability margin remains within the safe range during long-term execution is solidified as the effective basic control trajectory, serving as the sole reference benchmark for subsequent reinforcement learning action correction. Specifically: Define the recoverability evaluation index of the basic trajectory: ; when When the baseline control trajectory remains above the empirical threshold, it is considered to have long-term execution reliability and is solidified as a reference trajectory for subsequent reinforcement learning. in, This represents the overall stability evaluation index of the basic trajectory; T represents the total execution time or cycle of the trajectory, which is determined by the task requirements and the trajectory length.

[0020] S3. Based on the basic control trajectory generated in step S2, local action correction is achieved through reinforcement learning. The entire process is carried out within the safety fence of the structural experience-reachable space. The reward function is designed by combining structural stability margin and correction magnitude constraints, so that reinforcement learning optimization only explores within the safe neighborhood. Finally, an executable control trajectory that is both robust and adaptable to environmental changes is output, achieving a balance between task performance improvement and structural safety. The specific implementation process is as follows: S31. Based on the basic trajectory output in step S2, define a locally modifiable action space. Restrict the reinforcement learning output correction actions to fall within the empirically reachable neighborhood of the structure, and ensure that the correction magnitude does not exceed the allowable values ​​of the joints and loads. This guarantees exploration safety and avoids disrupting stability, providing a controlled action boundary for reinforcement learning. Specifically: Define a local correction action space, strictly constraining the learning range to the neighborhood of the reachable state, i.e., define the reinforcement learning action correction vector: ; in, This represents the action correction vector output by reinforcement learning at time t, used to adjust the base trajectory in step S2. Make local adjustments; This represents the set of safe action spaces, defines the allowed correction range, and is the reachable state space output from step S1. Calculated in conjunction with joint / load limits; This represents the maximum magnitude of motion correction, which is the maximum amount that reinforcement learning can modify the basic trajectory each time. It is determined by engineering parameters such as the maximum allowable instantaneous speed of the robot joints, the load response capability of the actuators, and the control cycle. This indicates that reinforcement learning controls the trajectory of the base at time t. The magnitude of the action correction is specifically the Euclidean norm of the vector; S32. The task completion rate is integrated with the structural stability margin and correction magnitude constraints into a reward function. The reward reflects both task efficiency and encourages smooth and safe movements. The structural margin weight and correction penalty coefficient are determined by the robot's experience and joint performance, ensuring that the reinforcement learning optimization direction is effective without compromising system stability. Specifically: By incorporating structural margin into the reward function and dynamically adjusting the learning direction, the reward function is defined as follows: ; in, This represents the immediate reward function value for reinforcement learning at time t, used to guide policy updates; It indicates the rewards related to task completion, such as reduction of end-position error, maintenance of gait balance, and successful object grasping, and directly reflects the task completion degree based on the task objective definition; Represents the structural stability margin function; This represents the weighting coefficient of structural stability margin in the reward function; it is an empirically engineered value set based on the task's safety priority requirements. This represents the motion smoothing penalty weight, which suppresses and corrects excessive motion amplitude. It is set by engineering parameters such as the maximum allowable speed of the joint and the load response capability. S33. Under the guidance of the safe action space in step S1 and the reward function in step S2, the reinforcement learning policy is trained. The policy outputs local action corrections, and the empirical weights guide the learning towards long-term reliable states. The corrections are only performed in the safe neighborhood, forming "empirical constraint-guided local optimization," which improves task performance while ensuring structural safety. Specifically: Reinforcement learning training not only relies on environment simulation, but also requires the dynamic fusion of the experience weights from step S1. To improve the reliability of the correction strategy, define the action correction strategy: ; During training, experience weights This is used to amplify the reward contribution of long-term reliable states, suppress the learning of accidental successes or high-risk actions, and form "experience-guided reinforcement learning". in, This represents a reinforcement learning strategy that outputs a given base trajectory. Action correction The probability distribution or policy function; It represents the expected value of a state in the reachable state space, used for reinforcement learning policy optimization; S34. Integrate reinforcement learning-corrected actions with the basic trajectory to form the final executable trajectory, and adapt to environmental disturbances and task changes through online gradient updates, always ensuring that action corrections are within a safe range. This achieves closed-loop control of robust execution, local optimization, and dynamic adaptation, providing reliable support for subsequent task completion. Specifically: After the reinforcement learning outputs the corrected action, it is integrated with the basic trajectory in step S2 to form the final executable trajectory. At the same time, it can be dynamically fine-tuned during execution, that is, the final execution trajectory is defined: ; Define the online trajectory update strategy: ; The following condition is always satisfied during the trajectory update process: This ensures that reinforcement learning corrections do not exceed the structural safety boundary; This indicates the online learning step size, used for dynamic fine-tuning of reinforcement learning corrections, and is set by engineering based on the control cycle and system response speed. This represents the gradient of the reward function with respect to the policy, used for online updates of action corrections, and is calculated by the reinforcement learning training algorithm. in, This indicates the final executable control trajectory, which is modified by reinforcement learning on the basic trajectory in step S2.

[0021] S4. Through multi-source real-time perception, task error evaluation, reinforcement learning correction, and online closed-loop feedback, the reinforcement learning correction trajectory in step S3 is further optimized to achieve safe, adaptive, and efficient closed-loop control of the humanoid robot when performing tasks. At the same time, the experience weights and strategies are updated by combining execution data feedback to achieve long-term adaptive optimization. The specific implementation process is as follows: S41. During execution, the robot collects joint angles, angular velocities, joint torques, center of mass position and velocity, contact forces, target position, and environmental conditions in real time. Through multi-sensor fusion, a complete execution state vector is formed, providing multi-dimensional input for error assessment and closed-loop correction, ensuring motion safety and environmental adaptability, and providing a data foundation for strategy iteration. Specifically: During task execution, information from a single joint or end effector is insufficient to guarantee stability and task success rate. It is necessary to collect real-time data on the robot's overall dynamic state, environmental interactions, and task objectives, defining a real-time execution state vector. ; in, The angular position vectors of each joint of the robot are represented by joint encoders or angle sensors and are measured in real time. Represents the joint angular velocity vector; Indicates joint torque or driving load; This represents the position vector of the robot's center of mass in space; Represents the velocity vector of the center of mass; This represents the contact force vector at the robot's end effector or foot. Represents the target location vector; Represents environmental status information (obstacles, terrain, dynamic objects, etc.); S42. Calculate multi-dimensional task errors based on perception data, including centroid position and velocity deviation, end-effector target deviation, contact force deviation, and environmental constraint costs. Quantify each indicator using weighting coefficients to achieve a unified assessment of task completion and structural safety, providing a reliable quantitative basis for closed-loop correction and ensuring robust robot performance in dynamic environments. Specifically: Based on the calculation of task deviation using perceived data, and by introducing stability indicators and environmental constraints, a unified quantification of task execution and safety is achieved, and an error function is defined: ; in, Represents the overall task error function; The reference centroid position is derived from the reinforcement learning correction trajectory in step S3. Predicted centroid location; Indicates the reference center of mass velocity; Indicates the reference location of the mission objective; Indicates the desired contact force; The environmental constraint cost function is derived from sensor measurements of obstacle distances, terrain slopes, etc. , , , and The weights of each term in the error function are set based on empirical values ​​or the importance of the task. S43. The reinforcement learning correction action from step S3 is fused with the error gradient and environmental constraint gradient to generate a closed-loop correction control quantity. The final execution trajectory is dynamically adjusted to achieve task adaptive correction. The action correction is strictly performed within the safety boundary, taking into account task completion efficiency, structural stability, and environmental adaptability, forming a safe and controllable online closed-loop execution trajectory, specifically as follows: According to the error function and the reinforcement learning correction trajectory output in step S3 Real-time generation of corrective control values: ; Final action to be performed: ; in, This indicates a closed-loop correction action; The gradient of the error with respect to the action is derived from the dynamic model or online differential calculation; This represents the closed-loop gain matrix, which is set according to the robot's joint driving capability and response speed. This indicates that reinforcement learning corrects the fusion coefficients, adjusting them based on experience or through simulation. It represents the gradient of environmental constraints on the action, calculated based on environmental sensor information; This indicates the environmental constraint fusion weight, which is set based on task priority and security requirements; Indicates the final action to be performed; S44. During robot execution, state, error, and corrective action data are collected. By updating the experience weights in step S1 and the reinforcement learning strategy in step S3, long-term adaptive closed-loop optimization is achieved. This enables the robot to continuously improve task completion rate and safety under complex environments and multi-task conditions, ensuring the dynamic optimization and fusion of experience guidance and reinforcement learning strategy. Specifically: The state, error, and correction action data collected during execution will be used to update the reinforcement learning policy in step S3 and the empirical weights in step S1, achieving long-term adaptive optimization. The data update process is defined as follows: ; ; ; in, This represents the state experience weight in step S1; This indicates the step size for updating the empirical weights, which is set based on the convergence speed and system response. This represents the amount of empirical weight update, calculated by considering the combined costs of task error, structural stability margin, and environmental constraints. This indicates a reinforcement learning action strategy; This indicates the update step size of the reinforcement learning strategy; The gradient of the reward function is represented and calculated by combining task completion reward, structural safety margin, and environmental constraints. This represents the empirical weight update function; It should be noted that the empirical weight update function The update amount used to calculate the state experience weights takes into account task error, structural safety margin, and environmental constraint costs. When the task error is large or the environmental constraint cost is high, the function reduces the weight of the state, reducing the priority of reinforcement learning policies for it. When the structural safety margin is large, the task error is small, and the environmental constraints are permissible, the function increases the weight of the state, encouraging policies to explore and utilize the state more, thereby ensuring structural safety and environmental adaptability while completing the task, and achieving long-term adaptive closed-loop optimization. It should be further explained that, in actual deployment, the structural experience-based reachable state space, experience weights, and reinforcement learning strategies can be uploaded to the cloud platform for centralized management and optimization, supporting safety experience sharing and collaborative training among multiple robot systems, thereby improving the learning efficiency and safety of the system in a large-scale deployment environment.

[0022] Example 2, as Figure 2 As shown, the present invention proposes a humanoid robot control system based on reinforcement learning, which is used to execute a humanoid robot control method based on reinforcement learning proposed in Embodiment 1. It includes: a structural experience reachable space construction module, a basic trajectory generation module, a reinforcement learning local correction module, and a task closure and feedback correction module.

[0023] The structural experience-based reachable space construction module is responsible for collecting information on the humanoid robot's joint angles, joint velocities, loads, and center of mass. Combined with historical task execution experience and dynamic models, it constructs and calibrates the robot's safe reachable motion and state spaces, generating a multi-dimensional set of state constraints. By integrating joint constraints, structural stability margins, and load capacity, it establishes a reachable state library, providing safe constraint boundaries for subsequent trajectory generation and reinforcement learning. The state library is dynamically updated during system operation to reflect environmental changes and structural health, ensuring that the robot's actions are always performed within a safe and reliable range. This module can also interface with a cloud-based big data platform to achieve the aggregation and sharing of multi-robot experience data, improving the system's adaptability in complex task scenarios. The basic trajectory generation module generates a preliminary executable trajectory based on the reachable state space and task planning instructions, providing the robot with a task skeleton. It decomposes the target task into a temporal sequence of motion points and joint postures, considering centroid stability, end-effector target position, and contact force distribution. Through a dynamic model, it predicts the robot's stability risks during execution and generates a basic control trajectory. This trajectory satisfies both task completion requirements and structural experience constraints, providing a reference and constraint benchmark for reinforcement learning corrections, ensuring the robot remains in a safe and controllable state during execution. This module supports task scheduling and trajectory collaborative optimization based on a cloud platform, making it suitable for multi-robot collaborative operation scenarios. The reinforcement learning local correction module, within the framework of the basic trajectory, combines structural experience reachability space and task feedback to locally optimize actions using reinforcement learning algorithms. It constructs a safe action neighborhood to constrain the reinforcement learning exploration space and designs a reward function that combines task completion metrics, structural stability margin, and action smoothness to generate corrected actions. The module can adjust the reinforcement learning output in real time according to different task priorities and environmental disturbances, improving action performance through local optimization while maintaining structural safety, ensuring the robot's stability and adaptability in complex environments. This module employs a combination of cloud training and edge execution, utilizing cloud computing resources for large-scale policy training and then deploying the optimized model to the robot for local execution. The task closed-loop and feedback correction module, based on the robot's sensors, perceives joint states, center of mass position, velocity, contact force, and environmental information in real time. It performs online error assessment on the basic trajectory and the reinforcement learning-corrected trajectory; calculates task completion deviation and environmental constraint deviation; generates corrective control variables through a closed-loop control strategy; and integrates these variables with the reinforcement learning-corrected actions to form the final execution trajectory. Simultaneously, it collects execution data for updating empirical weights and reinforcement learning strategies, achieving long-term adaptive optimization. This ensures that the robot can dynamically respond to environmental changes during task execution, maintaining an efficient, smooth, and safe corrected trajectory, forming a complete adaptive closed-loop control system. This module has IoT data access capabilities, integrating multi-source sensing data and performing big data analysis through a cloud platform to further optimize the closed-loop control strategy and improve the overall performance of the system in an internet and cloud computing service environment.

[0024] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A humanoid robot control method based on reinforcement learning, characterized in that, The specific implementation steps include the following: S1. Construct and dynamically update a structural experience-based reachable state space. By integrating multi-dimensional real-time states and long-term execution feedback of joint configuration, driving load, and posture stability, introduce structural stability margin and structural experience weight to hierarchically characterize the reachability of robot actions, forming a state space that is continuously updated during operation, serving as the basis for safety constraints in the control and reinforcement learning process. S2. Generate a basic control trajectory based on the structural experience-based reachable state space, prioritizing the structural safety, motion continuity, and state recoverability of the humanoid robot's actions. By selecting task-related postures, constructing a state-level trajectory skeleton, introducing a time parameterization mechanism for structural margin modulation, and verifying recoverability, a long-term executable basic control trajectory is formed. S3. Based on the basic control trajectory, reinforcement learning local action correction is performed. The action correction of reinforcement learning is restricted to the safe neighborhood defined by the structural empirical reachable state space. By integrating the structural stability margin and the action correction magnitude, a reward function is designed to guide the learning process. This ensures that the optimization exploration of reinforcement learning is constrained within the safe neighborhood, and the output control trajectory with optimized local performance does not destroy the overall stability. S4. By using multi-source sensors to perceive the robot's execution status and environmental information in real time, and combining task error assessment and environmental constraints, the control trajectory corrected by reinforcement learning is corrected online through closed-loop feedback. At the same time, the data collected during the execution process is used to update the structural experience weights and reinforcement learning strategies to achieve long-term adaptive optimization of the system.

2. The humanoid robot control method based on reinforcement learning according to claim 1, characterized in that, In step S1, the empirically reachable state space of the structure is constructed and dynamically updated, specifically including: Construct a structural state vector that uniformly describes the robot's current joint angles, joint angular velocities, joint drive loads, joint coupling relationships, and overall attitude state. Based on the structural state vector, a unified structural stability margin function is constructed from three dimensions: joint stroke safety distance, driving load safety margin, and robot overall posture safety distance. The safety margin of the current state from the structural instability boundary is continuously quantified by the value of this function. For each structural state, a time-evolving structural empirical weight is introduced. Through an empirical weight update function, the reliability of the structural state is accumulated based on long-term execution feedback, thereby distinguishing between theoretically calculated reachable states and actual long-term verified reliable states. The structural states are screened by combining the preset minimum stability margin threshold and the minimum empirical weight threshold to form an empirically reachable state space, and the boundaries of this space are dynamically corrected during robot operation.

3. The humanoid robot control method based on reinforcement learning according to claim 2, characterized in that, In step S2, the basic control trajectory is generated, specifically including: The current task objective is associated and matched with the structural empirical reachable state space. A task deviation metric function is used to filter the attitude states from the structural empirical reachable state space that satisfy both structural reachability and task functional requirements, forming a task-related attitude candidate set. Based on the task-related attitude candidate set, and according to the constraints of joint configuration change amplitude and driving load continuity, several key structural states are selected and connected in sequence to construct a state-level trajectory skeleton. A time modulation mechanism driven by structural stability margin is introduced into the state-level trajectory skeleton. The execution speed of the trajectory in different segments is dynamically adjusted by a time modulation factor. The execution pace is automatically slowed down in segments with low structural margin, and a time-parameterized basic control trajectory is generated. The structural stability margin of the generated basic control trajectory is evaluated as a whole over its complete execution cycle. Its long-term execution reliability is verified by a recoverability evaluation index, and the verified trajectory is solidified as a reference benchmark for subsequent reinforcement learning correction.

4. The humanoid robot control method based on reinforcement learning according to claim 3, characterized in that, In step S3, reinforcement learning local action correction is performed, specifically including: Define a locally correctable action space, constrain the action correction vector output by the reinforcement learning algorithm at each step within this space, and limit its correction magnitude to no more than the maximum allowable value determined by the joint capability and actuator performance, ensuring that the correction action is within the safe neighborhood of the structural empirical reachable state space; Construct a reward function that integrates a task reward term that directly reflects the task completion degree, a structure stability margin term that encourages state safety, and an action smoothing penalty term that suppresses excessive corrections. By adjusting the weight coefficients of each term, the optimization direction of reinforcement learning can be guided. Under the constraints of the locally modifiable action space and the guidance of the reward function, a reinforcement learning policy is trained. This policy outputs local action corrections based on the current state and the base trajectory, and structural experience weights are incorporated during training to amplify the influence of long-term reliable states. The action correction output by the reinforcement learning strategy is superimposed and integrated with the basic control trajectory to form a locally corrected control trajectory. An online gradient update strategy is designed to dynamically fine-tune the correction to ensure that the entire correction process always meets the structural safety boundary.

5. The humanoid robot control method based on reinforcement learning according to claim 4, characterized in that, In step S4, closed-loop feedback and long-term optimization specifically include: During task execution, information such as joint angles, joint angular velocities, joint torques, center of mass position, center of mass velocity, contact forces, task target position, and environmental status is collected and fused in real time to form a multi-dimensional real-time execution state vector. Based on the real-time execution state vector, a comprehensive task error function is calculated. This function quantifies the deviation between the current state and the expected state in many aspects, such as centroid position, centroid velocity, end target position, contact force, and environmental constraints. Based on the gradient of the comprehensive task error function, the environmental constraint gradient, and the locally corrected trajectory obtained in step S3, a closed-loop correction control quantity is calculated in real time, and this correction quantity is fused with the reinforcement learning correction action by a certain coefficient to generate the final executable control trajectory. By utilizing all the collected data, the structural experience weights and reinforcement learning strategies are updated synchronously through an update function, thus achieving a closed loop of experience accumulation and strategy optimization.

6. The humanoid robot control method based on reinforcement learning according to claim 2, characterized in that, The structural stability margin function is specifically defined as the minimum value among the calculated results of the joint travel safety distance function, the drive load safety margin function, and the overall attitude safety distance function.

7. The humanoid robot control method based on reinforcement learning according to claim 2, characterized in that, The empirical weight update function is specifically the sum of the empirical weight value of the previous control cycle, an empirical update coefficient, and the current execution feedback score. The execution feedback score is calculated by comprehensively considering multiple practical engineering indicators, including posture recovery time, joint vibration amplitude, driving temperature rise trend, and energy consumption changes.

8. The humanoid robot control method based on reinforcement learning according to claim 3, characterized in that, When constructing a state-level trajectory skeleton, the norm of the difference between joint angle vectors between two adjacent key states must not exceed a preset joint angle continuity threshold, and the norm of the difference between joint load vectors between two adjacent key states must not exceed a preset joint load continuity threshold.

9. A humanoid robot control method based on reinforcement learning according to claim 4, characterized in that, The weighting coefficient of the structural stability margin term in the reward function is set according to the priority requirements of safety for the specific task, and the weighting coefficient of the motion smoothness penalty term is set according to the maximum allowable instantaneous speed of the robot joint and the load response capability of the actuator.

10. A humanoid robot control system based on reinforcement learning, used to execute the humanoid robot control method based on reinforcement learning as described in any one of claims 1 to 9, characterized in that, include: The structural experience reachable space construction module is used to collect information on the joint angles, joint velocities, loads, and center of mass of the humanoid robot, and combine it with historical task execution experience to construct and dynamically update the structural experience-based reachable state space. The basic trajectory generation module is used to generate basic control trajectories that meet the requirements of structural safety, continuity and recoverability based on the structural experience-based reachable state space and task objectives. The reinforcement learning local correction module is used to optimize the action locally within the framework of the basic control trajectory, combining the constraints of the structural empirical reachable state space and the safety reward function, and output the corrected action using the reinforcement learning algorithm. The task closed-loop and feedback correction module is used to evaluate task execution errors and environmental constraints based on real-time sensing information from multiple sources of sensors, generate closed-loop correction control variables, and fuse them with correction actions to form the final execution trajectory. At the same time, it updates the experience weights in the structural experience reachable space construction module and the strategies in the reinforcement learning local correction module.

Images