A humanoid robot control method and system based on reinforcement learning

Abstract

The present application relates to the technical field of intelligent robot control, in particular to a humanoid robot control method and system based on reinforcement learning; the method comprises the following steps: constructing and dynamically updating a structure experience type reachable state space by fusing multi-dimensional real-time state and long-term execution feedback, to provide a safety constraint basis for the humanoid robot; then generating a basic control trajectory based on the space, which prioritizes guaranteeing the continuity and recoverability of the structure; subsequently strictly limiting the action correction of reinforcement learning within the safety neighborhood of the reachable state space, and guiding the learning process by fusing a reward function of structure stability margin and correction amplitude, to realize local performance optimization; online correcting the trajectory and updating the experience weight and strategy by means of multi-source sensing closed-loop feedback, to realize long-term adaptive optimization; the present application supports experience sharing, collaborative training and remote strategy updating among multiple robots through an internet platform, to improve the overall learning efficiency and deployment safety of the system in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent robot control, in particular to a humanoid robot control method and system based on reinforcement learning. BACKGROUND

[0002] As an advanced robot form that can adapt to human work and living environment, the motion control technology of humanoid robot is the key to support it to complete diversified tasks; with the development of artificial intelligence technology, data-driven methods such as reinforcement learning have shown significant potential in the field of robot motion planning and control, providing a new way to realize highly adaptive and intelligent robot behavior.

[0003] The Chinese invention patent with publication number CN120428573B discloses a humanoid robot control system and method based on reinforcement learning, which includes a first control subsystem and a second control subsystem; the first control subsystem includes a policy inference module, a state transition module and a robot control module; the second control subsystem includes a data acquisition module and a drive control module; the data acquisition module is used to complete the timestamp alignment and outlier filtering of sensor data, and transmit to the state transition module; the state transition module is used to fuse multi-source sensor data and construct a time sequence state feature containing real-time measurement value and historical time sequence information; the policy inference module is used to generate the multi-joint angle target value of the robot according to the time sequence state feature provided by the state transition module; the robot control module is used to analyze the multi-joint angle target value output by the policy inference module, select the control mode, and generate the control command including the parameter adjustment instruction.

[0004] It is worth noting that the development of cloud computing and large-scale Internet platforms provides an essential data foundation, distributed simulation environment and efficient model iteration capability for reinforcement learning training, making it possible to train and verify complex strategies; current research and practice are committed to combining the autonomous decision-making ability of reinforcement learning with the inherent high-dimensional, nonlinear and unstable dynamics characteristics of humanoid robots, in order to improve the task performance of robots in unstructured environments; in this process of technology integration, how to ensure that the learning process and the generated control strategy always meet the safety constraints of the physical structure of the robot, and realize smooth migration and reliable deployment from virtual training to entity execution, has become a key research topic to promote the mature application of this field. SUMMARY

[0005] The present application aims to solve the problems in the background art and proposes a humanoid robot control method and system based on reinforcement learning.

[0006] The technical solution of the present application is a humanoid robot control method based on reinforcement learning, which includes the following specific implementation steps:

[0007] S1, construct and dynamically update a structural experience-based reachable state space, introduce structural stability margin and structural experience weight to hierarchically depict the reachability of robot motion by fusing multi-dimensional real-time states of joint configuration, driving load, and attitude stability, and form a state space that is continuously updated during the running process as the basis for safety constraints in control and reinforcement learning process;

[0008] S2, generate a basic control trajectory based on the structural experience-based reachable state space, preferentially guarantee the structural safety, motion continuity, and state recoverability of humanoid robot motion, form a long-term executable basic control trajectory by selecting task-related attitude, constructing state-level trajectory skeleton, introducing structural margin modulated time parameterization mechanism, and verifying recoverability;

[0009] S3, perform local motion correction based on the basic control trajectory, limit the motion correction of reinforcement learning to the safe neighborhood defined by the structural experience-based reachable state space, design a reward function by fusing structural stability margin and motion correction amplitude to guide the learning process, so that the optimization exploration of reinforcement learning is constrained in the safe neighborhood, and output a control trajectory with optimized local performance without damaging the overall stability;

[0010] S4, real-time perceive the execution state and environmental information of the robot through multi-source sensors, combine task error evaluation and environmental constraints, perform online closed-loop feedback correction on the control trajectory corrected by reinforcement learning, and simultaneously update the structural experience weight and reinforcement learning strategy using the data collected during the execution process, to realize long-term adaptive optimization of the system.

[0011] Preferably, in step S1, the structural experience-based reachable state space is constructed and dynamically updated, specifically including:

[0012] Construct a structural state vector that uniformly describes the current joint angle, joint angular velocity, joint driving load, joint coupling relationship, and overall attitude state of the robot;

[0013] Based on the structural state vector, a unified structural stability margin function is constructed from three dimensions of joint travel safety distance, driving load safety margin, and overall robot attitude safety distance, and the function value is used to continuously quantify the safety margin of the current state from the structural instability boundary;

[0014] Introduce a structural experience weight that evolves over time for each structural state, and through an experience weight update function, accumulate the reliability of the structural state based on long-term execution feedback, so as to distinguish between theoretically calculated reachable states and actually long-term verified reliable states;

[0015] The structure state is screened by combining the preset minimum stability margin threshold and the minimum experience weight threshold, a structure experience type reachable state space is formed, and the boundary of the space is dynamically corrected during the operation of the robot.

[0016] Preferably, in step S2, the basic control trajectory is generated, specifically including:

[0017] The current task target is associated and matched with the structure experience type reachable state space, a task related posture candidate set that meets the structure reachability and conforms to the task function requirement is screened from the structure experience type reachable state space through a task deviation measurement function, and a task related posture candidate set is formed;

[0018] On the basis of the task related posture candidate set, a number of key structure states are selected and sequentially connected according to the joint configuration change amplitude constraint and the driving load continuity constraint, and a state level trajectory skeleton is constructed;

[0019] A time modulation mechanism driven by the structure stability margin is introduced for the state level trajectory skeleton, the execution speed of the trajectory in different sections is dynamically adjusted through a time modulation factor, the execution rhythm is automatically slowed down in the section with lower structure margin, and a time parameterized basic control trajectory is generated;

[0020] The structure stability margin of the generated basic control trajectory is overall evaluated within its complete execution cycle, the long-term execution reliability is verified through a recoverability evaluation index, and the verified trajectory is solidified as a reference benchmark for subsequent reinforcement learning correction.

[0021] Preferably, in step S3, the reinforcement learning local action correction is performed, specifically including:

[0022] A local correctable action space is defined, the action correction vector output by the reinforcement learning algorithm at each step is constrained in the space, and the correction amplitude is limited to not exceed the maximum allowed value determined by the joint capacity and the driver performance, so as to ensure that the corrected action is located in the safe neighborhood of the structure experience type reachable state space;

[0023] A reward function is constructed, which combines a task reward item directly reflecting the task completion degree, a structure stability margin item encouraging state safety, and an action smoothness penalty item inhibiting excessive correction, and the optimization direction of the reinforcement learning is guided by adjusting the weight coefficients of each item;

[0024] Under the constraints of the local correctable action space and the guidance of the reward function, a reinforcement learning strategy is trained, which outputs local action correction according to the current state and the basic trajectory, and the structure experience weight is integrated in the training process to amplify the influence of long-term reliable state;

[0025] The action modification output by the reinforcement learning strategy is superimposed and integrated with the base control trajectory to form a locally modified control trajectory, and an online gradient updating strategy is designed to dynamically fine-tune the modification, ensuring that the entire modification process always satisfies the structural safety boundary.

[0026] Preferably, in step S4, the closed-loop feedback and long-term optimization specifically include:

[0027] During task execution, joint angles, joint angular velocities, joint torques, center-of-mass positions, center-of-mass velocities, contact forces, task target positions, and environmental state information are collected and fused in real time to form a multi-dimensional real-time execution state vector.

[0028] Based on the real-time execution state vector, a comprehensive task error function is calculated, which quantifies the deviation of the current state from the desired state in terms of center-of-mass position, center-of-mass velocity, end target position, contact force, and environmental constraints.

[0029] According to the gradient of the comprehensive task error function, the gradient of the environmental constraint, and the locally modified trajectory obtained in step S3, a closed-loop correction control quantity is calculated in real time, and this correction quantity is fused with the reinforcement learning correction action according to a certain coefficient to generate the final executable control trajectory.

[0030] Using all the collected data, the structure experience weight and the reinforcement learning strategy are updated synchronously through the update function to realize the closed loop of experience accumulation and strategy optimization.

[0031] Preferably, the structure stability margin function is specifically defined as the minimum value of the joint travel safety distance function, the drive load safety margin function, and the overall posture safety distance function.

[0032] Preferably, the experience weight update function is specifically: the sum of the experience weight value of the last control period and the product of an experience update coefficient and the current execution feedback score.

[0033] The execution feedback score is calculated by integrating the posture recovery time, joint vibration amplitude, drive temperature rise trend, and energy consumption change.

[0034] Preferably, when constructing the state-level trajectory skeleton, the norm of the difference between the joint angle vectors of two adjacent key states should not exceed the preset joint angle continuity threshold, and the norm of the difference between the joint load vectors of two adjacent key states should not exceed the preset joint load continuity threshold.

[0035] Preferably, the weight coefficient of the structural stability margin term in the reward function is set according to the priority requirement of safety of the specific task, and the weight coefficient of the action smoothness penalty term is set according to the maximum instantaneous speed allowed by the robot joint and the load response capability of the driver.

[0036] The technical scheme of the present application: a humanoid robot control system based on reinforcement learning, which is used to execute the above-mentioned humanoid robot control method based on reinforcement learning, comprising:

[0037] The structural experience reachable space construction module is used to collect the joint angle, joint speed, load and center of mass information of the humanoid robot, and combine the historical task execution experience to construct and dynamically update the structural experience type reachable state space.

[0038] The basic trajectory generation module is used to generate the basic control trajectory meeting the requirements of structural safety, continuity and recoverability based on the structural experience type reachable state space and the task target.

[0039] The reinforcement learning local correction module is used to combine the constraints of the structural experience type reachable state space and the safety reward function under the framework of the basic control trajectory, and utilize the reinforcement learning algorithm to locally optimize the action, and output the corrected action.

[0040] The task closed loop and feedback correction module is used to evaluate the task execution error and environmental constraints based on the real-time sensing information of the multi-source sensor, generate the closed loop correction control quantity, and fuse it with the corrected action to form the final execution trajectory, and update the experience weight in the structural experience reachable space construction module and the strategy in the reinforcement learning local correction module.

[0041] Compared with the prior art, the above technical scheme of the present application has the following beneficial technical effects:

[0042] The application designs a humanoid robot control method and system based on reinforcement learning, first, by constructing a structure experience type reachable state space, the real-time state of robot joints, load, posture and long-term execution experience are deeply fused, forming a dynamically updated safe action boundary, thereby significantly improving the structural stability and motion reliability of the humanoid robot in the complex task execution process, effectively avoiding the instability, overload or mechanical damage problem caused by blind exploration; secondly, by constraining the action correction of reinforcement learning within the experience reachable safe neighborhood, and designing a reward function integrating structure margin and task target, the local action can be autonomously optimized under the premise of ensuring system safety, enhancing the adaptability of the robot to dynamic environment and multi-task conditions; in addition, the application has good system compatibility and expandability, not only can be embedded in existing industrial control systems to realize collaborative work with production line equipment and scheduling systems, but also can perform remote strategy update, experience sharing and collaborative learning through the Internet platform, supporting knowledge transfer and continuous optimization of multiple robots in different scenarios; finally, through real-time perception, online correction and long-term experience accumulation, the closed-loop control system realizes the effective balance between task execution efficiency, motion safety and system adaptability, providing key technical support for reliable deployment of humanoid robots in intelligent manufacturing, service collaboration and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A method flowchart of a humanoid robot control method based on reinforcement learning is proposed for the application.

[0044] Figure 2 A system architecture diagram of a humanoid robot control system based on reinforcement learning is proposed for the application. DETAILED DESCRIPTION

[0045] Embodiment one, as shown in the figure, the application proposes a humanoid robot control method based on reinforcement learning, including the following specific implementation steps: Figure 1

[0046] S1, by constructing a structure state expression method integrating joint configuration, driving load, posture stability and long-term execution feedback, introduce stability margin and experience weight to grade the reachable action of the robot, form a structure experience type reachable state space which is continuously updated during the running process, provide a unified and reliable action constraint basis for subsequent control and reinforcement learning, the specific implementation process is as follows:

[0047] ​S11, by unifying the joint angle, joint speed, driving load, joint coupling relationship and overall posture and other real perceptible structure information into a structure state vector, the configuration, stress and stability of the robot at any time are synchronously described, a state basis close to the actual execution condition is provided for reachability evaluation, that is, starting from the actual robot operation, a structure state expression mode facing execution reliability is constructed, and the structure state of the robot at time t is represented as: ;

[0048] Through the structure state vector, the structure achievable state of the robot at any time is uniformly mapped into the same state space;

[0049] Among them, represents the comprehensive structure state of the humanoid robot at time t, which is used to uniformly describe the current posture, motion trend, stress condition and overall stability state of the robot; represents the current angle value set of each joint of the robot at time t, which is used to reflect the overall configuration and posture form of the robot, and is directly measured by the joint position encoder; represents the speed of the change of the joint angle with time, which is used to describe the dynamic trend of the robot action, and is calculated by time difference calculation on the joint angle ; represents the equivalent driving torque or load level of each joint of the robot at the current time, which is used to depict the structure stress state, and is calculated by the drive current feedback, torque sensor or equivalent torque estimation model; represents the joint coupling state vector, which is used to describe the coordination or coupling relationship shown by multiple joints in the actual action process, reflects the structure linkage characteristics, and is obtained by statistical analysis on the joint angle, angular velocity and load change in the historical action data; represents the overall posture state of the robot in space, including the center of mass position, attitude angle and support contact information, which is calculated by the inertial measurement unit, foot sensor and attitude fusion algorithm;

[0050] S12, based on the structure state vector, a unified structure stability margin index is constructed from three dimensions of joint stroke, driving bearing capacity and overall posture stability, the weakest structure factor is taken as the basis for judgment, and the state whether it has safe and executable condition is continuously quantified and described, specifically:

[0051] After obtaining the structure state vector, a structure stability margin function is introduced, which is used to measure the "safety margin" of the current state from the structure instability boundary, and the structure stability margin function is defined as:

[0052] ;

[0053] When When greater than the preset threshold, mark the state as a structure stable reachable state;

[0054] wherein, denotes a structure stability margin function, which is used to quantitatively describe the minimum safety margin of the current structure state from instability or structure limit; denotes the minimum safety distance between the current joint angle and its mechanical stroke limit, which is determined by the joint design parameters, assembly tolerances, and the current joint angle and is calculated accordingly; denotes the residual proportion or safety margin of the current joint load relative to its rated safety load, which is determined by the driver rated parameters and the current driving load and is calculated accordingly; denotes the safety distance of the robot centroid projection relative to the support area boundary, which is determined by the overall posture parameters and is calculated in combination with the foot contact information;

[0055] S13, by introducing a structure experience weight evolving over time, the states screened by the stability margin are accumulated for long-term execution reliability, so that the states that have been repeatedly verified to be stable and feasible obtain higher credibility, thereby distinguishing between theoretically reachable states and practically reliable states, specifically:

[0056] Considering that theoretical reachability does not equal long-term executability, a structure experience weight is introduced to describe the degree to which a state is "repeatedly verified to be feasible" in real execution. For each structure state, an experience weight update function is introduced:

[0057] ;

[0058] When a state continuously obtains positive feedback in multiple executions, its experience weight is improved and is regarded as a high-confidence reachable state;

[0059] wherein, denotes a structure experience weight, which is the degree to which a structure state is verified to be reliable and repeatedly executable in long-term actual execution; denotes an experience update coefficient, which is used to adjust the relative influence proportion of historical experience and the latest execution feedback in experience weight update; denotes an execution feedback score, which is used to comprehensively reflect the performance quality of the current structure state in real execution and is calculated by comprehensively calculating engineering indexes such as posture recovery time, joint vibration amplitude, driving temperature rise trend, and energy consumption change; denotes a structure state the structure experience weight value corresponding to the structure state at the previous control time or the previous execution period, which is used to characterize the degree to which the structure state is verified to be stable, reliable, and repeatedly executable in the historical execution process;

[0060] S14, screen and update the structure state by combining the stability margin threshold and the experience weight threshold, form the structure experience-driven reachable state space, and dynamically correct the space boundary during the robot operation, provide real-time constraint basis for subsequent control trajectory generation and reinforcement learning action pruning, specifically:

[0061] Integrate the structure state, stability margin and experience weight to form the final humanoid robot reachable state space used for control and learning constraints, define the reachable state space as:

[0062] ;

[0063] Among them, represents the reachable state space, that is, under the dual constraints of structure stability and long-term execution experience, the state set that the humanoid robot can be allowed to enter and execute actions; represents the minimum stability margin threshold, which is used to limit the minimum safety margin that the structure state must have, and is determined by the robot structure safety design specification and actual operation safety requirements; represents the minimum experience weight threshold, which is used to limit the minimum experience level that the state is considered "long-term reliable", and is set according to the long-term operation statistics and fault tolerance requirements of the robot;

[0064] It should be noted that the reachable state space is continuously updated during the operation of the robot, and serves as the unified constraint boundary for subsequent basic control trajectory generation and reinforcement learning action pruning.

[0065] S2, based on the structure experience type reachable state space constructed in step S1, prioritize the structural safety, continuity and recoverability of the humanoid robot action without pursuing action optimality, select task-related postures from the experience reachable state, construct a state-level trajectory skeleton, introduce a structure margin modulated time parameterization mechanism, and verify the generated trajectory for recoverability, form a long-term executable basic control trajectory, and provide a stable and reliable reference path for subsequent restricted modification of reinforcement learning, the specific implementation process is as follows:

[0066] S21, by associating and matching the current task target with the structure experience type reachable state space, select a set of posture states that meet the task function requirements from the reachable states that have been verified by long-term execution, specifically:

[0067] Based on the reachable state space output in step S1 , construct a task-related posture candidate set for the current task target G: ;

[0068] Through this screening, only the posture states that are both structurally verified to be reachable and functionally meet the task requirements are retained;

[0069] where G denotes the current task goal, such as end-effector pose or specific operation target; denotes the task-related pose candidate set, i.e., the state set satisfying the task match within the reachable state space; denotes a single state vector in the reachable state space, containing information such as robot joint angles, angular velocities, loads, joint coupling states, overall poses, etc.; denotes the task deviation metric function, describing the match degree of state with the task goal G, defined according to the task requirement, such as end-effector pose error, pose error, or support state error, etc.; denotes the task deviation threshold, used to filter states matching the task goal, set according to the task accuracy requirement and robot capability experience;

[0070] S22, on the basis of the task pose candidate set, according to the joint configuration change amplitude and the driving load continuity constraint, select several structural states to construct the state-level trajectory skeleton, so that the robot action always maintains smooth joint change and controllable load change in the state transition process, thereby avoiding the implicit instability problem caused by state jumping, specifically:

[0071] construct a state-level trajectory skeleton composed of stable states, select several key states from the candidate set to form a state sequence: ;

[0072] and satisfy the following structural continuity constraints: ;

[0073] where denotes the trajectory skeleton, i.e., the state sequence composed of several key poses; denotes the jth key state in the trajectory skeleton; denotes the joint angle vector of the jth key state; denotes the joint load vector of the jth key state; denotes the joint angle continuity threshold, i.e., the maximum allowed change of joint angle between adjacent states, set according to the maximum angular velocity, control period, and mechanical characteristics; denotes the joint load continuity threshold, i.e., the maximum allowed change of joint load between adjacent states, set according to the rated load of the driver and the thermal response characteristics;

[0074] S23, for the constructed state-level trajectory skeleton, introduce a time parameterization mechanism driven by structural stability margin, automatically slow down the trajectory execution pace in the section with low structural margin, and maintain normal execution speed in the section with sufficient margin, thereby improving the stability and execution reliability of the overall action without changing the trajectory structure, specifically:

[0075] The time modulation mechanism driven by structural margin is introduced to automatically slow down the basic trajectory in the "tight" structure section, and the basic control trajectory is defined as:

[0076] ;

[0077] ;

[0078] When the structural margin decreases, the trajectory execution automatically slows down, thereby reducing the risk of instability;

[0079] wherein, represents the basic control trajectory output (joint target or equivalent control instruction) at time t; represents a trajectory interpolation function for generating a smooth and continuous trajectory between key states, commonly linear or spline interpolation, but the time modulation factor is modulated by the structural margin; represents a time modulation factor for dynamically adjusting the trajectory execution speed; represents a structural stability margin function for evaluating the safety of the state at time t; represents an empirical reference margin for normalizing the time modulation factor, determined by the robot experience and task safety requirements;

[0080] S24, by overall evaluation of the structural stability margin of the basic control trajectory during execution, verify the recoverability of the trajectory in the complete execution period, only the trajectory with the stability margin maintained in the safe interval in long-term execution is solidified as the effective basic control trajectory, as the only reference benchmark for subsequent reinforcement learning action correction, specifically:

[0081] Define the recoverability evaluation index of the basic trajectory:

[0082] ;

[0083] When is continuously higher than the experience threshold, it is considered that the basic control trajectory has long-term execution reliability and is solidified as the reference trajectory for subsequent reinforcement learning;

[0084] wherein, represents the overall stability evaluation index of the basic trajectory; T represents the total execution time or period of the trajectory, determined by the task requirements and the length of the trajectory.

[0085] S3, based on the basic control trajectory generated in step S2, local action correction is realized through reinforcement learning, the whole process is carried out in the safety fence of the structure experience reachable space, the reward function is designed in combination with the structure stability margin and the correction amplitude constraint, so that the reinforcement learning optimization only explores in the safe neighborhood, and finally an executable control trajectory which is both robust and adaptive to environmental changes is output, task performance improvement and structure safety are realized, and the specific implementation process is as follows:

[0086] S31, based on the basic trajectory output in step S2, define the local correctable action space, limit the reinforcement learning output correction action to fall within the structure experience reachable neighborhood, and the correction amplitude does not exceed the allowable value of joint and load, ensure the safety of exploration and avoid destroying stability, provide controlled action boundary for reinforcement learning, specifically:

[0087] Define the local correction action space, strictly constrain the learning range in the neighborhood of the reachable state, that is, define the reinforcement learning action correction vector:

[0088] ;

[0089] Wherein, represents the action correction vector output by reinforcement learning at time t, which is used to locally adjust the basic trajectory output in step S2; represents a set of safe action spaces, which defines the allowed correction range, calculated from the reachable state space output in step S1 and joint / load limit; represents the maximum amplitude of action correction, that is, the maximum amount of basic trajectory that can be modified by reinforcement learning each time, which is determined by the maximum instantaneous speed allowed by the robot joint, the load response capability of the driver, the control period and other engineering parameters; represents the action correction amplitude of the basic control trajectory output in step S2 by reinforcement learning at time t, specifically the Euclidean norm of the vector;

[0090] S32, the task completion degree is fused with the structure stability margin and the correction amplitude constraint as a reward function, the reward reflects not only the task efficiency, but also encourages action smoothing and safety, the structure margin weight and the correction penalty coefficient are determined by the robot experience and joint performance, to ensure that the reinforcement learning optimization direction is effective and does not destroy the system stability, specifically:

[0091] The structure margin is included in the reward function, and the learning direction is dynamically adjusted, and the reward function is defined as:

[0092] ;

[0093] Wherein, represents the reinforcement learning reward function value at time t, which is used to guide policy update; represents the task completion related reward, such as end position error reduction, gait balance maintenance, object grasping success, etc., which is directly defined according to the task target, and reflects the task completion degree; represents the structure stability margin function; represents the weight coefficient of the structure stability margin in the reward function, an empirical engineering value, which is set according to the safety priority requirement of the task; represents the action smoothing penalty weight, which inhibits the excessive modification of the action amplitude, and is set by the maximum speed allowed by the joint, load response capability and other engineering parameters;

[0094] S33, training the reinforcement learning strategy under the guidance of the safe action space in step S1 and the reward function in step S2, the strategy outputs a local action modification, the experience weight guides the learning to the long-term reliable state, the modification is only executed in the safe neighborhood, forming an "experience-constrained guided local optimization", which improves the task performance while ensuring the structural safety, and specifically:

[0095] The reinforcement learning training not only depends on the environment simulation, but also needs to dynamically integrate the experience weight in step S1 to improve the reliability of the modification strategy, and the action modification strategy is defined as:

[0096] ;

[0097] In the training process, the experience weight is used to amplify the reward contribution of the long-term reliable state and inhibit the learning of accidental success or high-risk actions, forming an "experience-guided reinforcement learning";

[0098] wherein, represents the reinforcement learning strategy, which outputs the probability distribution or strategy function of the action modification of the given basic trajectory ; represents the expected value of the state in the reachable state space, which is used for reinforcement learning strategy optimization;

[0099] S34, integrating the reinforcement learning modification action and the basic trajectory to form the final executable trajectory, and adapting to environmental disturbances and task changes through online gradient update, always ensuring that the action modification is within the safe range, realizing the closed-loop control of robust execution, local optimization and dynamic adaptation, and providing reliable support for subsequent task completion, and specifically:

[0100] After the reinforcement learning output modification action is integrated with the basic trajectory in step S2, the final executable trajectory is formed, which can also be dynamically fine-tuned during execution, i.e. the final execution trajectory is defined as: ;

[0101] The trajectory online update strategy is defined as: ;

[0102] The trajectory update process always satisfies: , ensuring that reinforcement learning correction does not break the structural safety boundary; represents the online learning step, used for dynamic fine-tuning of reinforcement learning correction, according to the control cycle and system response speed engineering settings; represents the gradient of the reward function with respect to the policy, used for online updating of action correction, calculated by the reinforcement learning training algorithm;

[0103] wherein, represents the final executable control trajectory, adding reinforcement learning correction to the base trajectory in step S2.

[0104] S4, through multi-source real-time perception, task error evaluation, reinforcement learning correction and online closed-loop feedback, the reinforcement learning correction trajectory of step S3 is further optimized to realize safe, adaptive and efficient closed-loop control of humanoid robots during task execution. At the same time, combined with the execution data feedback to update the experience weight and strategy, long-term adaptive optimization is realized, and the specific implementation process is as follows:

[0105] S41, the robot collects joint angle, angular velocity, joint torque, center of mass position and velocity, contact force, target position and environment state in real time during execution, forms a complete execution state vector through multi-sensor fusion, provides multi-dimensional input for error evaluation and closed-loop correction, ensures action safety and environmental adaptability, and provides data basis for policy iteration. Specifically,

[0106] During task execution, single joint or end information is not enough to ensure stability and task success rate, and real-time collection of full-body dynamic state, environmental interaction and task target information is needed to define a real-time execution state vector: ;

[0107] wherein, represents the angular position vector of each joint of the robot, measured by the joint encoder or angle sensor in real time; represents the joint angular velocity vector; represents the joint torque or driving load; represents the position vector of the center of mass of the robot in space; represents the center of mass velocity vector; represents the contact force vector of the robot end or foot; represents the task target position vector; represents the environment state information (obstacles, terrain, dynamic objects, etc.);

[0108] S42, calculate multi-dimensional task error based on perception data, including centroid position and velocity deviation, end target deviation, contact force deviation and environment constraint cost, unify each index by weight coefficient, realize unified evaluation of task completion and structure safety, provide reliable quantitative basis for closed-loop correction, ensure robot stable execution in dynamic environment, specifically:

[0109] Calculate task deviation based on perception data, introduce stability index and environment constraint, realize unified quantification of task execution and safety, define error function:

[0110] ;

[0111] Among them, represents the comprehensive task error function; represents the reference centroid position, which is derived from the step S3 reinforcement learning correction trajectory predicted centroid position; represents the reference centroid velocity; represents the task target reference position; represents the expected contact force; represents the environment constraint cost function, which is derived from sensor measurement obstacle distance, terrain slope, etc. 、 、 、 and represent the weight of each item of the error function, which is set according to experience value or task importance;

[0112] S43, fuse the reinforcement learning correction action of step S3 with error gradient and environment constraint gradient to generate closed-loop correction control quantity, dynamically adjust the final execution trajectory, realize task adaptive correction, action correction is strictly within the safety boundary, consider task completion efficiency, structure stability and environment adaptability, form safe and controllable online closed-loop execution trajectory, specifically:

[0113] According to the error function and the reinforcement learning correction trajectory output by step S3, real-time generate correction control quantity:

[0114] ;

[0115] Final execution action:

[0116] ;

[0117] Among them, represents the closed-loop correction action; represents the gradient of error to action, which is derived from the dynamics model or online differential calculation; represents the closed-loop gain matrix, which is set according to the joint driving capability and response speed of the robot; represents the reinforcement learning correction fusion coefficient, which is adjusted according to experience or through simulation; represents the gradient of the action under the environmental constraints, which is calculated according to the environmental sensor information; represents the environmental constraint fusion weight, which is set according to the task priority and safety requirements; represents the final execution action;

[0118] S44, the robot collects state, error and correction action data during the process, updates the experience weight of step S1 and the reinforcement learning strategy of step S3, realizes long-term adaptive closed-loop optimization, and continuously improves the task completion rate and safety of the robot in complex environment and multi-task conditions, guarantees the dynamic optimization fusion of experience guidance and reinforcement learning strategy, and specifically:

[0119] The state, error and correction action data collected during the execution process are used to update the reinforcement learning strategy of step S3 and the experience weight of step S1, realize long-term adaptive optimization, and define the data updating process:

[0120] ;

[0121] ;

[0122] ;

[0123] wherein, represents the state experience weight of step S1; represents the experience weight update step, which is set according to the convergence speed and system response; represents the experience weight update amount, which is calculated by comprehensively considering the task error, structural stability margin and environmental constraint cost; represents the reinforcement learning action strategy; represents the reinforcement learning strategy update step; represents the reward function gradient, which is calculated by combining the task completion reward, structural safety margin and environmental constraint; represents the experience weight update function;

[0124] It should be noted that the experience weight update function The update amount used to calculate the state experience weights takes into account task error, structural safety margin, and environmental constraint costs. When the task error is large or the environmental constraint cost is high, the function reduces the weight of the state, reducing the priority of reinforcement learning policies for it. When the structural safety margin is large, the task error is small, and the environmental constraints are permissible, the function increases the weight of the state, encouraging policies to explore and utilize the state more, thereby ensuring structural safety and environmental adaptability while completing the task, and achieving long-term adaptive closed-loop optimization.

[0125] It should be further explained that, in actual deployment, the structural experience-based reachable state space, experience weights, and reinforcement learning strategies can be uploaded to the cloud platform for centralized management and optimization, supporting safety experience sharing and collaborative training among multiple robot systems, thereby improving the learning efficiency and safety of the system in a large-scale deployment environment.

[0126] Example 2, as Figure 2 As shown, the present invention proposes a humanoid robot control system based on reinforcement learning, which is used to execute a humanoid robot control method based on reinforcement learning proposed in Embodiment 1. It includes: a structural experience reachable space construction module, a basic trajectory generation module, a reinforcement learning local correction module, and a task closure and feedback correction module.

[0127] The structural experience-based reachable space construction module is responsible for collecting information on the humanoid robot's joint angles, joint velocities, loads, and center of mass. Combined with historical task execution experience and dynamic models, it constructs and calibrates the robot's safe reachable motion and state spaces, generating a multi-dimensional set of state constraints. By integrating joint constraints, structural stability margins, and load capacity, it establishes a reachable state library, providing safe constraint boundaries for subsequent trajectory generation and reinforcement learning. The state library is dynamically updated during system operation to reflect environmental changes and structural health, ensuring that the robot's actions are always performed within a safe and reliable range. This module can also interface with a cloud-based big data platform to achieve the aggregation and sharing of multi-robot experience data, improving the system's adaptability in complex task scenarios.

[0128] The basic trajectory generation module generates a preliminary executable trajectory based on the reachable state space and task planning instructions, providing the robot with a task skeleton. It decomposes the target task into a temporal sequence of motion points and joint postures, considering centroid stability, end-effector target position, and contact force distribution. Through a dynamic model, it predicts the robot's stability risks during execution and generates a basic control trajectory. This trajectory satisfies both task completion requirements and structural experience constraints, providing a reference and constraint benchmark for reinforcement learning corrections, ensuring the robot remains in a safe and controllable state during execution. This module supports task scheduling and trajectory collaborative optimization based on a cloud platform, making it suitable for multi-robot collaborative operation scenarios.

[0129] The reinforcement learning local correction module combines structural experience reachable space and task feedback in the framework of the basic trajectory, and uses a reinforcement learning algorithm to locally optimize the action. By constructing a safe action neighborhood to limit the reinforcement learning exploration space, a reward function is designed to combine task completion indicators, structural stability margins, and action smoothness to generate a corrected action. The reinforcement learning output can be adjusted in real time according to different task priorities and environmental disturbances. Through local optimization, the performance of the action is improved, while the structural safety is maintained, ensuring the stability and adaptability of the robot in a complex environment. This module uses a combination of cloud training and edge execution. Large-scale policy training is performed using cloud computing resources, and the optimized model is distributed to the robot for local execution.

[0130] The task closed loop and feedback correction module performs online error evaluation on the basic trajectory and the reinforcement learning correction trajectory based on real-time sensing of joint state, center of mass position, velocity, contact force and environmental information by the robot sensors. Task completion deviation and environmental constraint deviation are calculated, and a correction control quantity is generated through a closed loop control strategy. The correction control quantity is combined with the reinforcement learning correction action to form the final execution trajectory. At the same time, execution data is collected for experience weight and reinforcement learning strategy update, realizing long-term adaptive optimization. The robot can dynamically respond to environmental changes during task execution, and the corrected trajectory remains efficient, smooth and safe, forming a complete adaptive closed loop control system. This module has Internet of Things data access capability, can integrate multi-source perception data and perform big data analysis through a cloud platform, further optimizing the closed loop control strategy and improving the overall performance of the system in an Internet and cloud computing service environment.

[0131] The embodiments of the application are described in detail above in conjunction with the drawings, but the application is not limited thereto. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the application.

Claims

1. A humanoid robot control method based on reinforcement learning, characterized by, The specific implementation steps include the following: S1. Construct and dynamically update a structural experience-based reachable state space. By integrating multi-dimensional real-time states and long-term execution feedback of joint configuration, driving load, and posture stability, introduce structural stability margin and structural experience weight to hierarchically characterize the reachability of robot actions, forming a state space that is continuously updated during operation, serving as the basis for safety constraints in the control and reinforcement learning process. Specifically, constructing and dynamically updating the empirically reachable state space of the structure includes: Construct a structural state vector that uniformly describes the robot's current joint angles, joint angular velocities, joint drive loads, joint coupling relationships, and overall posture state. Based on the structural state vector, a unified structural stability margin function is constructed from three dimensions: joint stroke safety distance, driving load safety margin, and robot overall posture safety distance. The safety margin of the current state from the structural instability boundary is continuously quantified by the value of this structural stability margin function. For each structural state, a time-evolving structural empirical weight is introduced. Through an empirical weight update function, the reliability of the structural state is accumulated based on long-term execution feedback, thereby distinguishing between theoretically calculated reachable states and actual long-term verified reliable states. The structural states are screened by combining the preset minimum stability margin threshold and the minimum empirical weight threshold to form a structural empirical reachable state space, and the boundary of the structural empirical reachable state space is dynamically corrected during robot operation. S2. Generate a basic control trajectory based on the structural experience-based reachable state space, prioritizing the structural safety, motion continuity, and state recoverability of the humanoid robot's actions. By selecting task-related postures, constructing a state-level trajectory skeleton, introducing a time parameterization mechanism for structural margin modulation, and verifying recoverability, a long-term executable basic control trajectory is formed. S3. Based on the basic control trajectory, reinforcement learning local action correction is performed. The action correction of reinforcement learning is restricted to the safe neighborhood defined by the structural empirical reachable state space. By integrating the structural stability margin and the action correction magnitude, a reward function is designed to guide the learning process. This ensures that the optimization exploration of reinforcement learning is constrained within the safe neighborhood, and the output control trajectory with optimized local performance does not destroy the overall stability. S4. By using multi-source sensors to perceive the robot's execution status and environmental information in real time, and combining task error assessment and environmental constraints, the control trajectory corrected by reinforcement learning is corrected online through closed-loop feedback. At the same time, the data collected during the execution process is used to update the structural experience weights and reinforcement learning strategies to achieve long-term adaptive optimization of the system.

2. The humanoid robot control method based on reinforcement learning according to claim 1, characterized in that, In step S2, the basic control trajectory is generated, specifically including: The current task objective is associated and matched with the structural empirical reachable state space. A task deviation metric function is used to filter the attitude states from the structural empirical reachable state space that satisfy both structural reachability and task functional requirements, forming a task-related attitude candidate set. Based on the task-related attitude candidate set, and according to the constraints of joint configuration change amplitude and driving load continuity, several key structural states are selected and connected in sequence to construct a state-level trajectory skeleton. A time modulation mechanism driven by structural stability margin is introduced into the state-level trajectory skeleton. The execution speed of the trajectory in different segments is dynamically adjusted by a time modulation factor. The execution pace is automatically slowed down in segments with low structural margin, and a time-parameterized basic control trajectory is generated. The structural stability margin of the generated basic control trajectory is evaluated as a whole over its complete execution cycle. Its long-term execution reliability is verified by a recoverability evaluation index, and the verified trajectory is solidified as a reference benchmark for subsequent reinforcement learning correction.

3. The humanoid robot control method based on reinforcement learning according to claim 2, characterized in that, In step S3, reinforcement learning local action correction is performed, specifically including: Define a locally correctable action space, constrain the action correction vector output by the reinforcement learning algorithm at each step within this locally correctable action space, and limit its correction magnitude to no more than the maximum allowable value determined by the joint capability and actuator performance, ensuring that the correction action is within the safe neighborhood of the structural empirical reachable state space. Construct a reward function that integrates a task reward term that directly reflects the task completion degree, a structure stability margin term that encourages state safety, and an action smoothing penalty term that suppresses excessive corrections. Adjust the weight coefficients of each term to guide the optimization direction of reinforcement learning. Under the constraints of the locally modifiable action space and the guidance of the reward function, a reinforcement learning policy is trained. This policy outputs local action corrections based on the current state and the base trajectory, and structural experience weights are incorporated during training to amplify the influence of long-term reliable states. The action correction output by the reinforcement learning strategy is superimposed and integrated with the basic control trajectory to form a locally corrected control trajectory. An online gradient update strategy is designed to dynamically fine-tune the action correction to ensure that the entire correction process always meets the structural safety boundary.

4. The humanoid robot control method based on reinforcement learning according to claim 3, characterized in that, In step S4, closed-loop feedback and long-term optimization specifically include: During task execution, information such as joint angles, joint angular velocities, joint torques, center of mass position, center of mass velocity, contact forces, task target position, and environmental status is collected and fused in real time to form a multi-dimensional real-time execution state vector. Based on the real-time execution state vector, a comprehensive task error function is calculated. This comprehensive task error function quantifies the deviations between the current state and the expected state in terms of centroid position, centroid velocity, end target position, contact force, and environmental constraints. Based on the gradient of the comprehensive task error function, the environmental constraint gradient, and the locally corrected trajectory obtained in step S3, a closed-loop correction control quantity is calculated in real time, and this correction quantity is fused with the reinforcement learning correction action by a certain coefficient to generate the final executable control trajectory. By utilizing all the collected data, the structural experience weights and reinforcement learning strategies are updated synchronously through an update function, thus achieving a closed loop of experience accumulation and strategy optimization.

5. The humanoid robot control method based on reinforcement learning according to claim 1, characterized in that, The structural stability margin function is specifically defined as the minimum value among the calculated results of the joint travel safety distance function, the drive load safety margin function, and the overall attitude safety distance function.

6. The humanoid robot control method based on reinforcement learning according to claim 1, characterized in that, The empirical weight update function is specifically the sum of the empirical weight value of the previous control cycle, an empirical update coefficient, and the current execution feedback score. The execution feedback score is calculated by comprehensively considering multiple practical engineering indicators, including posture recovery time, joint vibration amplitude, driving temperature rise trend, and energy consumption changes.

7. The humanoid robot control method based on reinforcement learning according to claim 2, characterized in that, When constructing a state-level trajectory skeleton, the norm of the difference between joint angle vectors between two adjacent key states must not exceed a preset joint angle continuity threshold, and the norm of the difference between joint load vectors between two adjacent key states must not exceed a preset joint load continuity threshold.

8. The humanoid robot control method based on reinforcement learning according to claim 3, characterized in that, The weighting coefficient of the structural stability margin term in the reward function is set according to the priority requirements of safety for the specific task, and the weighting coefficient of the motion smoothness penalty term is set according to the maximum allowable instantaneous speed of the robot joint and the load response capability of the actuator.

9. A humanoid robot control system based on reinforcement learning, used to execute the humanoid robot control method based on reinforcement learning as described in any one of claims 1 to 8, characterized in that, include: The structural experience reachable space construction module is used to collect information on the joint angles, joint velocities, loads, and center of mass of the humanoid robot, and combine it with historical task execution experience to construct and dynamically update the structural experience-based reachable state space. The basic trajectory generation module is used to generate basic control trajectories that meet the requirements of structural safety, continuity and recoverability based on the structural experience-based reachable state space and task objectives. The reinforcement learning local correction module is used to optimize the action locally within the framework of the basic control trajectory, combining the constraints of the structural empirical reachable state space and the safety reward function, and output the corrected action using the reinforcement learning algorithm. The task closed-loop and feedback correction module is used to evaluate task execution errors and environmental constraints based on real-time sensing information from multiple sources of sensors, generate closed-loop correction control variables, and fuse them with correction actions to form the final execution trajectory. At the same time, it updates the experience weights in the structural experience reachable space construction module and the strategies in the reinforcement learning local correction module.

Images