A dynamic behavior generation method, system, device and medium for a humanoid robot

Abstract

The application discloses a dynamic behavior generation method, system, device and medium of a humanoid robot. The method comprises the following steps: performing dynamic modeling processing on the multi-joint dynamics characteristics of the humanoid robot to obtain a multi-joint dynamics model; acquiring the robot state of the humanoid robot, the operation state of an operator who operates the humanoid robot and environmental task information, and fusing the multi-joint dynamics model to construct a state parameter set; constructing a multi-dimensional reward function of the humanoid robot according to the multi-joint dynamics model; performing strategy optimization processing on the humanoid robot according to the multi-joint dynamics model and the multi-dimensional reward function, and outputting a dynamic game action conforming to a physical law; and performing motion control on the joints of the humanoid robot according to the dynamic game action. The embodiment of the application can improve the working efficiency of the humanoid robot and can be widely applied to the technical field of robots.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a dynamic behavior generation method, system, device and medium for a humanoid robot. BACKGROUND

[0002] In related technologies, there is a way of presetting trajectories or rules to control robots, that is, relying on accurate dynamic models to achieve motion generation through hierarchical planning. However, it is found in actual applications that the method of presetting trajectories or rules to control robots is difficult to adapt to the randomness and dynamic changes of human operations, resulting in that the humanoid robot is difficult to truly realize human-robot collaborative work in an industrial manufacturing scene, and affecting the working efficiency of the humanoid robot.

[0003] In summary, the technical problems existing in related technologies need to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a dynamic behavior generation method, system, device and medium for a humanoid robot, which can improve the control efficiency of the humanoid robot.

[0005] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a dynamic behavior generation method for a humanoid robot, which comprises:

[0006] Performing dynamic modeling processing on the multi-joint dynamics characteristics of the humanoid robot to obtain a multi-joint dynamics model;

[0007] Obtaining the robot state of the humanoid robot, the operation state of an operator operating the humanoid robot, and environmental task information, and fusing the multi-joint dynamics model to obtain a state parameter set;

[0008] According to the multi-joint dynamics model, a multi-dimensional reward function of the humanoid robot is constructed;

[0009] According to the multi-joint dynamics model and the multi-dimensional reward function, performing strategy optimization processing on the humanoid robot to output a dynamic game action conforming to physical laws;

[0010] According to the dynamic game action, performing motion control on the joints of the humanoid robot.

[0011] In some embodiments, the model modeling processing on the multi-joint dynamics characteristics of the humanoid robot to obtain a multi-joint dynamics model comprises:

[0012] Performing structure parameter acquisition processing on the robot arm of the humanoid robot to obtain robot arm parameters;

[0013] The humanoid robot is subjected to coordinate system establishment processing to obtain the link coordinate system;

[0014] The kinematic parameters are obtained by performing kinematic mapping on the robot arm parameters according to the link coordinate system.

[0015] The humanoid robot was derived using the Lagrange equations based on the kinematic parameters to obtain the multi-joint dynamic model.

[0016] In some embodiments, acquiring the robot state of the humanoid robot, the operating state of the operator operating the humanoid robot, and environmental task information includes:

[0017] The robot's state is obtained by collecting and processing data on the joint angles, angular velocities, and current torque of the humanoid robot using sensors.

[0018] The operator's hand area is identified and its speed is calculated using a visual camera to obtain the operation state;

[0019] The visual camera is used to identify the location of the target object, thereby obtaining environmental task information.

[0020] In some embodiments, the state parameter set constructed by fusing the multi-joint dynamics model includes:

[0021] The external contact force of the humanoid robot is estimated based on the multi-joint dynamics model and the operating state to obtain the predicted contact force.

[0022] The predicted contact force is added to the environmental task information, and the updated environmental task information, robot state, and operation state are processed to construct a state space to obtain the state parameter set.

[0023] In some embodiments, the step of constructing the multi-dimensional reward function of the humanoid robot based on the multi-joint dynamics model includes:

[0024] The positional deviation between the end effector of the humanoid robot and the target part is calculated based on the multi-joint dynamics model to obtain the task completion reward.

[0025] The distance between the humanoid robot and the operator is quantified based on the multi-joint dynamics model to obtain a safety reward;

[0026] The ratio of the external contact force of the humanoid robot to a preset threshold is calculated based on the multi-joint dynamics model to obtain the contact force comfort reward.

[0027] The action response time of the humanoid robot is quantified based on the multi-joint dynamics model to obtain the game-adaptive reward.

[0028] The multi-dimensional reward function is obtained by weighted summation of the task completion reward, the safety reward, the contact comfort reward, and the game adaptation reward.

[0029] In some embodiments, the step of performing strategy optimization processing on the humanoid robot based on the multi-joint dynamics model and the multi-dimensional reward function to output dynamic game actions that conform to physical laws includes:

[0030] Based on the multi-joint dynamics model, dynamic equation constraints are constructed.

[0031] A reinforcement learning environment is constructed based on the set of state parameters.

[0032] The humanoid robot is processed to output a policy based on the reinforcement learning environment to obtain the current policy.

[0033] The current strategy is updated based on the constraints of the dynamic equation and the multi-dimensional reward function to obtain the dynamic game action.

[0034] In some embodiments, updating the current strategy according to the constraints of the dynamic equation and the multi-dimensional reward function to obtain the dynamic game action includes:

[0035] The humanoid robot is processed to perform actions according to the current strategy, and the execution result is obtained.

[0036] The execution result is processed by calculating the reward value according to the multi-dimensional reward function to obtain the reward result.

[0037] The network parameters of the humanoid robot are updated based on the reward results and the constraints of the dynamic equations to obtain the target model;

[0038] The dynamic game action is obtained by outputting the target model.

[0039] To achieve the above objectives, another aspect of this application proposes a dynamic behavior generation system for a humanoid robot, the system comprising:

[0040] The dynamics model building module is used to perform dynamics modeling on the multi-joint dynamics characteristics of humanoid robots to obtain multi-joint dynamics models.

[0041] The parameter set construction module is used to obtain the robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information, and integrate the multi-joint dynamics model to construct a state parameter set;

[0042] The reward function construction module is used to construct a multi-dimensional reward function for the humanoid robot based on the multi-joint dynamics model.

[0043] The strategy optimization module is used to perform strategy optimization processing on the humanoid robot based on the multi-joint dynamics model and the multi-dimensional reward function, and output dynamic game actions that conform to physical laws.

[0044] The motion control module is used to control the joints of the humanoid robot according to the dynamic game actions.

[0045] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0046] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0047] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0048] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, device, and medium for generating dynamic behavior of a humanoid robot. This solution obtains a multi-joint dynamic model by performing dynamic modeling on the multi-joint dynamic characteristics of the humanoid robot. Based on the dynamic model, the robot's inertia, gravity, and other dynamic characteristics can be used as constraints to limit the robot's output, improving the accuracy of dynamic action generation. Furthermore, this solution acquires the robot's state, the operator's operating state, and environmental task information, and integrates them with the multi-joint dynamic model to construct a state parameter set. This allows for the fusion of the robot's body state, human operating state, and task environment information to form a high-dimensional state vector, comprehensively describing the human-robot collaboration scenario and improving the data comprehensiveness of human-robot collaboration. Moreover, this solution constructs a multi-dimensional reward function for the humanoid robot based on the multi-joint dynamic model, which can improve the safety of human-robot collaboration, enhance interactive adaptability, and improve task completion accuracy. This solution also performs strategy optimization processing on the humanoid robot based on the multi-joint dynamic model and the multi-dimensional reward function, outputting dynamic game actions that conform to physical laws. This can shorten the humanoid robot's response time to operating actions and improve collaboration efficiency. Attached Figure Description

[0049] Figure 1 This is a flowchart of a method for generating dynamic behavior of a humanoid robot according to an embodiment of this application;

[0050] Figure 2 This application provides an embodiment of a humanoid robot and an industrial operator's collaborative assembly motion trajectory diagram;

[0051] Figure 3 This is a schematic diagram of the structure of a dynamic behavior generation system for a humanoid robot provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0054] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0055] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0058] 1) Humanoid robots, also known as bionic robots, are robots designed to mimic human appearance and behavior, especially those with similar physiques to humans. The structural design of humanoid robots represents a remarkable reshaping of the human body, requiring not only interdisciplinary integration but also the culmination of cutting-edge technologies. Their design principles primarily include the following aspects: the organic integration of bionics and mechanical engineering, breakthroughs in the integration of sensing technology and control theory, and precise coordination between drive mechanisms and execution actions.

[0059] 2) Reinforcement Learning (RL) is a machine learning method. Its fundamental framework is the Markov Decision Process, which allows an agent to learn optimal policies through trial and error in its interactions with the environment. The agent performs actions in the environment and receives feedback, or rewards, based on the outcomes of those actions. These reward signals guide the agent to adjust its policy to maximize long-term cumulative rewards.

[0060] In related technologies, humanoid robots mostly rely on preset trajectories or rule-based control, which makes it difficult to adapt to the randomness and dynamic changes of human operations. For example, during human-robot collaborative assembly, sudden adjustments in human movements (such as changes in the angle at which a part is held) can lead to collision risks for the robot. Furthermore, preset trajectories cannot balance task accuracy and safe distance. For instance, if the robot continuously approaches the target position while tightening a bolt, it may collide with the hands of production line workers. There is a lack of comfortable control over human-robot contact forces, and related control methods are prone to causing damage to parts or discomfort to production line workers due to improper force application. Moreover, game-theoretic strategies have not been designed for the high safety and high precision requirements of industrial scenarios, making them difficult to directly apply to human-robot hybrid operation scenarios.

[0061] In view of this, this application provides a method, system, device, and medium for generating dynamic behavior of a humanoid robot, which can be applied to human-computer interaction application scenarios. Specifically, the method for generating dynamic behavior of a humanoid robot provided in this application can be applied to the controller of the humanoid robot or to the server controlling the humanoid robot. Taking its application to the controller as an example, it can generate specific control programs and execute specific action generation steps based on the controller. This application establishes a multi-joint dynamic model of the humanoid robot, constructs a high-dimensional state parameter set including robot state, human operation state, and environmental information, designs a reward function that integrates task accuracy, safe distance, contact comfort, and game adaptability, and uses reinforcement learning algorithms to train a deep neural network to generate the optimal game strategy. The dynamic behavior generation method for humanoid robots provided in this application enables robots to respond to changes in human actions in real time in assembly, handling, and other scenarios, avoiding collisions and improving collaborative efficiency, providing a new paradigm for dynamic behavior generation in human-machine collaboration in industrial scenarios.

[0062] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0063] Figure 1 This is an optional flowchart of a method for generating dynamic behavior of a humanoid robot provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0064] Step S101: Perform dynamic modeling on the multi-joint dynamic characteristics of the humanoid robot to obtain a multi-joint dynamic model;

[0065] Step S102: Obtain the robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information, and integrate the multi-joint dynamics model to construct a state parameter set;

[0066] Step S103: Construct the multi-dimensional reward function of the humanoid robot based on the multi-joint dynamics model;

[0067] Step S104: Perform strategy optimization on the humanoid robot based on the multi-joint dynamics model and the multi-dimensional reward function, and output dynamic game actions that conform to physical laws.

[0068] Step S105: Perform motion control on the joints of the humanoid robot according to the dynamic game action.

[0069] Steps S101 to S105 of this embodiment involve obtaining a multi-joint dynamic model by performing dynamic modeling on the multi-joint dynamic characteristics of the humanoid robot. This multi-joint dynamic model can be established by creating a physical model of the robotic arm that includes motor characteristics, transmission parameters, and multi-joint dynamic characteristics. Furthermore, the feasible domain of joint torques is determined using the multi-joint dynamic model as a constraint for action output. This embodiment obtains the robot state, the operator's operating state, and environmental task information of the humanoid robot, and integrates these with the multi-joint dynamic model to construct a state parameter set. The robot state includes robot joint motion parameters such as joint position and angular velocity; the operating state includes human-robot interaction features such as the operator's hand position; and the environmental task information includes target task parameters such as the position and filling status of the target part. The state parameter set is constructed by integrating the output of the multi-joint dynamic model with the obtained vectors. This embodiment also constructs a multi-dimensional reward function for the humanoid robot based on the multi-joint dynamic model. This multi-dimensional reward function includes quantitative indicators of task completion, human-robot safety distance, contact force comfort, and human action responsiveness. Finally, based on the multi-joint dynamics model and multi-dimensional reward function, the humanoid robot is subjected to policy optimization to output dynamic game actions that conform to physical laws. Specifically, the policy network and value network are trained by reinforcement learning algorithm, and robot dynamics constraints are incorporated into the policy optimization process to generate real-time game actions that conform to physical laws, thereby controlling the robot's joint movements to adapt to human dynamic operations.

[0070] This application embodiment constructs a multi-joint dynamics model of a humanoid robot and designs a corresponding reward function based on the model. It then combines reinforcement learning algorithms to optimize the generation of dynamic behavior of the humanoid robot, enabling the humanoid robot to respond to changes in human actions in real time in scenarios such as assembly and handling, avoiding collisions and improving collaboration efficiency. This provides a new paradigm for dynamic behavior generation for human-machine collaboration in industrial scenarios.

[0071] In step S101 of some embodiments, the process of modeling the multi-joint dynamics characteristics of the humanoid robot to obtain a multi-joint dynamics model includes:

[0072] The structural parameters of the robotic arm of the humanoid robot are collected and processed to obtain the robotic arm parameters;

[0073] The humanoid robot is subjected to coordinate system establishment processing to obtain the link coordinate system;

[0074] The kinematic parameters are obtained by performing kinematic mapping on the robot arm parameters according to the link coordinate system.

[0075] The humanoid robot was derived using the Lagrange equations based on the kinematic parameters to obtain the multi-joint dynamic model.

[0076] In this embodiment, taking a 7-DOF redundant arm of a humanoid robot as an example, the structural parameters of the robot arm are obtained, such as link mass, length, center of mass position, and moment of inertia, to obtain the robot arm parameters. Then, a coordinate system is established for the humanoid robot. MDH (Modified Denavit-Hartenberg) modeling can be used to obtain the link coordinate system. MDH is a mainstream coordinate system modeling method that constructs a coordinate system by assigning a coordinate system to each joint and defining parameters. This embodiment can also construct a base coordinate system based on the center of the humanoid robot's shoulder joint. Then, a kinematic mapping is established based on the constructed coordinate system. The homogeneous transformation matrix between adjacent links is calculated using MDH parameters, and the center of mass position and velocity are calculated based on the matrix. This embodiment can also calculate the system kinetic and potential energy of the humanoid robot, obtaining the total potential energy as gravitational potential energy. Finally, the calculated kinematic parameters are substituted into the Lagrange equation for derivation, yielding the dynamic equation expression for the multi-joint dynamic model, as shown in the following equation:

[0077] ;

[0078] in, Indicates joint position; Represents the joint angular velocity vector; Indicates joint angular acceleration; Represents the inertia matrix; Represents the matrix of Coriolis force and centrifugal force; Represents the gravity vector; This represents the frictional resistance vector (usually modeled as the sum of viscous friction and Coulomb friction). This represents the output vector.

[0079] The embodiments of this application construct a multi-joint dynamic model, which enables constraints on the output of the humanoid robot, reducing dynamic behaviors that violate dynamic constraints and improving control accuracy.

[0080] In step S102 of some embodiments, obtaining the robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information includes:

[0081] The robot's state is obtained by collecting and processing data on the joint angles, angular velocities, and current torque of the humanoid robot using sensors.

[0082] The operator's hand area is identified and its speed is calculated using a visual camera to obtain the operation state;

[0083] The visual camera is used to identify the location of the target object, thereby obtaining environmental task information.

[0084] In this embodiment, the robot's state is obtained by collecting and processing data on the joint angles, angular velocities, and current torques of the humanoid robot using sensors. These sensors can be force sensors deployed on the wrists, ankles, and other parts of the humanoid robot to collect real-time data on internal forces and torques in three-dimensional space. Alternatively, 3D vision sensors and depth cameras can provide environmental point cloud data and object recognition, which are then combined with the output of an inertial measurement unit to obtain acceleration and angular velocity data. This embodiment also uses a vision camera to capture images of the operator's hand area. A neural network model for image recognition is used to perform hand detection and segmentation, key point localization, and coordinate reconstruction to identify the hand's position. The hand's movement speed is calculated based on the difference method using consecutive frames. Furthermore, this embodiment uses a vision camera to identify the position of target objects. Target objects can include target parts requiring task processing, obstacles, and their corresponding states, such as whether the target part has been loaded.

[0085] This application embodiment constructs a state space by acquiring the robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information, which can provide a data environment for subsequent reinforcement learning.

[0086] In step S102 of some embodiments, the state parameter set is constructed by fusing the multi-joint dynamics model, including:

[0087] The external contact force of the humanoid robot is estimated based on the multi-joint dynamics model and the operating state to obtain the predicted contact force.

[0088] The predicted contact force is added to the environmental task information, and the updated environmental task information, robot state, and operation state are processed to construct a state space to obtain the state parameter set.

[0089] In this embodiment of the invention, the output of the dynamic model can be incorporated into the state space. The external contact force of the humanoid robot is estimated based on the multi-joint dynamic model combined with the operational state, and the estimation formula is shown below:

[0090] ;

[0091] In the formula, This represents the estimated external contact force; The transpose inverse of the Jacobian matrix; Indicates joint torque; The inertial force term is represented by joint angular acceleration. cause; Represents the centrifugal force term, where The matrix of Coriolis force and centrifugal force, and the joint velocity. and location Related; This represents the gravity vector. In this embodiment, the predicted contact force is added to the environmental task information, and the updated environmental task information, robot state, and operational state are processed to construct a state space. By normalizing each vector, a set of state parameters is obtained. This set of state parameters includes joint position, joint velocity, joint torque, external contact force, position-related feature vectors, velocity-related feature vectors, etc.

[0092] The embodiments of this application update the state space by constructing a dynamic model, which makes the constructed state parameter set more comprehensive and improves the accuracy of reinforcement learning.

[0093] In step S103 of some embodiments, the step of constructing the multi-dimensional reward function of the humanoid robot based on the multi-joint dynamics model includes:

[0094] The positional deviation between the end effector of the humanoid robot and the target part is calculated based on the multi-joint dynamics model to obtain the task completion reward.

[0095] The distance between the humanoid robot and the operator is quantified based on the multi-joint dynamics model to obtain a safety reward;

[0096] The ratio of the external contact force of the humanoid robot to a preset threshold is calculated based on the multi-joint dynamics model to obtain the contact force comfort reward.

[0097] The action response time of the humanoid robot is quantified based on the multi-joint dynamics model to obtain the game-adaptive reward.

[0098] The multi-dimensional reward function is obtained by weighted summation of the task completion reward, the safety reward, the contact comfort reward, and the game adaptation reward.

[0099] In this embodiment of the application, the expression of the multi-dimensional reward function is as follows:

[0100] ;

[0101] in, The total reward function is a weighted sum of multiple sub-rewards, used to evaluate the agent's performance after executing an action and guide it to learn better strategies. As a reward for task completion, a positive reward is given when the deviation between the robot's end effector and the target part is less than the preset tolerance, based on the calculated positional deviation between the robot's end effector and the target part. As a safety reward, based on human-machine distance Quantification: A penalty is triggered when the distance is less than a safety threshold; For contact force comfort reward, based on external contact force Calculation of the ratio to the safety threshold; As a game-theoretic adaptive reward, a positive reward is given when the robot's response time to sudden changes in human actions is quantified and the response time is less than a preset threshold. These represent the weight coefficients of each sub-reward item, controlling the degree of influence of different reward items on the total reward (the larger the weight, the higher the priority of the corresponding sub-reward). This application embodiment sets excessively large joint torques to the game-adaptive reward based on a dynamic model, which can promote energy saving and extend mechanical life. The expression for the game-adaptive reward is as follows:

[0102] ;

[0103] In the formula, express The weighting coefficient (0.1 here) controls the intensity of the torque penalty (the larger the value, the more severe the penalty for excessive torque). This represents the summation of torque penalty terms over the seven joints of the arm (one embodiment of the humanoid robot arm has seven degrees of freedom). This represents the "standardized squared torque" of the i-th joint, used to eliminate the influence of differences in maximum torque between different joints (unifying the penalty scale). When When the torque is small, this term is 1 (the penalty reaches unit intensity); the smaller the torque, the closer this term is to 0 (the weaker the penalty).

[0104] The embodiments of this application construct a reward function based on a dynamic model, which can reduce the collision rate through safety distance rewards, improve assembly error accuracy through task completion rewards, and shorten the robot's response time to human actions through game-theoretic adaptive rewards, thereby improving the collaborative efficiency of humanoid robots.

[0105] In step S104 of some embodiments, the step of performing strategy optimization processing on the humanoid robot based on the multi-joint dynamics model and the multi-dimensional reward function to output dynamic game actions that conform to physical laws includes:

[0106] Based on the multi-joint dynamics model, dynamic equation constraints are constructed.

[0107] A reinforcement learning environment is constructed based on the set of state parameters.

[0108] The humanoid robot is processed to output a policy based on the reinforcement learning environment to obtain the current policy.

[0109] The current strategy is updated based on the constraints of the dynamic equation and the multi-dimensional reward function to obtain the dynamic game action.

[0110] In this embodiment, dynamic equation constraints are constructed based on a multi-joint dynamics model to influence the original actions output by the neural network in the humanoid robot. After processing with dynamic constraints, the formulas for the dynamic constraints are as follows:

[0111] ;

[0112] in, For network output, This is the minimum allowable torque (lower limit) for the i-th joint, to prevent the joint from failing to drive or malfunctioning due to excessively small torque. This represents the i-th original action output by the neural network, with a value range of [-1, 1]. This is the standardized control variable directly output by the network, used to simplify training (avoiding excessively large output ranges that could lead to training instability). This refers to the actual joint torque applied. This is the maximum allowable torque (upper limit) for the i-th joint, to prevent excessive torque from damaging the joint motor or mechanical structure; This represents the scaling factor, which will affect the network output. Mapping to the [0, 1] interval enables a linear conversion from standardized motion to actual torque (when...). hour, By constructing a reinforcement learning environment from the vectors in the state parameter set, the humanoid robot obtains vector data from the reinforcement learning environment as its current state. Based on this current state, the robot generates a current policy and outputs a current action. The policy output is constrained by the dynamic equations, and a reward is calculated based on the current action using a multi-dimensional reward function. This reward is then used to update the robot's policy network, resulting in a new policy that updates the current policy. This process continues until preset conditions are met, leading to the final dynamic game action.

[0113] This application embodiment optimizes the strategy of the humanoid robot by using a multi-joint dynamics model and a multi-dimensional reward function. It can control the output of the humanoid robot based on the dynamics model and update the policy network through the multi-dimensional reward function to output more accurate action strategies, thereby enabling the humanoid robot to generate more suitable dynamic behaviors and improving the collaborative efficiency of the humanoid robot.

[0114] In some embodiments, updating the current strategy according to the constraints of the dynamic equation and the multi-dimensional reward function to obtain the dynamic game action includes:

[0115] The humanoid robot is processed to perform actions according to the current strategy, and the execution result is obtained.

[0116] The execution result is processed by calculating the reward value according to the multi-dimensional reward function to obtain the reward result.

[0117] The network parameters of the humanoid robot are updated based on the reward results and the constraints of the dynamic equations to obtain the target model;

[0118] The dynamic game action is obtained by outputting the target model.

[0119] In this embodiment, a multi-dimensional reward function is used to calculate the reward value of the execution result. The task completion reward can be set as follows: when the deviation between the robot's end effector and the target hole position is <0.5mm, When the deviation exceeds 1mm, The safety reward can be set to trigger a penalty when the human-machine distance is less than 0.3m. Reward when distance > 0.5m Comfort reward can be set as: contact force hour, When it exceeds 5N The adaptive reward for game theory can be set as follows: if the robot responds within 0.4 seconds when a production line worker suddenly adjusts the posture of a part, ,otherwise The calculated rewards and actions are stored in the experience pool. 64 samples are extracted, and the advantage function is calculated.

[0120] ;

[0121] in, This represents the advantage value at time step i, which is used to guide the parameter updates of the policy network. Represents the immediate reward at time step i; Discount factor This is used to attenuate the weight of future value, balancing near-term and long-term returns; Indicates the value network's response to the next state. Value estimates (predictions from) (Initial cumulative rewards); Represents the value network in relation to the current state. Value estimates (predictions from) (Initial average cumulative reward). This embodiment of the application employs the PPO reinforcement learning algorithm to truncate gradients and update network parameters, ensuring training stability.

[0122] The solutions of the embodiments of the present invention will be described in detail and explained below with reference to specific application examples:

[0123] This application embodiment can be applied to human-machine collaborative assembly scenarios in industrial manufacturing. First, a multi-joint dynamics model is established. This application embodiment can construct a joint drive and braking model. Specifically, the embodiment uses the wrist joint (7th joint) of a 7-axis robot as an example, and the drive model uses motor torque... With a transmission ratio of i=10, the transmission efficiency is... Given a rotation radius r = 0.05 m, the upper limit of the torque is calculated as follows:

[0124] ;

[0125] Using the braking model to achieve the maximum torque of the brake. The load weight m = 2kg, and the coefficient of friction is... The lower limit of the torque is calculated as follows:

[0126] ;

[0127] The output of a humanoid robot can be dynamically constrained based on a joint actuation and braking model. In this embodiment, the robot's state—joint angles—is obtained through data acquisition and processing using sensors. angular velocity Current torque Human operation status: Hand position is acquired via a visual camera. ,speed Task Status: Target Bolt Hole Location Position of assembled parts Obstacles such as the edge of the workbench Standardization process: This involves standardizing the joint torque. Normalization to relative distance , This indicates the robot's position. In this embodiment, the parameter is initialized as: attenuation coefficient. Experience pool capacity Time step The network architecture of the humanoid robot is as follows: the policy network adopts a 3-layer fully connected structure, outputs the mean and variance of 7-dimensional joint torques, and finally obtains the dynamic behavior through the output of the humanoid robot's policy network.

[0128] This application embodiment can be specifically applied to a human-machine collaborative assembly scenario for engine gaskets. First, state perception is achieved by using robot vision to identify the position of the production line worker's hand and the target slot of the gasket, while a force sensor detects the grasping torque (3N). Then, a game-theoretic action is generated. When the production line worker's hand approaches the robot's working area (0.25m away), the strategy model outputs a negative torque to decelerate the robot arm while maintaining the gasket positioning error <0.3mm. Please refer to [link to relevant documentation]. Figure 2 Finally, a humanoid robot provides a dynamic response: if the production line operator suddenly speeds up the placement of the pads, the robot can trigger a rapid adjustment, synchronously adjusting the gripping posture within 0.4 seconds to avoid collisions and complete the assembly.

[0129] Please see Figure 3 This application also provides a dynamic behavior generation system for a humanoid robot, which can implement the above-described method. The system includes:

[0130] The dynamic model construction module 301 is used to perform dynamic modeling on the multi-joint dynamic characteristics of the humanoid robot to obtain a multi-joint dynamic model.

[0131] The parameter set construction module 302 is used to obtain the robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information, and to integrate the multi-joint dynamics model to construct a state parameter set;

[0132] The reward function construction module 303 is used to construct a multi-dimensional reward function for the humanoid robot based on the multi-joint dynamics model.

[0133] The strategy optimization module 304 is used to perform strategy optimization processing on the humanoid robot according to the multi-joint dynamics model and the multi-dimensional reward function, and output dynamic game actions that conform to physical laws.

[0134] The motion control module 305 is used to control the motion of the joints of the humanoid robot according to the dynamic game action.

[0135] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0136] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0137] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0139] The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0140] The memory 402 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401 using the methods described in the embodiments of this application.

[0141] Input / output interface 403 is used to implement information input and output;

[0142] The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0143] Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0144] The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0145] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0146] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0148] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0150] This application provides a method, system, device, and medium for generating dynamic behavior of a humanoid robot. This solution obtains a multi-joint dynamic model by performing dynamic modeling on the multi-joint dynamic characteristics of the humanoid robot. Based on this model, the robot's inertia, gravity, and other dynamic characteristics can be used as constraints to limit the robot's output, improving the accuracy of dynamic action generation. Furthermore, this solution acquires the robot's state, the operator's operating state, and environmental task information, and integrates these with the multi-joint dynamic model to construct a state parameter set. This set can fuse the robot's state, the operator's operating state, and task environment information to form a high-dimensional state vector, comprehensively describing the human-robot collaboration scenario and improving the data comprehensiveness of human-robot collaboration. Moreover, this solution constructs a multi-dimensional reward function for the humanoid robot based on the multi-joint dynamic model, which can improve the safety of human-robot collaboration, enhance interactive adaptability, and improve task completion accuracy. The solution also performs strategy optimization processing on the humanoid robot based on the multi-joint dynamic model and the multi-dimensional reward function, outputting dynamic game actions that conform to physical laws. This can shorten the humanoid robot's response time to operating actions and improve collaboration efficiency.

[0151] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0152] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0153] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0155] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0156] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0157] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0158] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating dynamic behavior of a humanoid robot, characterized in that, The method includes the following steps: The dynamic characteristics of the multi-joint dynamics of the humanoid robot are modeled to obtain a multi-joint dynamics model. The robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information are obtained, and a set of state parameters is constructed by integrating the multi-joint dynamics model. The multi-dimensional reward function of the humanoid robot is constructed based on the multi-joint dynamics model. The humanoid robot is subjected to strategy optimization based on the multi-joint dynamics model and the multi-dimensional reward function, and the output is a dynamic game action that conforms to the laws of physics. The joints of the humanoid robot are controlled to move according to the dynamic game action. The multi-dimensional reward function for the humanoid robot, constructed based on the multi-joint dynamics model, includes: The positional deviation between the end effector of the humanoid robot and the target part is calculated based on the multi-joint dynamics model to obtain the task completion reward. The distance between the humanoid robot and the operator is quantified based on the multi-joint dynamics model to obtain a safety reward; The ratio of the external contact force of the humanoid robot to a preset threshold is calculated based on the multi-joint dynamics model to obtain the contact force comfort reward. The action response time of the humanoid robot is quantified based on the multi-joint dynamics model to obtain the game-adaptive reward. The multi-dimensional reward function is obtained by weighted summation of the task completion reward, the safety reward, the contact comfort reward, and the game adaptation reward.

2. The method according to claim 1, characterized in that, The dynamic modeling of the multi-joint dynamic characteristics of the humanoid robot is performed to obtain a multi-joint dynamic model, including: The structural parameters of the robotic arm of the humanoid robot are collected and processed to obtain the robotic arm parameters; The humanoid robot is subjected to coordinate system establishment processing to obtain the link coordinate system; The kinematic parameters are obtained by performing kinematic mapping on the robot arm parameters according to the link coordinate system. The humanoid robot was derived using the Lagrange equations based on the kinematic parameters to obtain the multi-joint dynamic model.

3. The method according to claim 1, characterized in that, The acquisition of the robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information includes: The robot's state is obtained by collecting and processing data on the joint angles, angular velocities, and current torque of the humanoid robot using sensors. The operator's hand area is identified and its speed is calculated using a visual camera to obtain the operation state; The visual camera is used to identify the location of the target object, thereby obtaining environmental task information.

4. The method according to claim 1, characterized in that, The state parameter set is constructed by integrating the multi-joint dynamic model, including: The external contact force of the humanoid robot is estimated based on the multi-joint dynamics model and the operating state to obtain the predicted contact force. The predicted contact force is added to the environmental task information, and the updated environmental task information, robot state, and operation state are processed to construct a state space to obtain the state parameter set.

5. The method according to any one of claims 1 to 4, characterized in that, The step of optimizing the strategy of the humanoid robot based on the multi-joint dynamics model and the multi-dimensional reward function to output dynamic game actions that conform to physical laws includes: Based on the multi-joint dynamics model, dynamic equation constraints are constructed. A reinforcement learning environment is constructed based on the set of state parameters. The humanoid robot is processed to output a policy based on the reinforcement learning environment to obtain the current policy. The current strategy is updated based on the constraints of the dynamic equation and the multi-dimensional reward function to obtain the dynamic game action.

6. The method according to claim 5, characterized in that, The step of updating the current strategy based on the constraints of the dynamic equation and the multi-dimensional reward function to obtain the dynamic game action includes: The humanoid robot is processed to perform actions according to the current strategy, and the execution result is obtained. The execution result is processed by calculating the reward value according to the multi-dimensional reward function to obtain the reward result. The network parameters of the humanoid robot are updated based on the reward results and the constraints of the dynamic equations to obtain the target model; The dynamic game action is obtained by outputting the target model.

7. A dynamic behavior generation system for a humanoid robot, characterized in that, The system includes: The dynamics model building module is used to perform dynamics modeling on the multi-joint dynamics characteristics of humanoid robots to obtain multi-joint dynamics models. The parameter set construction module is used to obtain the robot state of the humanoid robot, the operation state of the operator operating the humanoid robot, and environmental task information, and integrate the multi-joint dynamics model to construct a state parameter set; The reward function construction module is used to construct a multi-dimensional reward function for the humanoid robot based on the multi-joint dynamics model. The strategy optimization module is used to perform strategy optimization processing on the humanoid robot based on the multi-joint dynamics model and the multi-dimensional reward function, and output dynamic game actions that conform to physical laws. A motion control module is used to control the joints of the humanoid robot according to the dynamic game actions. The reward function construction module is used to construct a multi-dimensional reward function for the humanoid robot based on the multi-joint dynamics model, including: The positional deviation between the end effector of the humanoid robot and the target part is calculated based on the multi-joint dynamics model to obtain the task completion reward. The distance between the humanoid robot and the operator is quantified based on the multi-joint dynamics model to obtain a safety reward; The ratio of the external contact force of the humanoid robot to a preset threshold is calculated based on the multi-joint dynamics model to obtain the contact force comfort reward. The action response time of the humanoid robot is quantified based on the multi-joint dynamics model to obtain the game-adaptive reward. The multi-dimensional reward function is obtained by weighted summation of the task completion reward, the safety reward, the contact comfort reward, and the game adaptation reward.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images