Humanoid robot control method based on character interaction and related equipment

Abstract

The invention discloses a humanoid robot control method based on character interaction and related equipment, and can be applied to the technical field of artificial intelligence. According to the invention, the figure interaction control strategy learning under the preset figure interaction task is carried out on the humanoid robot based on the reference motion track data and the enhanced motion track data; in the learning process, the reward weight of the inner-ring character interaction control strategy learning process is updated through the outer-ring automatic reward learning process according to a training task formed by the inner-ring character interaction control strategy learning process; a figure interaction control strategy is learned through an inner-ring figure interaction control strategy learning process according to the reward weight updated in an outer-ring automatic reward learning process, so that the humanoid robot can update the reward weight based on a figure interaction scene when learning the figure interaction control strategy; and physical simulation and deployment are carried out on the to-be-simulated figure interaction control strategy, so that the robot can accurately execute a complex interaction task, and the tracking precision of the robot is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a humanoid robot control method and related equipment based on human-computer interaction. Background Technology

[0002] In related technologies, the movement of humanoid robots mimics human actions, such as walking, running, and simple arm movements. Human-robot interaction refers to the perception, understanding, and generation of "how humans interact with objects," encompassing both identifying "who, when, where, and how they contact or manipulate what," and generating human or robot interaction actions that conform to physical and functional constraints based on goals or demonstrations. Current motion control for humanoid robots largely focuses on the robot itself, such as dancing, wearing work clothes, and walking, making it difficult for humanoid robots to perform complex tasks involving object interaction. Furthermore, existing motion tracking methods typically employ empirical, fixed reward weights to calculate the total reward mechanism, resulting in learned tracking strategies that cannot accurately track in complex environments.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a humanoid robot control method and related equipment based on human-human interaction, which can effectively improve the tracking accuracy of humanoid robots in complex environments and accurately perform complex tasks with object interaction.

[0005] To achieve the above objectives, one aspect of this application proposes a humanoid robot control method based on human-computer interaction, the method comprising the following steps: Acquire reference motion trajectory data of a humanoid robot under a preset human interaction task; The reference motion trajectory data is subjected to data augmentation processing to obtain augmented motion trajectory data; The humanoid robot learns a human interaction control strategy under a preset human interaction task using the reference motion trajectory data and the enhanced motion trajectory data, thereby obtaining the human interaction control strategy to be simulated. The process of learning the humanoid robot's human interaction control strategy under the preset human interaction task includes an inner-loop human interaction control strategy learning process and an outer-loop automatic reward learning process. The outer-loop automatic reward learning process updates the reward weights of the inner-loop human interaction control strategy learning process based on the training task composed of the inner-loop human interaction control strategy learning process. The inner-loop human interaction control strategy learning process learns the human interaction control strategy based on the updated reward weights from the outer-loop automatic reward learning process. Physical simulation of the humanoid robot is performed based on the interaction control strategy of the simulated character to obtain the target character interaction control strategy. The humanoid robot's physical interaction process is controlled according to the target person interaction control strategy.

[0006] In some embodiments, acquiring reference motion trajectory data of the humanoid robot under a preset human interaction task includes: Acquire motion trajectory data of a humanoid robot under a preset human interaction task to be verified; The motion trajectory data to be verified is physically verified to obtain the reference motion trajectory data.

[0007] In some embodiments, performing data augmentation processing on the reference motion trajectory data to obtain augmented motion trajectory data includes: Obtain the position offset corresponding to each of the aforementioned reference motion trajectory data; Based on the position offset, the reference centroid position data in the reference motion trajectory data is enhanced to obtain enhanced centroid position data. The reference wrist joint position data in the reference motion trajectory data is transformed to obtain the enhanced wrist joint position data in the pelvic coordinate system. In the pelvic coordinate system, the reference angle data in the reference motion trajectory data is nonlinearly solved to obtain the enhanced angle data; The enhanced center of mass position data, together with the corresponding enhanced wrist joint position data and the enhanced angle data, constitute the enhanced motion trajectory data.

[0008] In some embodiments, the inner-loop character interaction control strategy learning process includes: A first reward item is constructed based on the reference motion trajectory data or the enhanced motion trajectory data; Obtain the updated reward weights through the outer ring automatic reward learning process; Construct a total reward function based on the first reward item and the reward weight; Based on the proximal policy optimization algorithm and the total reward function, the humanoid robot learns the human interaction control strategy to be simulated.

[0009] In some embodiments, the outer-loop automatic reward learning process includes: Acquire joint change data generated during the learning process of the inner ring character interaction control strategy; Update the character interaction task data pool based on the joint change data; Based on the aforementioned character interaction task data pool, parameterized rules are learned using a reinforcement learning algorithm. Calculate the gradient objective of the parameterized rule learning process; The reward weights of the inner-loop character interaction control strategy learning process are updated according to the gradient objective.

[0010] In some embodiments, the process of controlling the physical interaction of the humanoid robot according to the target person interaction control strategy includes: Deploy the target human interaction control strategy to the control terminal corresponding to the target robot; Obtain the current human interaction task of the target robot; The target robot's interaction process when performing the current human interaction task is controlled by the target human interaction control strategy of the control terminal.

[0011] To achieve the above objectives, another aspect of this application proposes a humanoid robot control device based on human-computer interaction, the device comprising: The first module is used to acquire reference motion trajectory data of the humanoid robot under a preset human interaction task; The second module is used to perform data augmentation processing on the reference motion trajectory data to obtain enhanced motion trajectory data. The third module is used to learn a human interaction control strategy for the humanoid robot under a preset human interaction task using the reference motion trajectory data and the enhanced motion trajectory data, thereby obtaining the human interaction control strategy to be simulated. The process of learning the human interaction control strategy for the humanoid robot under the preset human interaction task includes an inner-loop human interaction control strategy learning process and an outer-loop automatic reward learning process. The outer-loop automatic reward learning process updates the reward weights of the inner-loop human interaction control strategy learning process based on the training task composed of the inner-loop human interaction control strategy learning process. The inner-loop human interaction control strategy learning process learns the human interaction control strategy based on the updated reward weights from the outer-loop automatic reward learning process. The fourth module is used to perform physical simulation of the humanoid robot based on the interaction control strategy of the target human being to be simulated, so as to obtain the interaction control strategy of the target human being. The fifth module is used to control the physical interaction process of the humanoid robot according to the target person interaction control strategy.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a humanoid robot control method and related equipment based on human-human interaction. This method acquires reference motion trajectory data of the humanoid robot under a preset human-human interaction task, performs data augmentation processing on the reference motion trajectory data to obtain enhanced motion trajectory data, and then learns a human-human interaction control strategy for the humanoid robot under the preset human-human interaction task based on the reference motion trajectory data and the enhanced motion trajectory data. During the learning process, the reward weight of the inner-loop human-human interaction control strategy learning process is updated according to the training task composed of the outer-loop automatic reward learning process and the inner-loop human-human interaction control strategy learning process. The learning process of the humanoid robot's interaction control strategy is based on the reward weights updated by the outer loop automatic reward learning process. This allows the humanoid robot to update the reward weights based on the human interaction scenario during the learning process, making the learned human interaction control strategy more consistent with the actual human interaction scenario. Then, based on the learned human interaction control strategy, the humanoid robot is physically simulated to obtain the target human interaction control strategy. Finally, the humanoid robot's physical interaction process is controlled according to the target human interaction control strategy, thereby effectively improving the tracking accuracy of the humanoid robot in complex environments and accurately executing complex tasks with object interaction. Attached Figure Description

[0016] Figure 1 This is a flowchart of a humanoid robot control method based on human-computer interaction provided in an embodiment of this application; Figure 2 This is a flowchart of the reference motion trajectory data of the humanoid robot under a preset human interaction task provided in the embodiments of this application; Figure 3 This is a flowchart of a process for performing data augmentation on reference motion trajectory data to obtain enhanced motion trajectory data, provided in an embodiment of this application. Figure 4 This is a flowchart of the learning process of the inner-loop character interaction control strategy provided in the embodiments of this application; Figure 5 This is a complete flowchart of the learning process of the inner-loop character interaction control strategy provided in the embodiments of this application; Figure 6This is a flowchart of the outer-loop automatic reward learning process provided in the embodiments of this application; Figure 7 This is a complete flowchart of the outer-loop automatic reward learning process provided in the embodiments of this application; Figure 8 This is a flowchart of the physical interaction process of the humanoid robot controlled according to the target person's interaction control strategy, provided in an embodiment of this application. Figure 9 This is a schematic diagram of the architecture of a humanoid robot control method based on human-computer interaction provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of the humanoid robot control device based on human-computer interaction provided in the embodiments of this application; Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms used in the embodiments of this application will be explained first. The nouns and terms used in the embodiments of this application shall be interpreted as follows: A humanoid robot is a robot with a human-like physical structure, typically including a head, torso, two upper limbs, and two lower limbs. By mimicking the skeletal structure and movement patterns of humans, humanoid robots can perform tasks such as walking, running, jumping, climbing stairs, and carrying objects in complex environments, and are widely used in service, rescue, and scientific research fields.

[0022] Human-Object Interaction (HOI) is a term used in the field of humanoid robot-object interaction, such as humanoid robots interacting with objects like boxes, which involve actions like lifting, putting down, and pushing.

[0023] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm that improves training stability and sample efficiency by limiting the magnitude of policy updates. PPO is commonly used in continuous control tasks, such as robot motion control.

[0024] SAC (Soft Actor Critic) is a reinforcement learning algorithm with strong exploratory capabilities. It enhances policy exploration by adding a term that maximizes action entropy to the traditional Actor-Critic algorithm.

[0025] Meta-reinforcement learning (Meta-RL) is a reinforcement learning technique used to automatically learn specific parameters in downstream learning tasks.

[0026] Isaac Gym is an end-to-end high-performance robot simulation platform developed by NVIDIA, which primarily uses GPU acceleration to achieve reinforcement learning and robot behavior simulation.

[0027] Motion preprocessing in robot interaction refers to analyzing and processing motion information in video data to provide structured and interpretable motion representations for robot learning, thereby improving the robot's ability to perform interactive tasks.

[0028] Augmentations in robot interaction refer to methods that enhance a robot's expressive capabilities and user perception through technologies such as virtual reality (VR) and augmented reality (AR).

[0029] Multi-motion training in robot interaction refers to training a robot to generate coordinated and natural motion trajectories using interaction data from multiple motion modalities (such as vision and touch). By combining human interaction behaviors (such as gestures and body language), the robot's motion generation capabilities are optimized.

[0030] MuJoCo plays the role of a core physics simulation engine in robot interaction, focusing on simulating the dynamic behavior of multi-joint robots and realizing complex interactions between robot joints through precise contact mechanics algorithms.

[0031] Real deployment in robot interaction refers to the process of applying a robot system directly from the design stage to a real environment, involving key aspects such as model verification, code generation, and scenario execution.

[0032] Initial object offset is a fixed or configurable offset applied to the object's "reference pose / target pose" or "grasping / interaction pose" at the start of a task, based on the original pose obtained from detection or modeling, and tailored to the task requirements. It compensates for sensor calibration errors, non-geometric centering of the grasping point, inconsistencies between the tool and object coordinate systems, and differences in scene layout, allowing the robot to initiate interactions from a more suitable and safer initial pose. This offset is typically determined once during task initialization and used as a benchmark for subsequent motion planning and control.

[0033] Motion retargeting in robot interaction is a technique that maps existing reference motions (such as human motion capture data or simulated character movements) onto another body structure (such as different robot forms). Motion retargeting must take into account differences in body size, degrees of freedom, and physical feasibility to achieve natural and coordinated motion transfer.

[0034] Physical verification (or physical AI verification) refers to a systematic examination, conducted under conditions closely resembling the real physical world, to determine whether a robot's interactive system conforms to objective laws of mechanics, materials, sensing, and control, and whether it meets interactive requirements such as safety, stability, and effectiveness. Its goal is to move from "digital correctness" to "physical correctness," ensuring the transferability, robustness, and compliance of strategies in real-world environments.

[0035] The motion trajectory in robot interaction refers to the sequence of motion paths and postures planned by a robot system to complete a specific task in multi-object collaborative operations.

[0036] In robot interaction, ik-based augmentations refer to a set of techniques that use inverse kinematics (ik) as the core to "enhance" the posture, path, and constraints in the interaction process. By superimposing learned, optimized, and prior rules onto the target or constraints in Cartesian space, it enables robots to generate more natural, safer, and more human-preferred joint solutions even when faced with redundant degrees of freedom, joint limitations, and unreachable targets, thereby improving the comfort and efficiency of the interaction.

[0037] In related technologies, current motion control for humanoid robots mostly focuses on the robot itself, such as dancing, wearing work clothes, and walking, making it difficult for humanoid robots to perform complex tasks involving object interaction. Furthermore, existing motion tracking methods typically employ empirical, fixed reward weights to calculate the total reward mechanism, resulting in learned tracking strategies that cannot accurately track in complex environments.

[0038] In view of this, this application provides a humanoid robot control method and related equipment based on human-human interaction, which can effectively improve the tracking accuracy of humanoid robots in complex environments and accurately perform complex tasks with object interaction.

[0039] The humanoid robot control method based on human-computer interaction provided in this application relates to the field of artificial intelligence technology. This humanoid robot control method based on human-computer interaction can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the humanoid robot control method based on human-computer interaction, but is not limited to the above forms.

[0040] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0041] The embodiments of this application will be described in detail below with reference to the accompanying drawings: Figure 1 This is an optional flowchart of a humanoid robot control method based on human-computer interaction provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S150: Step S110: Obtain reference motion trajectory data of the humanoid robot under the preset human interaction task; Step S120: Perform data augmentation processing on the reference motion trajectory data to obtain augmented motion trajectory data; Step S130: By referring to motion trajectory data and enhanced motion trajectory data, the humanoid robot learns the humanoid interaction control strategy under the preset human interaction task to obtain the human interaction control strategy to be simulated. Step S140: Perform physical simulation on the humanoid robot based on the interaction control strategy of the target character to be simulated, and obtain the interaction control strategy of the target character. Step S150: Control the physical interaction process of the humanoid robot according to the target person's interaction control strategy.

[0042] It is understood that the preset human interaction task in this embodiment refers to the process of interaction between the robot and an object in a specific scenario. For example, the interaction between the robot and an object to be moved from the unloading point to the storage point. In this process, the robot needs to switch between multiple actions with the object, such as bending down to hug the object, lifting the object, carrying the object, bending down to place the object on a shelf, and releasing the object. The trajectory of the robot interacting with the object is different in different tasks, but the robot actions written between multiple trajectories may have the same operation mode. Therefore, this embodiment can obtain the motion trajectory data in the preset human interaction task in advance as reference motion trajectory data, and then control the robot to learn the interaction control strategy based on the reference motion trajectory, so that the robot can automatically execute human interaction tasks with similar actions in the future, thereby improving the robot's intelligent function.

[0043] Specifically, this embodiment achieves intelligent human interaction tasks in complex tasks by setting up an inner-loop human interaction control strategy learning process and an outer-loop automatic reward learning process during the humanoid robot's learning of human interaction control strategies under preset human interaction tasks. This improves the tracking accuracy of the humanoid robot in human interaction. In this embodiment, the outer-loop automatic reward learning process updates the reward weights of the inner-loop human interaction control strategy learning process based on the training task composed of the inner-loop human interaction control strategy learning process, so that the reward weights in the humanoid robot's human interaction control strategy learning process are more consistent with the specific scenario of the current human interaction task. The inner-loop human interaction control strategy learning process learns the human interaction control strategy based on the updated reward weights of the outer-loop automatic reward learning process, thereby improving the learning accuracy of the control strategy and thus improving the trajectory tracking accuracy of the humanoid robot with objects in actual human interaction tasks.

[0044] Understandably, after learning the humanoid robot's corresponding human interaction control strategy, this embodiment performs virtual simulation of that strategy. Specifically, it uses virtual simulation software to simulate the trajectory tracking accuracy of the humanoid robot during human interaction under that strategy, thereby verifying the accuracy of the strategy. Once the accuracy verification is successful, the virtual humanoid robot model with the assigned human interaction control strategy is deployed onto the physical humanoid robot. This allows the physical humanoid robot to utilize the strategy for human interaction in the physical environment, improving its tracking accuracy during human interaction tasks and enabling it to accurately complete complex tasks involving object interaction in the physical environment.

[0045] It is understandable that, such as Figure 2As shown, the process of obtaining reference motion trajectory data of a humanoid robot under a preset human interaction task in this embodiment includes, but is not limited to, steps S210 to S220: Step S210: Obtain the motion trajectory data of the humanoid robot under the preset human interaction task to be verified; Step S220: Perform physical verification on the motion trajectory data to be verified to obtain reference motion trajectory data.

[0046] Specifically, the preset human-computer interaction task in this embodiment can be a scenario set by relevant personnel based on the current need for the humanoid robot to perform a physical task. In this scenario, the motion trajectory data of the humanoid robot performing human-computer interaction can include multiple tracks, which are used as motion trajectory data to be verified. Then, the motion trajectory data to be verified is physically verified using virtual reality (VR) to verify the feasibility and execution quality of the motion trajectory data in the humanoid robot performing the physical task. The motion trajectory data to be verified after passing physical verification is used as reference motion trajectory data, thereby improving the accuracy of the humanoid robot in learning human interaction control strategies.

[0047] It is understandable that, in this embodiment, after obtaining the reference motion trajectory data, in order to improve the accuracy of the humanoid robot's learning of human interaction control strategies, the reference motion trajectory data will be enhanced to expand the motion trajectory data for the preset human interaction task. For example... Figure 3 As shown, the process of performing data augmentation on the reference motion trajectory data to obtain augmented motion trajectory data in this embodiment includes, but is not limited to, steps S310 to S350: Step S310: Obtain the position offset corresponding to each reference motion trajectory data; Step S320: Enhance the reference centroid position data in the reference motion trajectory data based on the position offset to obtain enhanced centroid position data; Step S330: Perform coordinate system transformation on the reference wrist joint position data in the reference motion trajectory data to obtain the enhanced wrist joint position data in the pelvic coordinate system; Step S340: Perform a nonlinear solution on the reference angle data in the reference motion trajectory data in the pelvic coordinate system to obtain the enhancement angle data; Step S350: Combine the enhanced center of mass position data with the corresponding enhanced wrist joint position data and enhanced angle data to form the enhanced motion trajectory data.

[0048] Specifically, the reference motion trajectory data and the corresponding enhanced motion trajectory data in this embodiment are motion trajectory data under similar tasks. The difference lies in the offset between the object positions of the motion trajectories, but the hand contact details are the same as the anchor point motion data, thereby ensuring that the humanoid robot can effectively interact with objects in the actual application process using the obtained enhanced motion trajectory data.

[0049] In this embodiment, the reference motion trajectory data is coordinate trajectory data based on the world coordinate system. This embodiment applies a position offset to the XY axis of the object in the world coordinate system. Then, inverse kinematics (IK) is used to obtain the new joint positions in the humanoid robot while maintaining the contact details between the robot's hand and the object, thereby obtaining a new motion trajectory as enhanced motion trajectory data. It is understandable that for the... Reference motion trajectory data The enhancement process, due to the original reference motion trajectory data Internal location is based on world coordinate system Therefore, this embodiment converts it into the standard pelvic coordinate system used for IK calculation. The data is used to obtain new positions of the two arm joints.

[0050] Specifically, this embodiment first starts with the reference motion trajectory data. Get Object reference centroid position data at any time Reference wrist joint position data for the left and right wrist links. and reference angle data Given a translation value Then, calculate the enhanced position data of the object and the left and right hand endpieces in the world coordinate system: ; ; In the formula, This represents the position data of the centroid of the enhanced object; It represents the amount of translation of the object's center of mass; This indicates enhanced wrist joint position data at the ends of the left and right hands.

[0051] Enhanced distal wrist, enhanced wrist joint position data Through coordinate system transformation matrix and The positions are then converted sequentially into the robot driving coordinate system (torso) and the pelvic coordinate system (pelvis), and the specific conversion process is as follows: ; Then, in this embodiment, the Ipopt nonlinear planner is called in the Pelvis coordinate system to solve for the given reference angle data. New joint angle Specifically, this can be achieved by executing the specific steps in sequence. After the bias operation, the interaction task for the same character is obtained. Enhanced motion trajectory data It is understood that in this embodiment... and These values ​​can be set to 0.15 and 25 respectively, or adjusted according to the actual scenario.

[0052] This embodiment performs data augmentation on reference motion trajectory data to obtain multiple enhanced motion trajectory data under similar tasks, thereby providing data support for the subsequent learning process of human interaction control strategies and improving learning accuracy.

[0053] It is understandable that, such as Figure 4 As shown, the inner-loop character interaction control strategy learning process in this embodiment includes, but is not limited to, steps S410 to S440: Step S410: Construct the first reward item based on the reference motion trajectory data or enhanced motion trajectory data; Step S420: Obtain the updated reward weights after the automatic reward learning process through the outer ring; Step S430: Construct the total reward function based on the first reward item and the reward weight; Step S440: Based on the near-end policy optimization algorithm and the total reward function, the humanoid robot learns the human interaction control strategy to obtain the human interaction control strategy to be simulated.

[0054] Specifically, the inner-loop human interaction control strategy learning process in this embodiment utilizes the Proximal Policy Optimization (PPO) algorithm to optimize the human interaction control strategy in the humanoid robot. In this embodiment, the optimization process involves designing large-scale rewards based on reference motion trajectory data or enhanced motion trajectory data to guide the human interaction control strategy. The optimization process aims to maximize the K-term weighted reward function. .in, This represents the state of the humanoid robot at time t; This represents the actions performed by the humanoid robot according to the control strategy at time t; The discount factor at time t is a core hyperparameter in the human interaction learning control strategy, used to measure the strategy's emphasis on future rewards. It can be understood as the total reward function during the inner-loop human interaction control strategy learning process. , Indicates the reward weight; This represents the policy function of the Actor network, which outputs the probability of an action based on state s. The first reward term. It consists of penalty terms such as the difference between the humanoid robot and object state (e.g., joint position and body link position) and motion trajectory data (reference motion trajectory data or enhanced motion trajectory data), torque, and center of gravity projection, to incentivize the humanoid robot and object to accurately track the reference trajectory.

[0055] Understandably, this is due to the empirical selection of a large set of reward weights. This approach is cumbersome and suboptimal, and it contradicts the fact that the learning focus or reward weight of the HOI task flow changes with each stage. Therefore, this embodiment introduces an outer-loop automatic reward learning process. The model is used to predict the reward weights required during the PPO training process. .in, The model can generate a set of efficient reward weights based on the learning status and progress of the PPO task. For example, at the beginning of a box-moving task, The model is aware of the urgency of balanced training and therefore uses larger balanced weights in nearby PPO rounds. To update the value network This embodiment simplifies the PPO parameter update format and target into the following formula: ; In the formula, Indicates imperfect observation Policy network The training weights; Indicates perfect observation value network The training weights; Indicates the first The character interaction task data pool in the turn-based PPO perspective.

[0056] Specifically, this embodiment is based on a physical interaction simulation process (InterMimic), therefore, this embodiment introduces an interaction graph tracking reward. This is to encourage accurate contact between humanoid robots and objects. From the features of the interaction graph roll out, This could be a distance graph representation (i.e., a state graph representation) of the humanoid robot's link position relative to key feature points of the object. Therefore, the interaction graph tracking reward... Tracking error and the interaction graph reward weight representing the current meta-policy prediction. , This represents the baseline state at time t in the interaction graph. It is understandable that, due to the difficulty in obtaining some interaction features in the real world, this embodiment employs an asymmetric Actor-Critic structure to distill the privileged features in the Actor. Specifically, the perfect Critic state consists of robot body features, gravity projection, interaction graph, and object features, while the imperfect Actor state does not involve interaction graphs or object velocity and rotation features.

[0057] It is understandable that, such as Figure 5 In the complete implementation of the inner-loop human interaction control strategy learning process shown in this embodiment, multi-motion data is composed of reference motion trajectory data or enhanced motion trajectory data, and the control strategy of the humanoid robot in the human interaction process is initialized. and reward weight Then, multi-environment interaction was conducted through a simulation platform (IsaacGym), and a meta-policy based on the outer-loop automatic reward learning process was implemented. Update reward weights The task reward is calculated and the proximal optimization policy algorithm (PPO) is iteratively trained to develop the character interaction control strategy. Once the training iterations for the inner-loop character interaction control strategy learning process reach the maximum number of training rounds, the current character interaction control strategy is output. This serves as an interactive control strategy for the simulated character.

[0058] It is understandable that, such as Figure 6 As shown, the outer-loop automatic reward learning process in this embodiment includes, but is not limited to, steps S610 to S650: Step S610: Obtain joint change data generated during the learning process of the inner ring character interaction control strategy; Step S620: Update the character interaction task data pool based on joint change data; Step S630: Based on the character interaction task data pool, perform parameterized rule learning using a reinforcement learning algorithm; Step S640: Calculate the gradient objective of the parameterized rule learning process; Step S650: Update the reward weights of the inner loop character interaction control strategy learning process according to the gradient objective.

[0059] Specifically, in this embodiment, a meta-reinforcement learning framework is designed in the outer-loop learning task to learn the optimal reward function in the Markov decision-making (MDP) problem of the HOI task. For example, this embodiment can apply this framework to each step of the inner-loop character interaction control strategy learning process. Each PPO round is defined as a potential subtask. ,Right now ,in Total reward function in each subtask Same. Then, this embodiment retrieves data from each training task buffer pool. Learning a The model's parameterization rules enable it to compute corresponding parameters for different tasks. This minimizes the impact on the test task set. PPO strategy loss during the learning process of the interaction control strategy for the upper inner ring character .

[0060] Since the outer-loop automatic reward learning process can be summarized as an MDP process, this embodiment uses the SoftActor-Critic (SAC) reinforcement learning algorithm for learning. The model's parameterization rules. This MDP treats the PPO training task as an interactive environment during the inner-loop character interaction control policy learning process. The agent observes the state... Execute an action corresponding to a reward weight in the current PPO subtask. And generate rewards related to PPO training metrics. The gradient objective of the SAC policy is as follows: ; in, The update form is as follows: ; The update operation in this embodiment can be implemented based on the initial weight. The reward weights for different PPO subtasks are explored and learned. The cutoff decay rounds are used in this embodiment. and minimum factor .

[0061] Understandably, this embodiment selects a key tracking error metric as the learning reward for the meta-policy during the outer-loop automatic reward learning process. Specifically, this embodiment uses the changes in tracking error at key locations to guide the reward weights. The exploration and optimization of meta-strategy rewards. Defined as Changes in the total tracking error of the internal joint angle position, object position, and humanoid robot link position: ; In the formula, ; ; .

[0062] Therefore, it can be seen that the state of meta-reinforcement learning in this embodiment is determined by the basic HOI task (such as...). and The learning process consists of features such as learning progress, PPO reward weights, and actions. Because the policy parameters in the outer-loop automatic reward learning process are lightweight, and the training period is interval-based... The inner loop has only one cycle, so the strategy learning of the outer loop automatic reward learning process only increases the computational cost by a small amount, but it can effectively update the reward weight of the inner loop human interaction control strategy learning process, thereby improving the accuracy of the humanoid robot in the human interaction control strategy learning process.

[0063] Based on the above, it can be seen that, Figure 7 As shown, the complete implementation process of the outer-loop automatic reward learning process in this application embodiment can be as follows: first, a task data pool is constructed based on the joint change data generated during the inner-loop character interaction control strategy learning process, and then the data pool is updated in real time during the training process. Specifically, during the training process, the meta-policy is initialized. and give reward weights Subsequently, it interacts with the reinforcement learning environment (MDP) in the proximal policy optimization (PPO) algorithm during the inner-loop character interaction control strategy learning process, collects joint change data during the outer-loop automatic reward learning process, updates the task data pool, calculates high-level rewards, and iteratively trains the meta-policy in the SAC algorithm. Once the maximum number of training steps is reached, the current meta-policy is output. and the current meta-policy Corresponding reward weight The output is fed into the inner-loop character interaction control strategy learning process to update the reward weights during the inner-loop character interaction control strategy learning process.

[0064] It is understood that in this embodiment, after iteratively training the human interaction control strategy to be simulated through the inner-loop human interaction control strategy learning process and the outer-loop automatic reward learning process to obtain the human interaction control strategy to be simulated, the accuracy of the human interaction process of the humanoid robot corresponding to the human interaction control strategy is determined by simulation testing using simulation software. After the accuracy verification is passed, the physical interaction process of the humanoid robot is controlled according to the verified post-validation target human interaction control strategy. Specifically, as shown... Figure 8 As shown, this embodiment controls the physical interaction process of the humanoid robot according to the target person interaction control strategy, including but not limited to steps S810 to S830: Step S810: Deploy the target person's interaction control strategy to the control terminal corresponding to the target robot; Step S820: Obtain the target robot's current human interaction task; Step S830: Control the target robot's human interaction process when performing the current human interaction task through the target human interaction control strategy of the control terminal.

[0065] In this embodiment, the control terminal can be a processor located inside the target robot, or it can be a cloud platform, server, or other entity that interacts with the target robot. This embodiment deploys the target human interaction control strategy to the corresponding control terminal of the target robot, enabling the control terminal to accurately control the target robot to perform the current human interaction task based on the strategy.

[0066] It is understood that the method in the embodiments of this application can be based on... Figure 9 The architecture shown executes the specific data processing procedures. Among them, Figure 9 The architecture includes a motion trajectory parameter acquisition unit, a complex task learning unit (Multi-motion Training), a physical simulation unit (Mujoco), and a deployment unit (Real Deployment). Specifically, the motion trajectory parameter acquisition unit includes a motion trajectory data preprocessing module and a motion augmentation module. The preprocessing module preprocesses the acquired motion trajectory data to be verified, such as retargeting and physical AI verification of reference motion trajectory data, thus obtaining reference motion trajectory data that conforms to actual motion conditions. The augmentation module enhances the reference motion trajectory data, obtaining enhanced motion trajectory data for multiple similar scenarios. The complex task learning unit includes an inner-loop human interaction control strategy learning process (Inner-Loop: multi-envs trials) and an outer-loop automatic reward learning process (Outer-Loop learning: N epochs). The outer-loop automatic reward learning process updates the reward weights of the inner-loop human interaction control strategy learning process based on the training task comprised of the inner-loop human interaction control strategy learning process. The inner-loop human interaction control strategy learning process then learns the human interaction control strategy based on the updated reward weights from the outer-loop automatic reward learning process. The physical simulation unit performs physical simulation of the humanoid robot based on the human interaction control strategy to be simulated. The deployment unit controls the physical interaction process of the humanoid robot according to the target human interaction control strategy to improve the tracking accuracy of the humanoid robot in complex environments.

[0067] Please see Figure 10 This application also provides a humanoid robot control device based on human-computer interaction, the device comprising: The first module is used to acquire reference motion trajectory data of the humanoid robot under a preset human interaction task; The second module is used to perform data augmentation processing on the reference motion trajectory data to obtain augmented motion trajectory data. The third module is used to learn the humanoid robot's human interaction control strategy under a preset human interaction task by referring to motion trajectory data and the enhanced motion trajectory data, thereby obtaining the human interaction control strategy to be simulated. The process of learning the humanoid robot's human interaction control strategy under the preset human interaction task includes an inner-loop human interaction control strategy learning process and an outer-loop automatic reward learning process. The outer-loop automatic reward learning process updates the reward weights of the inner-loop human interaction control strategy learning process based on the training task composed of the inner-loop human interaction control strategy learning process. The inner-loop human interaction control strategy learning process learns the human interaction control strategy based on the updated reward weights of the outer-loop automatic reward learning process. The fourth module is used to perform physical simulation of the humanoid robot based on the interaction control strategy of the target human, so as to obtain the interaction control strategy of the target human. The fifth module is used to control the physical interaction process of the humanoid robot according to the target person's interaction control strategy.

[0068] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0069] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0070] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0071] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1110 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1120 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1120 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1120 and is called and executed by the processor 1110 using the methods described in the embodiments of this application. The input / output interface 1130 is used to implement information input and output; The communication interface 1140 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1150 transmits information between various components of the device (e.g., processor 1110, memory 1120, input / output interface 1130, and communication interface 1140); The processor 1110, memory 1120, input / output interface 1130 and communication interface 1140 are connected to each other within the device via bus 1150.

[0072] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0073] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0074] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0075] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0076] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0077] This application provides a humanoid robot control method and related equipment based on human-human interaction. After acquiring reference motion trajectory data of the humanoid robot under a preset human-human interaction task, the reference motion trajectory data is augmented to obtain enhanced motion trajectory data. Then, based on the reference motion trajectory data and the enhanced motion trajectory data, the humanoid robot learns a human-human interaction control strategy for the preset human-human interaction task. During the learning process, the reward weight of the inner-loop human-human interaction control strategy learning process is updated according to the training task composed of the outer-loop automatic reward learning process. The inner-loop human-human interaction control strategy learning process learns the human-human interaction control strategy based on the updated reward weights of the outer-loop automatic reward learning process. This allows the humanoid robot to update the reward weights based on the human-human interaction scenario during the learning process, making the learned human-human interaction control strategy more consistent with the actual human-human interaction scenario. Next, based on the learned human-human interaction control strategy to be simulated, the humanoid robot undergoes physical simulation to obtain the target human-human interaction control strategy. Finally, the humanoid robot's physical interaction process is controlled according to the target human-human interaction control strategy, thereby effectively improving the tracking accuracy of the humanoid robot in complex environments and accurately executing complex tasks with object interaction.

[0078] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0079] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0082] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0085] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A character-interaction-based humanoid robot control method, characterized by, The method comprises the following steps: Obtaining reference motion trajectory data of a humanoid robot under a preset human interaction task; Performing data enhancement processing on the reference motion trajectory data to obtain enhanced motion trajectory data; Learning a human interaction control strategy of the humanoid robot under the preset human interaction task through the reference motion trajectory data and the enhanced motion trajectory data to obtain a to-be-simulated human interaction control strategy; wherein the process of learning the human interaction control strategy of the humanoid robot under the preset human interaction task comprises an inner-loop human interaction control strategy learning process and an outer-loop automatic reward learning process; the outer-loop automatic reward learning process updates a reward weight of the inner-loop human interaction control strategy learning process according to a training task composed of the inner-loop human interaction control strategy learning process; the inner-loop human interaction control strategy learning process learns a human interaction control strategy according to the reward weight updated by the outer-loop automatic reward learning process; Performing physical simulation on the humanoid robot based on the to-be-simulated human interaction control strategy to obtain a target human interaction control strategy; Controlling an entity interaction process of the humanoid robot according to the target human interaction control strategy.

2. The method of claim 1, wherein, The reference motion trajectory data of the humanoid robot under the preset human interaction task is obtained by: Obtaining to-be-verified motion trajectory data of the humanoid robot under the preset human interaction task; Performing physical verification on the to-be-verified motion trajectory data to obtain the reference motion trajectory data.

3. The method of claim 1, wherein, The data enhancement processing on the reference motion trajectory data to obtain the enhanced motion trajectory data comprises: Respectively obtaining position offset amounts corresponding to each piece of the reference motion trajectory data; Enhancing reference center-of-mass position data in the reference motion trajectory data based on the position offset amounts to obtain enhanced center-of-mass position data; Performing coordinate system conversion on reference wrist joint position data in the reference motion trajectory data to obtain enhanced wrist joint position data in a pelvis coordinate system; Performing nonlinear solving on reference angle data in the reference motion trajectory data in the pelvis coordinate system to obtain enhanced angle data; Combining the enhanced center-of-mass position data, the corresponding enhanced wrist joint position data and the enhanced angle data to form the enhanced motion trajectory data.

4. The method of claim 1, wherein, The inner-loop human interaction control strategy learning process comprises: Constructing a first reward item according to the reference motion trajectory data or the enhanced motion trajectory data; Obtaining a reward weight updated by the outer-loop automatic reward learning process; Constructing a total reward function according to the first reward item and the reward weight; Controlling the humanoid robot to learn a human interaction control strategy based on a proximal policy optimization algorithm and the total reward function to obtain the to-be-simulated human interaction control strategy.

5. The method of claim 1, wherein, The outer-loop automatic reward learning process comprises: Obtaining joint change data generated by the inner-loop human interaction control strategy learning process; Updating a human interaction task data pool according to the joint change data; Learning a parameterized rule based on a reinforcement learning algorithm according to the human interaction task data pool; Calculating a gradient target of the parameterized rule learning process; The reward weight of the inner-loop human-robot interaction control policy learning process is updated according to the gradient target.

6. The method of claim 1, wherein, The entity interaction process of the humanoid robot controlled according to the target human-robot interaction control policy comprises: deploying the target human-robot interaction control policy to a control terminal corresponding to a target robot; obtaining a current human-robot interaction task of the target robot; controlling, by the target human-robot interaction control policy of the control terminal, a human-robot interaction process of the target robot in performing the current human-robot interaction task.

7. A character-interaction-based humanoid robot control device characterized by comprising: The apparatus comprises: a first module configured to obtain reference motion trajectory data of a humanoid robot under a preset human-robot interaction task; a second module configured to perform data enhancement processing on the reference motion trajectory data to obtain enhanced motion trajectory data; a third module configured to learn a human-robot interaction control policy under the preset human-robot interaction task for the humanoid robot by using the reference motion trajectory data and the enhanced motion trajectory data, to obtain a to-be-simulated human-robot interaction control policy; wherein the process of learning the human-robot interaction control policy under the preset human-robot interaction task for the humanoid robot comprises an inner-loop human-robot interaction control policy learning process and an outer-loop automatic reward learning process; the outer-loop automatic reward learning process updates a reward weight of the inner-loop human-robot interaction control policy learning process according to a training task composed of the inner-loop human-robot interaction control policy learning process; and the inner-loop human-robot interaction control policy learning process learns a human-robot interaction control policy according to the reward weight updated by the outer-loop automatic reward learning process; a fourth module configured to perform physical simulation on the humanoid robot based on the to-be-simulated human-robot interaction control policy, to obtain a target human-robot interaction control policy; a fifth module configured to control an entity interaction process of the humanoid robot according to the target human-robot interaction control policy.

8. An electronic device, comprising: comprise: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1 to 6.

Images