A gait generation and control method for humanoid robots based on emotion perception

Abstract

This invention provides a method for gait generation and control of humanoid robots based on emotion perception, comprising: Step 1, given human-computer interaction input, performing emotion perception by extracting discrete emotional states from semantic cues, and mapping the inferred emotional states to emotion intensity and task-related physical control commands; Step 2, training a gait generator based on a constructed structured motion dataset; Step 3, generating kinematic trajectories with emotional features based on the aforementioned emotion intensity and physical control commands; Step 4, tracking the aforementioned kinematic trajectories by a unified reinforcement learning-based controller to achieve multi-emotion gait walking of the robot. This invention achieves fine quantification and smooth expression of emotion intensity, independently inputting emotion intensity and physical control commands through a decoupled parameterized interface, separating high-level gait generation from low-level joint control, significantly improving adjustability and robustness.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction and robot control, and in particular to a method for generating and controlling the gait of a humanoid robot based on emotion perception. Background Technology

[0002] Motion control of humanoid robots has always been an important research direction in the field of robotics. To achieve stable and efficient walking capabilities, researchers have proposed a variety of control methods and made significant progress at both the theoretical and applied levels.

[0003] Early research on humanoid walking primarily relied on model-based control methods. For example, linear inverted pendulum models, zero-moment point methods, and model predictive control (MMC) achieved stable gait planning and control by simplifying robot dynamics and combining them with optimization algorithms. Furthermore, hierarchical optimization methods incorporating whole-body control further enhanced multi-joint coordination. These methods offer advantages such as high stability and good physical interpretability, demonstrating reliable performance in structured environments. However, their high dependence on accurate dynamic models and manually designed contact strategies limits their adaptability and scalability in complex environments and dynamic tasks.

[0004] In recent years, with the improvement of computing power, reinforcement learning methods have gradually become mainstream. These methods directly learn control policies through large-scale interactive training in simulation environments, enabling robots to acquire stable and robust motion capabilities without the need for precise modeling. Combined with techniques such as domain randomization, reinforcement learning methods significantly improve their generalization ability in complex environments. Furthermore, imitation learning methods, by incorporating human motion capture data as a reference, enable robots to learn more natural, human-like movement patterns. Recent research has further extended this to complex full-body behaviors such as running, jumping, and dancing, achieving highly dynamic and high-degree-of-freedom motion control.

[0005] While the aforementioned methods have made significant progress in motion stability and naturalness, their core objectives remain primarily focused on trajectory tracking accuracy and physical plausibility. They typically treat the reference motion as a fixed target and lack the ability to model high-level semantic information (such as emotion and intention). A few studies have attempted to express emotion through actions, but these mostly rely on predefined actions or discrete emotion categories, making it difficult to achieve continuous and controllable emotion expression.

[0006] In motion generation, existing methods mainly fall into two categories. One category is based on explicit motion representations, typically employing a hierarchical "generate-track" framework: a high-level generator produces kinematic trajectories based on text, trajectory, or environmental information, while a low-level controller tracks them. These methods offer strong expressive power and flexibility, but due to the lack of physical constraints in the high-level generation process, they are prone to producing trajectories that do not conform to dynamic laws (such as foot slippage or body floating), thus increasing control difficulty. The other category is based on implicit motion representations, which directly generate control strategies or latent representations through end-to-end learning, thereby improving physical consistency and temporal continuity. However, because these methods involve implicit and difficult-to-interpret representations, they often struggle to achieve semantic alignment and fine-tuning of specific motion attributes (such as emotional intensity).

[0007] On the other hand, emotion perception technology has also been extensively studied in the field of human-computer interaction. Traditional methods mainly rely on visual (facial expressions) and speech signals for emotion recognition. However, in recent years, breakthroughs in natural language understanding using large language models have made it possible to extract emotional semantics from text. These methods possess good generalization and semantic understanding capabilities, enabling them to infer implicit emotional states from complex dialogues. However, existing research often treats emotion perception as an independent module applied to the interaction or decision-making layer, lacking effective integration with underlying motion control.

[0008] In summary, although existing technologies have made some progress in humanoid robot motion control, action generation, and emotion perception, the following main problems and shortcomings still exist:

[0009] (1) Existing motion control methods mainly focus on stability and physical feasibility, and lack the ability to model and express high-level semantic information such as emotions;

[0010] (2) Emotional perception and motor control are usually independent of each other and have not yet formed a closed-loop mechanism from semantic understanding to physical motor expression;

[0011] (3) Explicit motion generation methods lack physical consistency constraints, while implicit methods lack interpretability and controllability, making it difficult to support fine-grained emotion modulation.

[0012] (4) Existing methods generally lack systematic sentiment-labeled motion data and corresponding modeling methods, which limits the authenticity and diversity of sentiment expression.

[0013] Therefore, it is necessary to propose a new technical solution that deeply integrates the emotional semantics in natural language with the gait generation and control process of humanoid robots, so as to achieve continuous, controllable and expressive emotion-driven motion while ensuring physical feasibility and control stability. Summary of the Invention

[0014] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the existing technology by providing a method for generating and controlling the gait of a humanoid robot based on emotion perception, comprising the following steps:

[0015] Step 1: Given human-computer interaction input, perform emotion perception by extracting discrete emotional states from semantic cues, and map the inferred emotional states to emotion intensity. Then, emotional intensity is introduced. With physical control parameters The prior mapping between them integrates domain knowledge and physical constraints into reasoning to obtain task-related physical control instructions;

[0016] Step 2: Based on the psychological emotion classification method, the emotions expressed by humans while walking are divided into three categories: neutral, positive, and negative. The positive includes happiness and confidence; the negative includes sadness, fear, and shyness.

[0017] Eight professional dancers then performed multi-directional, multi-step walking sequences in three categories: neutral, positive, and negative. The whole-body motion data was recorded using an optical motion capture system, and a structured periodic motion dataset with emotion annotations was constructed by combining self-supervised data processing.

[0018] Step 3: Construct a gait generator. Train the gait generator based on the structured periodic motion dataset with emotion annotations constructed in Step 2. Then, based on the emotion intensity and physical control commands in Step 1, the gait generator combines periodic signals to generate periodic kinematic trajectories with emotion features. The kinematic trajectories include robot joint positions, joint velocities, and root node poses.

[0019] Step 4: A unified reinforcement learning-based controller tracks the kinematic trajectory generated in step 3 to enable the robot to walk in multiple emotional gaits.

[0020] Step 1 includes:

[0021] In the first stage, self-designed prompt words are imported to guide the large language model to act as an emotion recognition agent, and to parse the dominant emotion semantic tags (such as happiness, sadness, fear, etc.) and their corresponding intensities (i.e., the structured and quantitative results of semantic information such as degree adverbs and tone in the language) from the user's natural language commands.

[0022] In the second stage, a priori mapping relationship between artificially defined emotional intensity and physical control parameters (such as "high intensity happiness → high cadence, high speed") is injected into the cue words, transforming the large language model into a gait parameter generator. To ensure the physical feasibility of the movement, robot physical constraints are introduced, including the maximum walking speed in the x-direction. Maximum walking speed in the y direction and maximum angular velocity Ensure that the generated parameters satisfy the dynamics and actuator constraints;

[0023] By guiding a large language model to generate control parameters containing semantic and emotional information, a complete parameter set compatible with the gait generator interface is constructed, enabling diversified gait generation and precise control driven by semantic and emotional perception.

[0024] Step 2 includes: 8 professional dancers performing multi-directional, multi-step walking sequences in three categories: neutral, positive, and negative, and recording full-body motion data using an optical motion capture system, which is uniformly represented in the standardized human body model SMPL (Skinned Multi-Person Linear Model) format.

[0025] Step 2 also includes: transferring human gait data to the humanoid robot through remapping, and introducing an adaptive height correction mechanism to alleviate the problem of foot suspension: identifying support phases by detecting frames where the foot horizontal velocity and yaw angular velocity are below a threshold, and dynamically compensating for the robot's root height. The threshold is generally set to 0.2.

[0026] Step 2 further includes: performing independent kinematic mapping on the upper body and lower body separately using a segmented remapping strategy, and then fusing the results to decouple the motion trajectories. The remapped trajectory is represented as a trajectory of length [missing information]. state sequence , where the t-th frame Includes the following: Root node location Root node pose Root node linear velocity Root node angular velocity Joint angle and joint angular velocity ;in This represents the position of the root node in the x-direction of the world coordinate system. This represents the position of the root node in the y-direction of the world coordinate system. This represents the position of the root node in the z-direction of the world coordinate system. This represents the rotation angle of the root node about the x-axis. This represents the rotation angle of the root node about the y-axis. Represents the rotation angle of the root node about the z-axis; This represents the root node's velocity in the x-direction in the world coordinate system. This represents the velocity of the root node in the y-direction in the world coordinate system. This represents the velocity of the root node in the z-direction in the world coordinate system. This represents the angular velocity of the root node about the x-axis. This represents the angular velocity of the root node about the y-axis. This represents the angular velocity of the root node about the z-axis.

[0027] Step 2 also includes: Since gait data has obvious periodicity, and also facilitates the learning of the generator network, the obtained trajectory needs to be further segmented into periods. Each segment contains a complete walking cycle. :

[0028] ,

[0029] in This represents a collection of gait data from two or more cycles. N is the total number of cycles. , The first The revelation index and termination index of each segment;

[0030] The specific extraction method includes: selecting local trajectory peaks from key reference joints (such as hip joint pitch) as candidate segmentation points, and then using dynamic programming for global optimization under physical constraints to extract the longest effective peak subsequence, thereby ensuring temporal continuity and physical consistency; however, abnormal gaits deviating from normal walking patterns still exist in the segmented dataset, which need to be further filtered out by calculating the average longitudinal velocity. lateral velocity and yaw rate To extract motion features, and at the same time based on the duration of the segment Get step frequency Where fps is the frame rate, and linear regression analysis is used to determine the linear velocity norm. With step frequency The correlation between the two factors is statistically consistent with biomechanical principles, and the correlation is determined by... Modeling is performed, where k and b are the slope and intercept of the linear model, respectively. Significant outliers and low-speed unstable segments are removed to construct a physically reasonable dataset.

[0031] Ultimately, the self-supervised input feature is defined as a combination of a three-dimensional average velocity vector and a frequency offset, where the frequency offset... Calculated from regression residuals:

[0032] .

[0033] In step 3, the gait generator is constructed using the following method: the gait generator is constructed based on a structured periodic motion dataset with emotion annotation. The gait generator is a lightweight motion generation network that generates whole-body periodic motion trajectories with emotion intensity and physical control commands as input.

[0034] The gait generator is a lightweight gait generation network based on Multilayer Perceptron (MLP). Used for frame-by-frame target pose regression: input features Incorporating sinusoidal time coding Emotional intensity and physical control commands Output the whole body reference state This includes root attitude, linear velocity and angular velocity, as well as joint position and velocity;

[0035] To address the issues of subjective bias and discrete noise in manual sentiment annotation, a Transformer-based classifier is used as a label smoother to reconstruct the latent sentiment distribution: for the Each motion segment will determine the robot's state. With sinusoidal time coding The input sequence is obtained by concatenation, where It represents the phase angle of the nth motion segment at time t, which is usually driven by the periodic rhythm of the motion (such as the gait cycle) and reflects the current position in the complete motion cycle; and a classification token (CLStoken) is added before the input sequence to aggregate the global sequence representation;

[0036] By capturing global spatiotemporal dependencies through a multi-layer self-attention mechanism and suppressing annotation noise and anomalies through information bottlenecks, the output latent state corresponding to CLStoken is extracted and mapped by a normalized exponential function to obtain a continuous probability distribution in the sentiment category space as the refined soft label sentiment intensity. That is, the original discrete hard labels (one-hot vectors, such as "happy" corresponding to [1,0,0,0,0]) are transformed into probability vectors reflecting the confidence of each sentiment category (such as [0.75,0.10,0.08,0.07,0.07]), thereby constructing a continuous probability manifold in the sentiment category space, effectively alleviating the subjectivity of manual annotation and providing a more robust sentiment intensity supervision signal for subsequent gait parameter generation.

[0037] Step 4 includes: randomly sampling the emotional intensity and physical control parameters based on the gait generator, generating diverse periodic gait sequences covering two or more emotional categories and movement styles online, and constructing a long-term training dataset through cyclic expansion and splicing;

[0038] The training employs a phased strategy: First, the robot is trained on flat terrain to master stable basic walking ability and tracking skills for different emotional gaits. Then, two or more complex terrains are introduced for training, enabling the robot to maintain gait stability and tracking accuracy under different terrains, thereby improving the environmental robustness of the strategy and reducing the simulation-to-real gap. Simultaneously, the BeyondMimic method is extended to include multiple emotions and actions. By introducing large-scale, diverse gait data containing different combinations of emotion intensities and control parameters, a general strategy capable of uniformly tracking two or more emotional gaits is trained. This gives the strategy flexible generalization capabilities within a continuous emotional space and physical constraints. Each gait data point is jointly determined by physical control parameters and emotional parameters. Physical control parameters include the robot's x-axis velocity, y-axis velocity, rotational angular velocity, and step frequency offset during movement, used to control the robot's direction, speed, and gait rhythm. Emotional parameters, composed of emotion labels and emotion intensity, are used to adjust the overall style and movement performance of the gait. For example, a high intensity of sadness can be paired with control parameters such as x-direction velocity 0.5, y-direction velocity 0.0, rotational angular velocity 0.0, and step frequency offset -0.1.

[0039] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0040] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0041] The method proposed in this invention has the following beneficial effects:

[0042] (1) By modeling emotion as a continuous variable rather than an implicit semantic input, discrete labels are transformed into a continuous probability manifold through a Transformer-based label smoothing mechanism, thus achieving fine quantification and smooth expression of emotion intensity.

[0043] (2) The present invention independently inputs emotional intensity and physical control commands through a decoupled parameterized interface, and separates high-level gait generation from low-level joint control, which significantly improves the adjustability and robustness of the system.

[0044] (3) The present invention independently inputs emotional intensity and physical control commands through a decoupled parameterized interface, and separates high-level gait generation from low-level joint control, which significantly improves the adjustability and robustness of the system. Attached Figure Description

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0046] Figure 1 This is a flowchart illustrating the method of the present invention.

[0047] Figure 2 Build a flowchart for the dataset.

[0048] Figure 3 This is a diagram showing the overall shape and structure of the UnitreeG1.

[0049] Figure 4 This is a graph showing how the waist tilt angle changes with the intensity of emotions.

[0050] Figure 5 Emotion recognition accuracy chart.

[0051] Figure 6a A diagram illustrating a robot's fearful gait.

[0052] Figure 6b A diagram illustrating the shy gait of a robot.

[0053] Figure 6c This is a diagram illustrating the robot's open gait.

[0054] Figure 6d A diagram illustrating the robot's sad gait.

[0055] Figure 6e This is a schematic diagram of a robot's confident gait. Detailed Implementation

[0056] This invention provides a method for generating and controlling the gait of a humanoid robot based on emotion perception, comprising the following steps:

[0057] Step 1: Given human-computer interaction input, perform emotion perception by extracting discrete emotional states from semantic cues, and map the inferred emotional states to emotion intensity. Then, emotional intensity is introduced. With physical control parameters The prior mapping between them integrates domain knowledge and physical constraints into reasoning to obtain task-related physical control instructions;

[0058] Step 2: Based on the psychological emotion classification method, the emotions expressed by humans while walking are divided into three categories: neutral, positive, and negative. The positive includes happiness and confidence; the negative includes sadness, fear, and shyness.

[0059] Eight professional dancers then performed multi-directional, multi-step walking sequences in three categories: neutral, positive, and negative. The whole-body motion data was recorded using an optical motion capture system, and a structured periodic motion dataset with emotion annotations was constructed by combining self-supervised data processing.

[0060] Step 3: Construct a gait generator. Train the gait generator based on the structured periodic motion dataset with emotion annotations constructed in Step 2. Then, based on the emotion intensity and physical control commands in Step 1, the gait generator combines periodic signals to generate periodic kinematic trajectories with emotion features. The kinematic trajectories include robot joint positions, joint velocities, and root node poses.

[0061] Step 4: A unified reinforcement learning-based controller tracks the kinematic trajectory generated in step 3 to enable the robot to walk in multiple emotional gaits.

[0062] Step 1 includes:

[0063] In the first stage, self-designed prompt words are imported to guide the large language model to act as an emotion recognition agent, and to parse the dominant emotion semantic tags (such as happiness, sadness, fear, etc.) and their corresponding intensities (i.e., the structured and quantitative results of semantic information such as degree adverbs and tone in the language) from the user's natural language commands.

[0064] In the second stage, a priori mapping relationship between artificially defined emotional intensity and physical control parameters (such as "high intensity happiness → high cadence, high speed") is injected into the cue words, transforming the large language model into a gait parameter generator. To ensure the physical feasibility of the movement, robot physical constraints are introduced, including the maximum walking speed in the x-direction. Maximum walking speed in the y direction and maximum angular velocity Ensure that the generated parameters satisfy the dynamics and actuator constraints;

[0065] By guiding a large language model to generate control parameters containing semantic and emotional information, a complete parameter set compatible with the gait generator interface is constructed, enabling diversified gait generation and precise control driven by semantic and emotional perception.

[0066] Step 2 includes: 8 professional dancers performing multi-directional, multi-step walking sequences in three categories: neutral, positive, and negative, and recording full-body motion data using an optical motion capture system, which is uniformly represented in the standardized human body model SMPL (Skinned Multi-Person Linear Model) format.

[0067] Step 2 also includes: transferring human gait data to the humanoid robot through remapping, and introducing an adaptive height correction mechanism to alleviate the problem of foot suspension: identifying support phases by detecting frames where the foot horizontal velocity and yaw angular velocity are below a threshold, and dynamically compensating for the robot's root height. The threshold is generally set to 0.2.

[0068] Step 2 further includes: performing independent kinematic mapping on the upper body and lower body separately using a segmented remapping strategy, and then fusing the results to decouple the motion trajectories. The remapped trajectory is represented as a trajectory of length [missing information]. state sequence , where the t-th frame Includes the following: Root node location Root node pose Root node linear velocity Root node angular velocity Joint angle and joint angular velocity ;in This represents the position of the root node in the x-direction of the world coordinate system. This represents the position of the root node in the y-direction of the world coordinate system. This represents the position of the root node in the z-direction of the world coordinate system. This represents the rotation angle of the root node about the x-axis. This represents the rotation angle of the root node about the y-axis. Represents the rotation angle of the root node about the z-axis; This represents the root node's velocity in the x-direction in the world coordinate system. This represents the velocity of the root node in the y-direction in the world coordinate system. This represents the velocity of the root node in the z-direction in the world coordinate system. This represents the angular velocity of the root node about the x-axis. This represents the angular velocity of the root node about the y-axis. This represents the angular velocity of the root node about the z-axis.

[0069] Step 2 also includes: Since gait data has obvious periodicity, and also facilitates the learning of the generator network, the obtained trajectory needs to be further segmented into periods. Each segment contains a complete walking cycle. :

[0070] ,

[0071] in This represents a collection of gait data from two or more cycles. N is the total number of cycles. , The first The revelation index and termination index of each segment;

[0072] The specific extraction method includes: selecting local trajectory peaks from key reference joints (such as hip joint pitch) as candidate segmentation points, and then using dynamic programming for global optimization under physical constraints to extract the longest effective peak subsequence, thereby ensuring temporal continuity and physical consistency; however, abnormal gaits deviating from normal walking patterns still exist in the segmented dataset, which need to be further filtered out by calculating the average longitudinal velocity. lateral velocity and yaw rate To extract motion features, and at the same time based on the duration of the segment Get step frequency Where fps is the frame rate, and linear regression analysis is used to determine the linear velocity norm. With step frequency The correlation between the two factors is statistically consistent with biomechanical principles, and the correlation is determined by... Modeling is performed, where k and b are the slope and intercept of the linear model, respectively. Significant outliers and low-speed unstable segments are removed to construct a physically reasonable dataset.

[0073] Ultimately, the self-supervised input feature is defined as a combination of a three-dimensional average velocity vector and a frequency offset, where the frequency offset... Calculated from regression residuals:

[0074] .

[0075] In step 3, the gait generator is constructed using the following method: the gait generator is constructed based on a structured periodic motion dataset with emotion annotation. The gait generator is a lightweight motion generation network that generates whole-body periodic motion trajectories with emotion intensity and physical control commands as input.

[0076] The gait generator is a lightweight gait generation network based on Multilayer Perceptron (MLP). Used for frame-by-frame target pose regression: input features Incorporating sinusoidal time coding Emotional intensity and physical control commands Output the whole body reference state This includes root attitude, linear velocity and angular velocity, as well as joint position and velocity;

[0077] To address the issues of subjective bias and discrete noise in manual sentiment annotation, a Transformer-based classifier is used as a label smoother to reconstruct the latent sentiment distribution: for the Each motion segment will determine the robot's state. With sinusoidal time coding The input sequence is obtained by concatenation, where It represents the phase angle of the nth motion segment at time t, which is usually driven by the periodic rhythm of the motion (such as the gait cycle) and reflects the current position in the complete motion cycle; and a classification token (CLStoken) is added before the input sequence to aggregate the global sequence representation;

[0078] By capturing global spatiotemporal dependencies through a multi-layer self-attention mechanism and suppressing annotation noise and anomalies through information bottlenecks, the output latent state corresponding to CLStoken is extracted and mapped by a normalized exponential function to obtain a continuous probability distribution in the sentiment category space as the refined soft label sentiment intensity. That is, the original discrete hard labels (one-hot vectors, such as "happy" corresponding to [1,0,0,0,0]) are transformed into probability vectors reflecting the confidence of each sentiment category (such as [0.75,0.10,0.08,0.07,0.07]), thereby constructing a continuous probability manifold in the sentiment category space, effectively alleviating the subjectivity of manual annotation and providing a more robust sentiment intensity supervision signal for subsequent gait parameter generation.

[0079] Step 4 includes: randomly sampling the emotional intensity and physical control parameters based on the gait generator, generating diverse periodic gait sequences covering two or more emotional categories and movement styles online, and constructing a long-term training dataset through cyclic expansion and splicing;

[0080] The training employs a phased strategy: First, the robot is trained on flat terrain to master stable basic walking ability and tracking skills for different emotional gaits. Then, two or more complex terrains are introduced for training, enabling the robot to maintain gait stability and tracking accuracy under different terrains, thereby improving the environmental robustness of the strategy and reducing the simulation-to-real gap. Simultaneously, the BeyondMimic method is extended to include multiple emotions and actions. By introducing large-scale, diverse gait data containing different combinations of emotion intensities and control parameters, a general strategy capable of uniformly tracking two or more emotional gaits is trained. This gives the strategy flexible generalization capabilities within a continuous emotional space and physical constraints. Each gait data point is jointly determined by physical control parameters and emotional parameters. Physical control parameters include the robot's x-axis velocity, y-axis velocity, rotational angular velocity, and step frequency offset during movement, used to control the robot's direction, speed, and gait rhythm. Emotional parameters, composed of emotion labels and emotion intensity, are used to adjust the overall style and movement performance of the gait. For example, a high intensity of sadness can be paired with control parameters such as x-direction velocity 0.5, y-direction velocity 0.0, rotational angular velocity 0.0, and step frequency offset -0.1.

[0081] To address the issue of emotion expression in human-computer interaction and robot control, this invention provides a method for generating and controlling the gait of a humanoid robot based on emotion perception, such as... Figure 1 As shown. This method has been tested and proven effective on multiple robots. This example uses the Unitree G1 robot as an example, and the specific robot's external structure is shown below. Figure 3 As shown, the method specifically includes the following steps:

[0082] Step 1: Given the human-computer interaction input, the operator repeatedly inputs the following text multiple times through the natural language interface: "Work is too tiring" and "The weather is so nice today!"; Based on the input text, the large language model extracts the dominant sentiment semantic labels as sad and happy, and then determines the emotional intensity... With physical control parameters The prior mapping between them integrates domain knowledge and physical constraints into the reasoning, generating gait parameters as shown in Table 1 below (example).

[0083] Table 1

[0084]

[0085] Step 2: Add time codes to the parameters in Table 1. As a conditional input, it drives the gait generator to output periodic gait data. The data and processing flow required to construct the gait generator are as follows: Figure 2 As shown, it supports the output of gait motion in a single frame at 50Hz. The specific output content is shown in Table 2 below (taking N frames as an example).

[0086] Table 2

[0087]

[0088] Step 3: Using preset parameters, a large amount of rich gait data generated by the gait generator is pre-trained using the Proximal Policy Optimization (PPO) algorithm (neural network structure: 154-dimensional input layer, 29-dimensional output layer). After training for 3×10⁴ steps in the IsaacSim simulation environment with 4096 parallel environments, it is deployed to a real device. The parameters of the input layer are shown in Table 3 below.

[0089] Table 3

[0090]

[0091] Step 4: Use the reference motion posture, joint velocity, and joint position obtained in Step 2 as the unified control strategy obtained in Step 3. The system inputs various emotions and gaits to enable real-world walking. In subjective experiments, 83% of testers were able to correctly identify the emotion expressed by the robot's sad gait. Furthermore, in rating the matching degree between the robot's sad gait and the emotion, testers gave an average score of 9.13 out of 10. Specific accuracy rates for all emotion recognition are as follows: Figure 5 As shown, the matching scores (for 41 test participants) are shown in Table 4 below.

[0092] Table 4

[0093]

[0094] Specific examples of robot multi-emotional gait include... Figure 6a , Figure 6b , Figure 6c , Figure 6d , Figure 6e As shown.

[0095] In step 2, for the same emotion label, changes in emotion intensity (0~1) generate changes in joint position of the action, as shown below. Figure 4 As shown (taking the lumbar joint pitch angle as an example).

[0096] This invention provides a method for generating and controlling the gait of a humanoid robot based on emotion perception. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for gait generation and control of a humanoid robot based on affective perception, characterized by, Includes the following steps: Step 1, given human-machine interaction input, sentiment perception is performed by extracting discrete emotional states from semantic cues, and the inferred emotional states are mapped to emotional intensity Then the emotional intensity is introduced into the prior mapping between the physical control parameters , and the domain knowledge and physical constraints are integrated into the reasoning to obtain task-related physical control instructions; Step 2: Based on the psychological emotion classification method, the emotions expressed by humans while walking are divided into three categories: neutral, positive, and negative. The positive includes happiness and confidence; the negative includes sadness, fear, and shyness. The dancers then performed multi-directional, multi-step walking sequences in three categories: neutral, positive, and negative. The full-body motion data was recorded using an optical motion capture system, and a structured periodic motion dataset with emotion annotations was constructed by combining self-supervised data processing. Step 3: Construct a gait generator. Train the gait generator based on the structured periodic motion dataset with emotion annotations constructed in Step 2. Then, based on the emotion intensity and physical control commands in Step 1, the gait generator combines periodic signals to generate periodic kinematic trajectories with emotion features. The kinematic trajectories include robot joint positions, joint velocities, and root node poses. Step 4: A unified reinforcement learning-based controller tracks the kinematic trajectory generated in step 3 to enable the robot to walk in multiple emotional gaits.

2. The method of claim 1, wherein, Step 1 includes: In the first stage, prompt words are imported to guide the large language model to act as an emotion recognition agent, parsing the dominant emotional semantic tags and their corresponding intensities from the user's natural language commands. In the second stage, a predefined prior mapping relationship between emotional intensity and physical control parameters is injected into the cue words, transforming the large language model into a gait parameter generator and introducing robot physical constraints, including the maximum walking speed in the x-direction. Maximum walking speed in the y direction and maximum angular velocity ; By guiding a large language model to generate control parameters containing semantic and emotional information, a complete parameter set compatible with the gait generator interface is constructed.

3. The method of claim 2, wherein, Step 2 includes: dancers performing multi-directional, multi-step walking sequences in three categories: neutral, active, and passive, and recording full-body motion data using an optical motion capture system, which is uniformly represented in the standardized human model SMPL format.

4. The method of claim 3, wherein, Step 2 also includes: transferring human gait data to the humanoid robot through remapping, and introducing an adaptive height correction mechanism: identifying support phases by detecting frames where the foot horizontal velocity and yaw angular velocity are below the threshold, and dynamically compensating for the robot's root height.

5. The method of claim 4, wherein, Step 2 further includes: performing independent kinematic mapping on the upper body and lower body separately using a segmented remapping strategy, and then fusing the results to decouple the motion trajectories. The remapped trajectory is represented as a trajectory of length [missing information]. state sequence , where the t-th frame Includes the following: Root node location Root node pose Root node linear velocity Root node angular velocity Joint angle and joint angular velocity ;in This represents the position of the root node in the x-direction of the world coordinate system. This represents the position of the root node in the y-direction of the world coordinate system. This represents the position of the root node in the z-direction of the world coordinate system. This represents the rotation angle of the root node about the x-axis. This represents the rotation angle of the root node about the y-axis. Represents the rotation angle of the root node about the z-axis; This represents the root node's velocity in the x-direction in the world coordinate system. This represents the velocity of the root node in the y-direction in the world coordinate system. This represents the velocity of the root node in the z-direction in the world coordinate system. This represents the angular velocity of the root node about the x-axis. This represents the angular velocity of the root node about the y-axis. This represents the angular velocity of the root node about the z-axis.

6. The method of claim 5, wherein, Step 2 also includes: further segmenting the obtained trajectory into periods, with each segment containing a complete walking cycle. : ， in This represents a collection of gait data from two or more cycles. N is the total number of cycles. , The first The revelation index and termination index of each segment; The specific extraction method includes: selecting local trajectory peaks detected by key reference joints as candidate segmentation points, and then using dynamic programming for global optimization under physical constraints to extract the longest effective peak subsequence; and calculating the average longitudinal velocity. lateral velocity and yaw rate To extract motion features, and at the same time based on the duration of the segment Get step frequency Where fps is the frame rate, and linear regression analysis is used to determine the linear velocity norm. With step frequency The correlation, the correlation is determined by Modeling is performed, where k and b are the slope and intercept of the linear model, respectively. Significant outliers and low-speed unstable segments are removed to construct a physically reasonable dataset. Finally, the self-supervised input feature is defined as a combination of the three-dimensional mean velocity vector and a frequency offset calculated from the regression residual: 。 7. The method of claim 6, wherein, In step 3, the gait generator is constructed using the following method: the gait generator is constructed based on a structured periodic motion dataset with emotion annotation. The gait generator is a lightweight motion generation network that generates whole-body periodic motion trajectories with emotion intensity and physical control commands as input. The gait generator is a lightweight gait generation network based on a multilayer perceptron (MLP). Used for frame-by-frame target pose regression: input features Incorporating sinusoidal time coding Emotional intensity and physical control commands Output the whole body reference state This includes root attitude, linear velocity and angular velocity, as well as joint position and velocity; A Transformer-based classifier is used as a label smoother to reconstruct the latent sentiment distribution: for the Each motion segment will determine the robot's state. With sinusoidal time coding The input sequence is obtained by concatenation, where This represents the phase angle of the nth motion segment at time t; and a classification token CLStoken is added before the input sequence to aggregate the global sequence representation; By leveraging a multi-layer self-attention mechanism to capture global spatiotemporal dependencies, suppressing labeling noise and anomalies through information bottlenecks, extracting the output latent state corresponding to CLStoken and mapping it through a normalized exponential function, we obtain a continuous probability distribution in the sentiment category space as the refined soft label sentiment intensity. In other words, the original discrete hard labels are transformed into probability vectors reflecting the confidence of each sentiment category, thereby constructing a continuous probability manifold in the sentiment category space.

8. The method of claim 7, wherein, Step 4 includes: randomly sampling the emotional intensity and physical control parameters based on the gait generator, generating diverse periodic gait sequences covering two or more emotional categories and movement styles online, and constructing a long-term training dataset through cyclic expansion and splicing; The training adopts a phased strategy: first, the robot is trained on flat terrain to enable it to master stable basic walking ability and tracking skills for different emotional gaits; then, two or more complex terrains are introduced for training to enable the robot to maintain gait stability and tracking accuracy under different terrains; at the same time, the BeyondMimic method is extended to multiple emotions and multiple actions by introducing large-scale and diverse gait data containing different emotional intensities and control parameter combinations to train a general strategy that can uniformly track two or more emotional gaits.

9. An electronic device, comprising: It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

10. A storage medium, characterized by It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in any one of claims 1 to 8.

Images