Hierarchical intention-driven humanoid robot man-machine interaction system and method thereof

Abstract

The invention provides a hierarchical intention-driven humanoid robot man-machine interaction system and a method thereof, relates to the field of man-machine interaction, effectively solves the defects of a deep learning method in long-time task planning, overcomes the limitation of a large-scale pre-training model in the aspect of fine-grained control, and improves the working efficiency. It is ensured that the robot can freely adapt to and stably execute tasks in a dynamic and complex environment, and the intelligent level of man-machine interaction and the robustness of the system are improved. The system extracts key object point information in a scene through a scene understanding module; predicting a future short-term attitude sequence of the mechanical arm through an attitude prediction module; a robot behavior decision module realized through a multi-modal large model completes robot high-order behavior prediction according to key object point information and a future short-term attitude sequence; and a robot planning code is generated through a robot planning code generation module according to the key object point information, the future short-term attitude sequence and robot high-order behavior prediction. The method is suitable for the man-machine interaction application field.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and more specifically to a hierarchical intention-driven humanoid robot human-computer interaction method. Background Technology

[0002] Research on robot behavior prediction not only enhances the adaptability and collaborative capabilities of humanoid robots in human-robot interaction tasks, but more importantly, it enables robots to anticipate and understand human intentions, thus allowing for more appropriate responses. Compared to passively executing commands, robots with predictive capabilities can proactively adjust their behavior, improving the naturalness and intelligence of interactions. In humanoid robot interaction tasks, existing behavior prediction methods can be categorized into two types based on the granularity of prediction: 1) Pose prediction methods based on deep learning These methods primarily focus on accurate posture (action) prediction for robots over short timescales. Their core idea is to analyze historical posture sequences between humans and robots, using a data-driven approach to infer possible future posture changes. Typically, these methods rely on deep learning techniques such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), Graph Neural Networks (GNNs), and Transformers. These models can perform deep modeling of features such as historical trajectories and skeletal points, learning motion patterns and dynamic trends to achieve accurate action prediction. Some methods also incorporate vision-language pre-trained models to extract richer semantic information, improving the stability and reliability of predictions.

[0003] Such methods are widely used in the field of human-robot collaboration, especially in tasks requiring high-precision short-term posture prediction. However, although they can accurately characterize the short-term posture evolution of robots, they struggle to handle decision-making problems involving complex reasoning, environmental perception, and long-term planning, limiting their applicability in higher-level behavior prediction.

[0004] 2) High-order behavior prediction methods based on large-scale pre-trained models Compared to traditional deep learning methods that focus on detailed action modeling, this type of approach emphasizes higher-level behavioral reasoning and decision generation. Its core advantage lies in its powerful semantic understanding, environmental perception, and reasoning capabilities, enabling robots to directly predict future policy-level behaviors. For example, when a user indicates a plan to rest in the living room, the robot can not only understand this need but also intelligently infer its potential needs by combining the user's posture, facial expressions, and current environmental information. It can then take corresponding actions in advance, such as preparing tea, adjusting the lighting, or regulating the room temperature, thereby providing more proactive and forward-looking assistance and services.

[0005] This method integrates multiple technologies such as natural language processing and computer vision, enabling the extraction of deep semantic information from text commands, visual input, and even speech information, and generating reasonable behavioral decisions based on context. Some large-scale pre-trained models (such as GPT and PaLM) can even directly generate robot control code (such as Python or ROS control scripts), allowing robots to adapt and adjust in complex environments. For example, in the field of service robots, this method can help robots understand user intentions and generate complete path planning or interaction strategies based on environmental information, thereby achieving more intelligent responses. However, although this method shows great potential in high-level reasoning and complex task planning, its real-time performance and execution accuracy may still be inferior to methods based on refined motion modeling, especially in task scenarios requiring rapid decision-making or high-precision posture adjustment.

[0006] In summary, deep learning-based posture prediction methods can accurately predict robot posture changes over short periods, ensuring the stability and executability of actions by analyzing historical data. However, these methods primarily focus on short-term accuracy and have limited understanding of global task objectives, making them ill-suited for multi-stage, long-term task planning. In complex task scenarios, deep learning methods often fail to generate coherent long-term behavioral strategies, resulting in robots lacking reasonable global planning and adaptability in dynamic environments.

[0007] In contrast, high-order behavior prediction methods based on large-scale pre-trained models rely on powerful reasoning capabilities to handle complex long-term task planning, comprehensively considering factors such as environmental perception, task objectives, and user needs to generate more reasonable behavioral decisions. This gives robots greater adaptability and flexibility when performing long-term tasks, especially in multi-factor comprehensive judgment. However, although these methods perform well in global reasoning, their ability to predict fine-grained actions is relatively weak, which may lead to posture instability or action deviations when performing delicate tasks, thus affecting the accuracy and efficiency of the task. Therefore, relying solely on high-order behavior prediction methods cannot ensure the robot's accurate execution at every detailed step, thereby affecting the overall task performance. Summary of the Invention

[0008] The purpose of this invention is to propose a hierarchical, intent-driven humanoid robot human-robot interaction method. This method cleverly integrates short-term fine-grained posture prediction with long-term task planning to ensure that the robot accurately controls its own posture in a short period of time, avoiding operational deviations caused by unstable movements, while possessing the ability to rationally plan and dynamically adjust for long-term tasks. Through this method, the robot can achieve efficient coordination between global task optimization and local precise execution, ensuring the continuity, stability, and flexibility of task execution, thereby improving the robot's task execution efficiency in complex scenarios, enhancing the naturalness and intelligence of human-robot interaction, and enabling the robot to adapt to diverse interaction needs more smoothly and accurately.

[0009] To achieve the above objectives, the present invention provides the following technical solution: This invention proposes a hierarchical intent-driven humanoid robot human-computer interaction system, which includes a scene understanding module, a posture prediction module, a robot behavior decision-making module, and a robot planning code generation module. The scene understanding module is used to extract key object point information in the scene based on observation data; The attitude prediction module is used to predict the future short-term attitude sequence of the robotic arm based on the observation data. The robot behavior decision-making module is used to predict high-order robot behavior based on key object point information and future short-term posture sequences. The robot planning code generation module is used to generate robot planning code based on key object point information, future short-term posture sequences, and predictions of higher-order robot behaviors.

[0010] Furthermore, the aforementioned robot behavior decision-making module is implemented based on a multimodal large model.

[0011] This invention also proposes a hierarchical intent-driven humanoid robot human-computer interaction method. This method is implemented based on the aforementioned hierarchical intent-driven humanoid robot human-computer interaction system and includes the following steps: Step S1: Extract key points of objects in the scene; Step S2: Predict the future short-term posture sequence of the robotic arm; Step S3: Optimize the robot behavior decision-making module; Step S4: Use key points, the future short-term pose sequence of the robotic arm, scene information and human facial emotion data as multimodal information, perform semantic processing on the multimodal information, and use the optimized robot behavior decision module to process the semantically processed multimodal information to generate high-order robot behavior prediction. Step S5: Generate the specific code to be executed by the robot based on the object's key points, the robot arm's future short-term posture sequence, and higher-order behavior prediction.

[0012] Furthermore, the robot behavior decision-making module is optimized by pre-setting specific usage scenarios and based on the collected case data.

[0013] Furthermore, step S1 above specifically includes: Step S11: Extract local features of the scene image using the DINOv2 model. The feature map is obtained, and bilinear interpolation is used to resize the feature map back to the size of the original image. ; Step S12: Use the SAM model to segment the image and obtain multiple object mask sets. ; Step S13: Extract the corresponding features from the mask of each object. And for the feature information of each object, the K-means clustering algorithm is used to divide its features into... K There are 3 categories, and the center point of each cluster is considered the key point of the object; Step S14: Use an RGB-D camera to acquire the depth information of key points in the image. Combine the two-dimensional coordinates of each key point with its corresponding depth value to obtain the coordinates of the key point in three-dimensional space.

[0014] Furthermore, step S2 above specifically includes: Step S21: Extract the pose sequence features of the human and robot, and fuse and encode them; Step S22: Using the encoded last frame of human hand pose information as conditional input, the Transformer decoder is used to predict the robot's sequence features, and the predicted sequence features are converted into the robot's spatial pose position through IDCT. Step S23: Use SmoothNet to smooth and optimize the predicted sequence to obtain the future short-term posture sequence of the robotic arm.

[0015] Furthermore, step S21 above specifically includes: Step S211: Use DCT to transform the pose sequences of the human and the robot respectively; Step S212: Perform feature embedding on the transformed pose sequences of the human and robot respectively; Step S213: Use a local Transformer encoder to encode the local displacement of the robot using contextual features, and use a global Transformer encoder to encode the fused human and robot pose sequences.

[0016] Furthermore, step S5 above specifically includes: Step S51: Overlay the set of key points onto the original RGB image and label the position of each key point with numbers; Step S52: Combine the predicted pose sequence and higher-order semantic behavior with visual cues to query the required number of stages from GPT-4o, and generate sub-objective constraints and path constraints for each stage; Step S53: Generate the specific code to be executed by the robot based on the sub-goal constraints and path constraints.

[0017] The hierarchical intent-driven humanoid robot human-computer interaction method described in this invention can be entirely implemented using computer software. Therefore, correspondingly, this invention also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform any of the above-described hierarchical intent-driven humanoid robot human-computer interaction methods.

[0018] The present invention also provides a computer device, the device including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a hierarchical intention-driven humanoid robot human-computer interaction method as described in any of the above.

[0019] The technical solution proposed in this invention has produced significant beneficial effects, mainly reflected in the following aspects: 1. Achieves proactive adaptation and collaboration with user activities: By fusing high-order semantic prediction of behavior with fine-grained pose prediction, this method introduces global task semantic guidance into task planning, enabling the robot to proactively optimize its behavior. Simultaneously, the method incorporates fine-grained pose prediction into task planning, allowing the robot to avoid interfering with the user's key activity processes and provide timely and appropriate assistance. For example, in a cooking scenario, the robot can dynamically adjust its position and behavior to avoid interfering with the user's activities, thereby achieving more efficient collaboration.

[0020] 2. Enhanced system intelligence and environmental adaptability: This method, through deep fusion of multimodal information and collaborative optimization of high- and low-order behaviors, enables the robot to adapt more flexibly to complex and dynamic interactive environments. For example, when a user is cooking, the robot can not only avoid physical interference but also automatically prepare the necessary auxiliary tools or materials according to the user's needs, improving collaborative efficiency.

[0021] 3. Enhanced naturalness of human-computer interaction and user experience: This method significantly reduces the abruptness and intrusiveness of robot behavior by continuously sensing and learning user behavior and the environment. The robot can provide natural and accurate assistance services without interrupting user activities, thus optimizing the overall user experience.

[0022] This invention is applicable to the field of hierarchical intent-driven humanoid robot human-computer interaction technology. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is an overall architecture diagram of a hierarchical intent-driven humanoid robot human-computer interaction system proposed in this invention. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0028] Implementation Method 1, see Figure 1 This embodiment describes a hierarchical intent-driven humanoid robot human-computer interaction system, such as... Figure 1As shown, it specifically includes a scene understanding module, a posture prediction module, a robot behavior decision-making module, and a robot planning code generation module; The scene understanding module is used to extract key object point information in the scene based on observation data; The attitude prediction module is used to predict the future short-term attitude sequence of the robotic arm based on the observation data. The robot behavior decision-making module is used to predict high-order robot behavior based on key object point information and future short-term posture sequences. The robot planning code generation module is used to generate robot planning code based on key object point information, future short-term posture sequences, and predictions of higher-order robot behaviors.

[0029] Furthermore, the robot behavior decision-making module described above can be implemented based on a multimodal large model.

[0030] The hierarchical intent-driven humanoid robot human-computer interaction system proposed in this embodiment consists of four core parts: a scene understanding module, a posture prediction module, a robot behavior decision-making module based on a multimodal large model, and a robot planning code generation module. Its overall architecture is as follows: Figure 1 As shown in the diagram, the scene understanding module primarily aims to extract key point information of objects within the scene, providing a foundation for subsequent high-order robot behavior prediction and robot planning code generation. The posture prediction module aims to accurately predict the robot arm's future short-term posture sequence. The robot behavior decision-making module, based on a multimodal large model, aims to predict the robot's high-order behavior by fusing multimodal data (such as posture sequences, scene information, and human facial emotion data) with the reasoning capabilities of a large language model. The robot planning code generation module aims to generate specific code for robot execution based on the extracted key point information of objects, the predicted robot posture sequence, and the high-order behavior predictions.

[0031] Implementation Method Two: This implementation method proposes a hierarchical intention-driven humanoid robot human-computer interaction method. The interaction method is implemented based on the human-computer interaction system proposed in Implementation Method One above. The human-computer interaction method includes the following steps: Step S1: Extract key points of objects in the scene; Step S2: Predict the future short-term posture sequence of the robotic arm; Step S3: Optimize the robot behavior decision-making module; Step S4: Use key points, the future short-term pose sequence of the robotic arm, scene information and human facial emotion data as multimodal information, perform semantic processing on the multimodal information, and use the optimized robot behavior decision module to process the semantically processed multimodal information to generate high-order robot behavior prediction. Step S5: Generate the specific code to be executed by the robot based on the object's key points, the robot arm's future short-term posture sequence, and higher-order behavior prediction.

[0032] Implementation Method 3: This implementation method provides a detailed description of the hierarchical intention-driven humanoid robot human-computer interaction method proposed in Implementation Method 2 above. Step S1: Extract key points of objects in the scene to provide a foundation for subsequent high-order robot behavior prediction and robot planning code generation.

[0033] To achieve this goal, the specific steps are as follows: Step S11: Extract image features: Use the DINOv2 model to extract local features of the image. This implementation uses the self-supervised learning model DINOv2 to effectively extract features at different levels in an image, such as material, texture, and shape. This allows the model to understand the content of the image and capture more detailed visual information. However, because the feature maps generated by DINOv2 have relatively low resolution, they cannot fully preserve the image's detailed information. To ensure that each pixel has corresponding feature information, this implementation uses bilinear interpolation to resize the feature maps back to the original image size, thereby obtaining... This ensures that each pixel has corresponding feature information.

[0034] Step S12: Image segmentation using the SAM model: In this step, the SAM model is used to segment the image and obtain multiple object mask sets. This implementation uses the SAM model to efficiently identify different objects in an image and generate a precise mask for each object. For example, in a desktop image, the SAM model generates independent masks for objects such as a cup, a mobile phone, and a book. These masks identify the specific location and outline of each object in the image.

[0035] Step S13: Extract key points on each object: First, extract the mask for each object. (i.e., extract the corresponding features for each object) For each object's feature information, this implementation uses the K-means clustering algorithm to divide its features into K categories. The purpose of clustering is to classify different regions of an object. For example, for the object of a cup, there may be multiple different regions, such as the rim, body, and handle. To achieve clustering, cosine similarity is used as a metric to measure the similarity between different features. The center point of each cluster is considered as a candidate keypoint of the object, representing the object's typical location or feature.

[0036] Step S14: Mapping Key Points to 3D Space: This implementation uses an RGB-D camera to acquire depth information of key points in the image. An RGB-D camera can simultaneously capture the color (RGB) and depth (D) information of an image, thus providing a corresponding depth value for each pixel in the image. Using these depth values, key points in the 2D image can be mapped to a real-world coordinate system in 3D space. Specifically, the 2D coordinates (position in the image) of each key point are combined with its corresponding depth value to obtain the coordinates of that point in 3D space. This conversion process allows key points that originally only existed in the 2D image to obtain true 3D coordinate information.

[0037] Step S2: Predict the future short-term posture sequence of the robotic arm; Specifically, the steps include the following: Step S21: Extracting Human and Robot Pose Sequence Features: First, DCT is used to transform the pose sequences to accurately capture subtle pose changes. Then, feature embedding is performed on the pose sequences of both the robot and the human, converting high-dimensional pose information into low-dimensional feature vectors. To better capture the robot's local pose information, a local Transformer encoder is used to encode the robot's local displacements using contextual features, thereby learning its short-term motion patterns. Simultaneously, a global Transformer encoder is used to encode the fused human and robot pose sequences, extracting their macroscopic motion trends to comprehensively utilize both local and global feature information. In this process, the use of feature embedding, the local Transformer encoder, and the global Transformer encoder is consistent with the implementation method in InteRact.

[0038] Step S22: Predict the robot's future pose sequence: First, the human hand pose information from the last observed frame is used as input, and a Transformer decoder is employed to predict the robot's sequence features. For example, when the human is about to complete a placement action, the model predicts the target position and pose the robot should achieve in the next step based on the pose information of the human hand's end-effector, thus enabling coordination between the human and the robot. Subsequently, the predicted sequence features are converted into the robot's spatial pose position using IDCT.

[0039] Step S23: Smooth the predicted sequence using SmoothNet: First, feature mapping is performed on the input pose sequence using a linear encoder. Then, the feature representation is further optimized using a residual module to eliminate jitter and discontinuous changes in the sequence. For example, in an unsmoothed predicted sequence, the robot may exhibit slight jitter or positional jumps. With SmoothNet optimization, these abnormal fluctuations will be effectively suppressed, resulting in a smoother, more continuous robot pose sequence.

[0040] Step S3: Optimize the robot behavior decision-making module; Specifically, to achieve personalized adaptation of the intelligent agent, this implementation method adopts a contextualized case data collection method. By pre-setting specific usage scenarios, it systematically collects personalized preference data of target users. Based on the collected case data, the intelligent agent is fine-tuned to train a home service robot system that best adapts to user behavior habits and personalized needs.

[0041] Step S4: Use key points, the future short-term pose sequence of the robotic arm, scene information and human facial emotion data as multimodal information, perform semantic processing on the multimodal information, and use the optimized robot behavior decision module to process the semantically processed multimodal information to generate high-order robot behavior prediction. Specifically: Step S41: Semantic Processing: First, key points, the robotic arm's future short-term pose sequence, scene information, and human facial emotion data are used as multimodal information. Then, semantic processing is performed on the pose, facial expression, and speech modal data. Specifically, the original pose image and pose key points are used as input, and a semantically valuable pose description is generated through a cross-modal cueing mechanism. Facial expressions are processed using the same method, while speech information is converted into text using ASR technology.

[0042] Step S42: Behavior Decision and Feedback Generation: Based on the multimodal semantic information obtained in Step S41, emotion recognition and intent prediction are performed through large model prompt design to generate structured decision results, including human emotional state, human behavioral intent, robot language feedback, robot action execution, and smart home control commands, realizing high-order robot behavior prediction. The above information is stored in structured JSON format, with the standard format: {"human_emotion": "[emotion recognition result]", "human_intention": "[intent prediction result]", "robot_word": "[robot language feedback]", "robot_action": "[robot action execution]", "robot_control": "[smart home control commands]"}.

[0043] Step S5: Generate the specific code to be executed by the robot based on the object's key points, the robot arm's future short-term posture sequence, and higher-order behavior prediction.

[0044] Specifically: This implementation uses the ReKep method. First, based on the keypoint candidate set obtained in the previous steps, it is overlaid on the original RGB image, and the position of each keypoint is labeled with a number. For example, in a desktop image, assuming a cup is detected and three keypoints are extracted: the rim, the body, and the handle, numbers are labeled on these positions to visually show the positional relationships between the keypoints. Combining the predicted pose sequence and higher-order semantic behavior, visual prompting is used to query the GPT-4o for the required number of stages, and sub-objective constraints and path constraints corresponding to each stage are generated. It is worth noting that the robot execution function finally generated in this implementation does not directly manipulate the numerical values ​​of the keypoint positions. Instead, it utilizes the capabilities of the Visual Language Model (VLM) to specify spatial relationships through arithmetic operations, which are only instantiated at the time of invocation. For example, in the task of grasping a cup, L2 distance can be used to calculate the relative position between the rim and the handle to determine the optimal grasping point.

[0045] This implementation proposes an innovative hierarchical intent-driven humanoid robot human-robot interaction method. This method combines the advantages of fine-grained pose prediction based on deep learning and high-order behavior prediction based on large-scale pre-trained models. Through semantic guidance of the global task, combined with precise pose control in the short term and reasonable planning for the long term, this method fully leverages the high accuracy of low-order prediction and the global adaptability of high-order inference, thereby achieving efficient execution and precise control of the robot in complex tasks. Compared with existing technologies, this implementation effectively addresses the shortcomings of deep learning methods in long-term task planning, while overcoming the limitations of large-scale pre-trained models in fine-grained control. This ensures that the robot can adapt freely and stably execute tasks in dynamic and complex environments, improving the intelligence level of human-robot interaction and the robustness of the system. This innovative solution not only optimizes the accuracy and efficiency of robot task execution but also provides theoretical and technical support for the deployment of intelligent robots in more practical application scenarios.

[0046] Furthermore, this method achieves proactive adaptation and collaboration with user activities. It predicts the robot arm's future short-term posture sequence as fine-grained posture prediction, predicts the robot's higher-order behaviors through a robot behavior decision module, and then integrates higher-order semantic prediction with fine-grained posture prediction. This introduces global task semantic guidance into task planning, enabling the robot to proactively optimize its behavior. Simultaneously, the method incorporates fine-grained posture prediction into task planning, allowing the robot to avoid interfering with the user's key activities and provide timely and appropriate assistance. For example, in a cooking scenario, the robot can dynamically adjust its position and behavior to avoid interfering with the user's activities, thereby achieving more efficient collaboration.

[0047] Furthermore, the system's intelligence and environmental adaptability are enhanced: this method, through deep fusion of multimodal information and collaborative optimization of high- and low-order behaviors, enables the robot to adapt more flexibly to complex and dynamic interactive environments. For example, when a user is cooking, the robot can not only avoid physical interference but also automatically prepare the necessary auxiliary tools or materials according to the user's needs, improving collaborative efficiency.

[0048] Furthermore, this method enhances the naturalness of human-computer interaction and user experience: by continuously sensing and learning user behavior and the environment, it significantly reduces the abruptness and intrusiveness of robot behavior. The robot can provide natural and precise assistance without interrupting user activities, thus optimizing the overall user experience.

[0049] In summary, the hierarchical intent-driven humanoid robot human-robot interaction method proposed in this embodiment endows the robot with more intelligent task planning and interaction capabilities through deep collaboration of high and low order behaviors and semantic guidance. It demonstrates good adaptability and user-friendliness in diverse application scenarios, and significantly improves the efficiency and quality of robot-assisted tasks.

[0050] Implementation Method 4: The hierarchical intent-driven humanoid robot human-computer interaction method proposed in the above implementation methods can be entirely implemented using computer software. Therefore, this implementation method proposes a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes any one of the hierarchical intent-driven humanoid robot human-computer interaction methods described in the above implementation methods.

[0051] Implementation Method 5: This implementation method proposes a computer device, which includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a hierarchical intent-driven humanoid robot human-computer interaction method as described in any of the above implementation methods.

[0052] This embodiment provides a computer device. The hardware device in this part is a general model and is not shown in the figure. The system includes a processor and a memory. The processor and the memory can be connected by a bus or other means. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, as well as corresponding program instructions / modules. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions and modules stored in the memory, so as to realize the hierarchical intent-driven humanoid robot human-computer interaction method and steps in the above method embodiment.

[0053] In the above descriptions, PaLM stands for Pathways Language Model. GPT stands for Generative Pre-trained Transformer. ReKep stands for Relational Keypoint Constraints. DINOv2 stands for Dual-Stage Implicit Object-Oriented Network version 2. SAM stands for Segment Anything Model. DCT stands for Discrete Cosine Transform. IDCT stands for Inverse Discrete Cosine Transform. SmoothNet stands for Smooth Neural Network. InteRact stands for Intent Prediction via Robot Action-ConditionedTransformer. ASR stands for Automatic Speech Recognition.

[0054] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A hierarchical intent-driven humanoid robot human-computer interaction system, characterized in that, It includes a scene understanding module, a posture prediction module, a robot behavior decision-making module, and a robot planning code generation module; The scene understanding module is used to extract key object point information in the scene based on observation data; The attitude prediction module is used to predict the future short-term attitude sequence of the robotic arm based on observation data; The robot behavior decision-making module is used to predict high-order robot behavior based on key object point information and future short-term posture sequences. The robot planning code generation module is used to generate robot planning code based on key object point information, future short-term posture sequences, and predictions of higher-order robot behaviors.

2. The hierarchical intent-driven humanoid robot human-computer interaction system according to claim 1, characterized in that, The robot behavior decision-making module is implemented based on a multimodal large model.

3. The human-computer interaction method for a hierarchical intent-driven humanoid robot human-computer interaction system as described in claim 1 or 2, characterized in that, The method is as follows: S1: Extract key points of objects in the scene; S2: Predicts the future short-term posture sequence of the robotic arm; S3: Optimize the robot behavior decision-making module; S4: Key points, the future short-term pose sequence of the robotic arm, scene information, and human facial emotion data are used as multimodal information. The multimodal information is semantically processed, and the optimized robot behavior decision module is used to process the semantically processed multimodal information to generate high-order robot behavior predictions. S5: Generates specific code for robot execution based on object key points, the robot arm's future short-term posture sequence, and higher-order behavior predictions.

4. The hierarchical intention-driven humanoid robot human-computer interaction method according to claim 3, characterized in that, The robot's behavior decision-making module is optimized by pre-setting specific usage scenarios and based on collected case data.

5. A hierarchical intention-driven humanoid robot human-computer interaction method according to claim 3, characterized in that, S1 specifically refers to: S11: Extracting local features of scene images using the DINOv2 model. The feature map is obtained, and bilinear interpolation is used to resize the feature map back to the size of the original image. ; S12: Use the SAM model to segment the image and obtain multiple object mask sets. ; S13: Extract the corresponding features from the mask of each object. And for the feature information of each object, the K-means clustering algorithm is used to divide its features into... K There are 3 categories, and the center point of each cluster is considered the key point of the object; S14: Use an RGB-D camera to acquire depth information of key points in the image. Combine the two-dimensional coordinates of each key point with its corresponding depth value to obtain the coordinates of the key point in three-dimensional space.

6. The hierarchical intent-driven humanoid robot human-computer interaction method according to claim 3, characterized in that, S2 specifically refers to: S21: Extract human and robot posture sequence features, and fuse and encode them; S22: The last frame of human hand pose information after encoding is used as a conditional input. The Transformer decoder is used to predict the robot's sequence features, and the predicted sequence features are converted into the robot's spatial pose position through IDCT. S23: SmoothNet is used to smooth and optimize the predicted sequence to obtain the future short-term posture sequence of the robotic arm.

7. A hierarchical intention-driven humanoid robot human-computer interaction method according to claim 5, characterized in that, S21 specifically refers to: S211: DCT is used to transform the posture sequences of humans and robots respectively; S212: Perform feature embedding on the transformed human and robot pose sequences respectively; S213: A local Transformer encoder is used to encode the local displacement of the robot using contextual features, while a global Transformer encoder is used to encode the fused human and robot posture sequences.

8. A hierarchical intent-driven humanoid robot human-computer interaction method according to claim 3, characterized in that, S5 specifically refers to: S51: Overlay the set of key points onto the original RGB image and label the position of each key point with a number; S52: Combining the predicted posture sequence and higher-order semantic behavior, use visual cues to query the GPT-4o for the required number of stages, and generate sub-objective constraints and path constraints for each stage; S53: Generate the specific code for robot execution based on sub-goal constraints and path constraints.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the hierarchical intent-driven humanoid robot human-computer interaction method according to any one of claims 3-8.

10. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a hierarchical intent-driven humanoid robot human-computer interaction method as described in any one of claims 3-8.

Images