Humanoid robot multi-mode instruction analysis system

Abstract

The invention discloses a multi-mode instruction analysis system for a humanoid robot. Comprising a voice input module, a visual input module, a voiceprint feature extraction module, an object recognition and pose estimation module, a multi-modal alignment network based on a space-time attention mechanism, a scene semantic tree construction module, an instruction node mapping module, a confidence evaluation module and a decision module. According to the system, accurate alignment of voice and visual information is realized through a space-time attention mechanism, environment information is represented in combination with a scene semantic tree structure, and the instruction analysis accuracy is improved. And dynamically evaluating the confidence coefficient by adopting a fuzzy instruction backtracking algorithm, and if the confidence coefficient is lower than a threshold value, starting multi-round dialogue clarification to reduce misoperation. According to the method, multi-modal data are fused, the historical interaction learning ability is optimized, the understanding efficiency and interaction robustness of complex instructions are remarkably improved, the method is suitable for scenes such as family service and logistics storage, and the intelligent level of man-machine cooperation is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of robot intelligence technology, and in particular relates to a multimodal instruction parsing system for a humanoid robot. Background Art

[0002] With the advancement of artificial intelligence and robotics, humanoid robots are increasingly being used in fields such as home services, healthcare, and industrial production. Traditional human-machine interaction methods rely primarily on a single modality (such as voice or vision), making it difficult to accurately understand user commands in complex environments. These include the limitations of voice commands, the ambiguity of visual information, the challenges of multimodal fusion, and the inability to handle ambiguous commands. Therefore, a multimodal command parsing system for humanoid robots is urgently needed. Summary of the Invention

[0003] The purpose of the present invention is to provide a multimodal instruction parsing system for a humanoid robot to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned objectives, the present invention provides the following technical solutions: a multimodal instruction parsing system for a humanoid robot, comprising: a voice input module for receiving a user's voice instructions; a visual input module for acquiring environmental visual information; a voiceprint feature extraction module for extracting user voiceprint features from voice instructions; an object recognition and posture estimation module for identifying objects from visual information and estimating their posture; a multimodal alignment network based on a spatiotemporal attention mechanism for performing spatiotemporal alignment and feature fusion of voice instructions and visual information; a scene semantic tree construction module for constructing a semantic representation of a scene based on the multimodal alignment results; an instruction node mapping module for mapping voice instructions to corresponding nodes of a scene semantic tree; a confidence evaluation module for evaluating the confidence of instruction parsing using a fuzzy instruction backtracking algorithm; and a decision module for generating an action sequence when the confidence is greater than a preset threshold, and initiating a multi-round dialogue clarification mechanism when the confidence is less than the preset threshold.

[0005] Preferably, the multimodal alignment network based on the spatiotemporal attention mechanism includes: a temporal attention subnetwork for aligning the temporal dimension features of speech and visual information; a spatial attention subnetwork for focusing on key spatial areas in visual information; and a feature fusion subnetwork for deeply fusing the aligned multimodal features.

[0006] Preferably, the temporal attention subnetwork adopts a bidirectional LSTM structure to achieve temporal alignment of speech and visual information.

[0007] Preferably, the scene semantic tree construction module constructs a semantic tree in the following manner: taking the robot's environment as the root node; taking the identified objects and their spatial relationships as child nodes; and assigning semantic attributes and spatial posture information to each node.

[0008] Preferably, the scene semantic tree also includes functional attributes and operation constraints of objects.

[0009] Preferably, the fuzzy instruction backtracking algorithm includes: an instruction keyword extraction unit, used to extract key action words and target object words from voice instructions; a semantic similarity calculation unit, used to calculate the similarity between instruction keywords and scene semantic tree nodes; a backtracking verification unit, used to verify the rationality of the current instruction based on historical interaction records; and a confidence calculation unit, which calculates the final confidence by integrating the semantic similarity and backtracking verification results.

[0010] Preferably, the input parameters of the confidence calculation unit also include user identity information and historical interaction success rate.

[0011] Preferably, the multi-round dialogue clarification mechanism includes: an ambiguity detection unit for identifying ambiguous points in instructions; a question generation unit for generating clarification questions based on ambiguous points; and a dialogue management unit for managing the multi-round dialogue status and integrating user feedback.

[0012] Preferably, the preset threshold is 0.8.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] The present invention achieves precise alignment of speech and visual information through a spatiotemporal attention mechanism, thereby improving the recognition accuracy of target objects and actions. The scene semantic tree is used to structure the representation of environmental information, enabling the robot to more efficiently associate user commands with the physical world. The fuzzy command backtracking algorithm is used to dynamically evaluate the parsing confidence, and when the confidence is insufficient, multiple rounds of dialogue clarification are automatically triggered to reduce misoperations. The user's historical interaction data is combined to optimize command understanding and improve personalized interaction capabilities. Repeated inquiries or incorrect executions caused by ambiguous commands are reduced, and task completion efficiency is improved. The deep integration of natural language and visual environment is supported, enabling the robot to understand complex commands more naturally. Through multimodal fusion, dynamic confidence evaluation and intelligent dialogue management, the command understanding ability and interaction robustness of the humanoid robot are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic flow diagram of the present invention.

[0016] Figure 2 It is a flow chart of the multimodal alignment network of the present invention.

[0017] Figure 3 It is a flow chart of fuzzy instruction backtracing of the present invention.

[0018] Figure 4 It is a flowchart of the multi-round dialogue clarification process of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] Example 1:

[0021] like Figure 1-Figure 4A multimodal command parsing system for a humanoid robot is shown, comprising: a voice input module for receiving user voice commands; a visual input module for acquiring visual information of the environment; a voiceprint feature extraction module for extracting user voiceprint features from voice commands; an object recognition and pose estimation module for identifying objects from visual information and estimating their pose; a multimodal alignment network based on a spatiotemporal attention mechanism for performing spatiotemporal alignment and feature fusion between voice commands and visual information; a scene semantic tree construction module for constructing a semantic representation of the scene based on the multimodal alignment results; a command node mapping module for mapping voice commands to corresponding nodes in the scene semantic tree; a confidence assessment module for evaluating the confidence of command parsing using a fuzzy command backtracking algorithm; and a decision module for generating an action sequence when the confidence is greater than a preset threshold and initiating a multi-round dialogue clarification mechanism when the confidence is less than the preset threshold. The multimodal alignment network based on the spatiotemporal attention mechanism comprises: a temporal attention subnetwork for aligning the temporal dimension features of voice and visual information; a spatial attention subnetwork for focusing on key spatial regions in the visual information; and a feature fusion subnetwork for deeply fusing the aligned multimodal features. The temporal attention subnetwork uses a bidirectional LSTM structure to achieve temporal alignment of speech and visual information. The scene semantic tree construction module constructs a semantic tree using the robot's environment as the root node, identified objects and their spatial relationships as child nodes, and assigning semantic attributes and spatial pose information to each node. The scene semantic tree also contains the functional attributes and operational constraints of objects. The fuzzy instruction backtracking algorithm includes: an instruction keyword extraction unit for extracting key action words and target object words from voice instructions; a semantic similarity calculation unit for calculating the similarity between instruction keywords and scene semantic tree nodes; a backtracking verification unit for verifying the rationality of the current instruction based on historical interaction records; and a confidence calculation unit for calculating the final confidence score by combining semantic similarity and backtracking verification results. Input parameters for the confidence calculation unit also include user identity information and historical interaction success rates. The multi-turn dialogue clarification mechanism includes: an ambiguity detection unit for identifying ambiguous points in instructions; a question generation unit for generating clarification questions based on ambiguous points; and a dialogue management unit for managing multi-turn dialogue states and integrating user feedback. The preset threshold is 0.8.

[0022] Through the above technical solution, the present invention realizes the precise alignment of speech and visual information through the spatiotemporal attention mechanism, thereby improving the recognition accuracy of target objects and actions. The scene semantic tree is used to structure the representation of environmental information, so that the robot can more efficiently associate user instructions with the physical world. The fuzzy instruction backtracking algorithm is used to dynamically evaluate the parsing confidence, and when the confidence is insufficient, multiple rounds of dialogue clarification are automatically triggered to reduce misoperation. The user's historical interaction data is combined to optimize instruction understanding and improve personalized interaction capabilities. Repeated inquiries or incorrect executions caused by ambiguous instructions are reduced, and the efficiency of task completion is improved. The deep integration of natural language and visual environment is supported, so that the robot can understand complex instructions more naturally. Through multimodal fusion, dynamic confidence evaluation and intelligent dialogue management, the humanoid robot's instruction understanding ability and interaction robustness are significantly improved.

[0023] Example 2:

[0024] like Figure 1-Figure 4 As shown, the operating process of this embodiment is as follows: When a user issues a voice command, the voice input module collects audio signals through a microphone array, while the visual input module acquires three-dimensional visual information of the environment through a stereo camera. The voiceprint feature extraction module uses a deep neural network to extract the speaker's voiceprint features from the voice signal for subsequent personalized interaction. The object recognition and pose estimation module uses a deep learning model to identify object categories in the visual scene and calculate the object's precise pose in the robot coordinate system.

[0025] The multimodal alignment network, based on a spatiotemporal attention mechanism, first performs semantic parsing on the voice command, extracting key action words and target object descriptors. Simultaneously, the network analyzes the spatiotemporal features of the visual information to establish a spatiotemporal correlation between the voice command and the visual scene. The attention mechanism calculates the similarity between voice and visual features, achieving precise alignment between the voice command and the visual scene. For example, when a user says, "Give me the cup on the left," the network accurately correlates the word "left" in the voice with the spatial position of the cup in the visual scene.

[0026] The scene semantic tree construction module transforms the multimodal alignment results into a structured semantic representation. Using objects in the scene as nodes, this module constructs a tree structure that includes object attributes, spatial relationships, and functional semantics. The command node mapping module maps parsed voice commands to corresponding nodes in the semantic tree, forming an executable task representation. The confidence assessment module uses a fuzzy command backtracking algorithm to calculate the confidence score of the current command parsing by analyzing factors such as the degree of match between the command and the scene and historical interaction data.

[0027] When the confidence score exceeds a preset threshold, the decision module generates a specific action sequence, such as grabbing and moving instructions. If the confidence score is insufficient, the system automatically initiates a multi-round dialogue clarification mechanism, asking targeted questions to obtain additional information. For example, if a user says "take that tool" and there are multiple tools in the scene, the system will ask, "Do you mean the wrench on the left or the screwdriver on the right?" Throughout this process, the system continuously updates the scene semantic tree and user preference model, gradually improving the accuracy of command understanding. Through this multimodal fusion and dynamic evaluation mechanism, the system can effectively handle ambiguous instructions in complex environments and achieve natural and smooth human-computer interaction.

[0028] Example 3:

[0029] like Figure 1-Figure 4 As shown, this embodiment achieves precise alignment of speech and visual information through a spatiotemporal attention mechanism. The temporal attention subnetwork first frames the speech signal and extracts the acoustic features of each frame. At the same time, the visual input module collects the environmental video stream at a fixed frame rate. The temporal attention subnetwork calculates the temporal correlation between speech frames and video frames and establishes a cross-modal temporal alignment relationship. For each time segment in the voice command, the subnetwork automatically learns the most relevant visual frame sequence to eliminate the temporal misalignment problem caused by the different acquisition frequencies of speech and vision. For example, when the user says "pick up the moving cup", the temporal attention subnetwork can accurately associate the temporal dynamic description of "moving" with the motion trajectory of the cup in the visual sense.

[0030] The spatial attention subnetwork focuses on key spatial regions within the visual scene. This subnetwork receives temporally aligned visual features and uses a self-attention mechanism to calculate the importance weights of different image regions. For complex scenes containing multiple objects, the spatial attention subnetwork automatically focuses on relevant regions based on the object descriptions in voice commands. For example, when a user says, "Please pass me the red box on the left," the subnetwork suppresses feature responses for the box on the right and other irrelevant objects, while strengthening the feature representation of the region containing the red box on the left. The spatial attention weights are cross-validated with the output of the object recognition module to ensure that the focused region aligns with the object attributes described in the voice command.

[0031] The feature fusion subnetwork uses a hierarchical cross-attention mechanism to achieve deep fusion of multimodal features. This subnetwork first establishes a cross-attention mapping between speech and visual features to explore the semantic associations between the two modalities. For each semantic unit in the speech, such as an action verb or object noun, the feature fusion subnetwork searches for the most matching response pattern in the visual feature space. A feature gating mechanism then dynamically adjusts the contribution weights of features from different modalities, enhancing speech features when the speech command is clear and the visual information is ambiguous, and vice versa. The fused multimodal features retain the uniqueness of the original modalities while incorporating a joint representation across modalities, providing richer input for subsequent semantic parsing.

[0032] The scene semantic tree construction module converts the fused multimodal features into a structured scene representation. This module uses object recognition results as tree nodes and spatial relationships and action associations as edges to construct a hierarchical semantic graph.

[0033] Example 4:

[0034] like Figure 1-Figure 4 As shown, the temporal attention sub-network of this embodiment uses a bidirectional LSTM structure to achieve temporal alignment of speech and visual information. Its working principle is as follows:

[0035] When a voice command is input, the speech feature extraction module first converts the speech signal into a temporal feature sequence. This sequence contains the speech's spectral characteristics, intonation changes, and the temporal distribution of semantic key points. Simultaneously, the visual input module continuously collects the ambient video stream and, through the object recognition and pose estimation module, extracts visual temporal features, including object motion trajectories, posture changes, and scene dynamics. Because the time scales of speech and visual information may differ—for example, users may be delayed or advanced in describing an object—a temporal alignment mechanism is required to ensure the synchronization of multimodal information.

[0036] The bidirectional LSTM structure simultaneously captures contextual dependencies between speech and visual features through forward and backward temporal modeling. The forward LSTM layer processes input features in chronological order, learning the positive temporal association between speech commands and visual information. For example, when a user says "pick up the cup," the appearance of the cup in the visual information should align with the key words in the speech command. The backward LSTM layer processes the temporal data in reverse order to capture possible delays or advances. For example, when a user describes an action, they may first point to the target object and then issue a speech command. Through the joint optimization of the bidirectional LSTM, the temporal attention subnetwork dynamically adjusts the temporal offset of speech and visual features, ensuring that key actions and semantics are precisely matched in the temporal dimension.

[0037] The temporal attention subnetwork further incorporates an attention mechanism to calculate the association weights between speech and visual features at each time step. For each time point in the speech feature sequence, the attention mechanism automatically assigns attention to the corresponding time point in the visual feature sequence, thereby filtering out the visual information most relevant to the current speech command. For example, when the user says, "Push the red box to the left," the temporal attention mechanism emphasizes the time period in the visual sequence where the red box appears and suppresses interference from irrelevant background information. This dynamic weight allocation effectively reduces the impact of noise and improves the robustness of multimodal fusion.

[0038] During the feature fusion phase, the time-aligned speech and visual features are fed into the spatial attention subnetwork to further optimize information correlation in the spatial dimension. The output of the temporal attention subnetwork not only contains the aligned temporal features but also carries information about cross-modal attention distribution, providing a precise temporal alignment basis for the subsequent construction of the scene semantic tree. By synergizing the bidirectional LSTM with the attention mechanism, the system can adapt to different user interaction habits, such as speech rate or gesture delay, thereby improving the humanoid robot's command parsing ability in complex environments.

[0039] Embodiment 5:

[0040] like Figure 1-Figure 4 As shown, the scene semantic tree construction module of this embodiment constructs the semantic tree in the following manner. First, the system uses the robot's current physical environment as the root node of the semantic tree. This root node contains basic semantic attributes of the environment, such as global information such as room type and spatial dimensions. When the robot enters a home living room, the root node is assigned the semantic label "living room" and the three-dimensional spatial extent of the living room is recorded. This environment-based root node design allows subsequent object recognition and command parsing to be performed within a clear spatial context.

[0041] After establishing the root node, the system detects each object in the environment using the object recognition and pose estimation modules. Each recognized object is attached as a separate child node to the root node, forming a hierarchical tree structure. For example, if the sofa, coffee table, and TV in the living room are recognized, these objects become direct children of the root node. The system not only records the object's category information but also uses visual sensors to obtain the precise spatial position of each object, including its 3D coordinates and orientation in the environment. This storage of spatial pose information enables the robot to accurately understand the relative positions of objects.

[0042] To describe the scene in more detail, the system further analyzes the spatial relationships between objects and establishes a corresponding sub-node hierarchy. When it detects that the coffee table is in front of the sofa and the TV is facing the sofa, these spatial relationships are converted into special association nodes and inserted into the semantic tree. For example, the "in front" relationship node connects the sofa and coffee table object nodes, while the "facing" relationship node connects the sofa and TV nodes. Each relationship node has clear spatial semantic attributes, such as distance, azimuth, and other parameters. These attributes provide the basis for spatial reasoning for subsequent command parsing.

[0043] Each node in the semantic tree is endowed with rich semantic attributes. For object nodes, the system records semantic information such as functional attributes and usage status. For example, a TV node might have operational attributes such as "turn on / off" and "adjustable volume." Relational nodes store relational semantics such as spatial orientation and contact status. These semantic attributes are mapped to the action predicates in the user's voice commands. When a user says "turn on the TV," the system can quickly locate the TV node and match its "turn on / off" attribute.

[0044] In terms of dynamic environment updates, when the robot moves or objects in the environment change, the system updates the semantic tree structure in real time. New objects are dynamically added as child nodes, moving objects have their position information updated, and missing objects are marked as invalid nodes.

[0045] Example 6:

[0046] like Figure 1-Figure 4 As shown in the figure, in a home service scenario, when a humanoid robot executes a user's command "Put the water cup on the coffee table in the kitchen" through the multimodal command parsing system, the system first receives the voice command through the voice input module, while the visual input module obtains real-time visual information of the living room environment. The voiceprint feature extraction module identifies the specific user's voiceprint from the speech, and the object recognition and pose estimation module detects objects such as the coffee table and water cup and their spatial positions.

[0047] A multimodal alignment network based on a spatiotemporal attention mechanism aligns keywords such as "tea table," "water cup," and "kitchen" in voice commands with visually detected objects in spatiotemporal order, establishing semantic associations. The scene semantic tree construction module uses the robot's living room environment as the root node and creates a semantic tree structure encompassing objects such as the tea table and water cup. The water cup node not only contains spatial pose information but also specifically annotates its container properties (capable of holding liquids) and operational constraints (fragile items must be handled with care).

[0048] The command node mapping module maps the action in the voice command to the operational constraints of the water cup node in the scene semantic tree, ensuring that the robot adheres to the gentle handling constraint when executing the action. The system also identifies the kitchen as the target location and establishes a spatial association in the semantic tree. The confidence assessment module, using a fuzzy command backtracking algorithm, analyzes the command and finds that there are multiple water cups on the coffee table in the current environment, causing the command to be ambiguous and resulting in a confidence score below a preset threshold.

[0049] At this point, the decision module initiates a multi-round dialogue clarification mechanism, with the robot proactively querying the user. Upon confirmation, the system updates the confidence level of the target cup node in the scene semantic tree and reassesses whether the execution threshold has been met. Finally, the decision module generates a complete sequence of actions, including grasping, moving, and placing, allowing the robot to accurately execute the task of moving the designated cup from the coffee table to the kitchen. Throughout this process, the system dynamically maintains the functional attributes and operational constraints of objects in the scene semantic tree to ensure accurate and secure task execution.

[0050] Embodiment seven:

[0051] like Figure 1-Figure 4 As shown, this embodiment achieves accurate parsing and verification of ambiguous instructions through a multi-stage processing flow. In the instruction keyword extraction stage, the system uses a speech feature extraction model based on the attention mechanism to separate key action words and target object words from the user's voice instructions.

[0052] The semantic similarity calculation unit uses a deep matching model based on word vectors to calculate the similarity between extracted command keywords and nodes in the scene semantic tree. The scene semantic tree, constructed by the object recognition and pose estimation modules, contains information about the categories, attributes, and spatial relationships of all objects in the environment. During the calculation process, the system not only considers similarity at the lexical level but also conducts a comprehensive assessment based on the visual features and spatial position of the objects.

[0053] The retrospective verification unit verifies the legitimacy of current commands by analyzing historical user interactions. The system maintains a time-series-based interaction knowledge base, recording users' past command preferences and operating habits. When encountering ambiguous commands, the system searches for user preferences in similar historical scenarios. For example, if a user has repeatedly selected a specific red object in similar scenarios in the past, the system prioritizes that object as a possible match. The verification process utilizes a graph neural network-based reasoning model that captures deep-level correlations within user behavior patterns.

[0054] The confidence calculation unit employs a multi-factor fusion strategy, combining a weighted semantic similarity score and backtesting results. The similarity score reflects the immediate match between the command and the scenario, while the backtesting results reflect the user's historical behavior patterns. The system uses an adaptive weight adjustment mechanism to dynamically adjust the contribution of these two factors based on the complexity of the interaction scenario and the reliability of historical data. The resulting confidence score is used by the decision-making module. When the score falls below a threshold, a multi-round dialogue clarification mechanism is triggered. The entire algorithm is implemented using an end-to-end deep learning framework, with online optimization using continuous human-computer interaction data to continuously improve its ability to handle ambiguous commands.

[0055] Embodiment 8:

[0056] like Figure 1-Figure 4 As shown, the confidence calculation unit of this embodiment further optimizes the accuracy of command parsing by incorporating user identity information and historical interaction success rates. When a user issues a voice command, the system first identifies the user through the voiceprint feature extraction module and correlates the user's historical interaction records. For example, if user A has previously issued the command "bring the cup over here" in similar scenarios and the system has a high probability of successful execution, the confidence of the current command will be increased accordingly. Conversely, if user B has a low historical success rate for similar commands, the system will lower the initial confidence and be more inclined to initiate multiple rounds of dialogue verification.

[0057] The historical interaction success rate is calculated by analyzing the execution results of a user's past commands, including successful execution, partial execution failures, and complete misunderstandings. The system maintains a separate interaction record database for each user, recording the command type, scenario characteristics, and execution results. When evaluating the current command, the confidence calculation unit prioritizes matching the success rate data for similar historical scenarios. For example, if a user frequently successfully issues the "open refrigerator" command in a kitchen environment, the confidence level of subsequent similar commands will be automatically increased based on this historical data, reducing unnecessary confirmation dialogues.

[0058] For new users or scenarios lacking historical data, the system uses a default confidence threshold and gradually builds a personalized evaluation model as the number of interactions increases. The confidence calculation unit weights and fuses user identity information with semantic similarity and backtesting results, using the historical interaction success rate as a dynamic adjustment factor. For example, when the semantic similarity calculation result reaches a critical value, if the user's historical success rate is high, the final confidence level can exceed the threshold, triggering direct execution; otherwise, the clarification process begins. This mechanism effectively balances the system's exploration of new instructions with its utilization of verified instructions.

[0059] During multiple rounds of dialogue clarification, the system adjusts its inquiry strategy based on the user's identity. For users with a high historical success rate, the system tends to use brief, confirmatory questions; for users with a lower success rate or new users, more detailed, guiding questions may be used. Furthermore, after each successful interaction, the system updates the user's history in real time, forming a closed-loop learning mechanism for continuous optimization. This dynamic adjustment strategy significantly improves the fluidity and personalized experience of human-computer interaction.

[0060] Embodiment 9:

[0061] like Figure 1-Figure 4 As shown, the multi-round dialogue clarification mechanism of this embodiment resolves command ambiguity through dynamic interaction. When a user issues an ambiguous command, the ambiguity detection unit first analyzes the command for potential ambiguities. This unit combines the scene semantic tree with the command node mapping results to identify objects, locations, or actions that are not explicitly specified in the command.

[0062] The question generation unit automatically constructs natural language questions based on detected ambiguous points. This unit uses a semantic template-based generation method to convert ambiguous points into complete interrogative sentences. The question generation process references the outputs of the object recognition and pose estimation modules to ensure that questions contain distinguishable object features or spatial location information. Furthermore, this unit avoids generating open-ended questions and instead provides limited options to reduce the complexity of the user's responses.

[0063] The dialogue management unit is responsible for maintaining the context of multiple rounds of dialogue. It records user feedback for each clarification question and dynamically updates the node attributes of the scene semantic tree.

[0064] The dialogue management unit uses a finite state machine model to determine subsequent actions based on the current stage of the conversation: if all ambiguous points have been resolved, the command parsing process is retriggered; if any clarifications remain, the next round of questions is generated. This unit also integrates data from the voiceprint feature extraction module to isolate conversation threads in multi-user scenarios, ensuring that different users' clarification processes do not interfere with each other.

[0065] During multiple rounds of dialogue, the system continuously calls the confidence assessment module to verify the clarification results. After each user feedback, the system recalculates the confidence level of the instruction parsing. If the confidence level reaches a preset threshold, the clarification process is terminated and the decision module generates the action sequence; otherwise, the system continues to identify the remaining ambiguous points. This iterative clarification mechanism significantly reduces the risk of misoperation caused by ambiguous instructions.

[0066] Embodiment 10:

[0067] like Figure 1-Figure 4As shown, in this embodiment, when the preset threshold of the decision module is 0.8, the system achieves reliable command parsing through a precise confidence assessment mechanism. When a user issues a voice command such as "Pass me the cup on the table," the system first collects the audio signal through the voice input module, while the visual input module captures an image of the environment containing the table and cup. The voiceprint feature extraction module extracts the speaker's personalized feature parameters from the speech waveform. These features are compared with the registered voiceprints in the user database to confirm the user's identity and load the corresponding personalized interaction parameters.

[0068] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0069] The above description is only used to illustrate the technical solution of the present invention and is not intended to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal instruction parsing system for a humanoid robot, characterized in that: include: A voice input module, used to receive user voice commands; Visual input module, used to obtain environmental visual information; Voiceprint feature extraction module, used to extract user voiceprint features from voice commands; Object recognition and pose estimation module, used to identify objects and estimate their pose from visual information; A multimodal alignment network based on the spatiotemporal attention mechanism is used to align speech commands with visual information in spatiotemporal order and fuse their features; The scene semantic tree construction module is used to construct the semantic representation of the scene based on the multimodal alignment results; A command node mapping module, used to map voice commands to corresponding nodes in the scene semantic tree; The confidence evaluation module uses a fuzzy instruction backtracking algorithm to evaluate the confidence of instruction parsing; The decision module generates an action sequence when the confidence level is greater than a preset threshold, and initiates a multi-round dialogue clarification mechanism when the confidence level is less than a preset threshold.

2. The humanoid robot multimodal instruction parsing system according to claim 1, characterized in that: The multimodal alignment network based on the spatiotemporal attention mechanism includes: Temporal attention sub-network, used to align the temporal dimension features of speech and visual information; The spatial attention sub-network is used to focus on key spatial regions in visual information; The feature fusion subnetwork is used to deeply fuse the aligned multimodal features.

3. The humanoid robot multimodal instruction parsing system according to claim 2, characterized in that: The temporal attention subnetwork adopts a bidirectional LSTM structure to achieve temporal alignment of speech and visual information.

4. The humanoid robot multimodal instruction parsing system according to claim 1, characterized in that: The scene semantic tree construction module constructs the semantic tree in the following way: The robot's environment is the root node; The identified objects and their spatial relationships are used as child nodes; Assign semantic attributes and spatial pose information to each node.

5. The humanoid robot multimodal instruction parsing system according to claim 4, characterized in that: The scene semantic tree also includes functional attributes and operation constraints of objects.

6. The humanoid robot multimodal instruction parsing system according to claim 1, characterized in that: The fuzzy instruction backtracking algorithm includes: An instruction keyword extraction unit, used to extract key action words and target object words from voice instructions; A semantic similarity calculation unit, used to calculate the similarity between the instruction keyword and the scene semantic tree node; A backtracking verification unit is used to verify the rationality of the current instruction based on historical interaction records; The confidence calculation unit calculates the final confidence by integrating the semantic similarity and backtracking verification results.

7. The humanoid robot multimodal instruction parsing system according to claim 6, characterized in that: The input parameters of the confidence calculation unit also include user identity information and historical interaction success rate.

8. The humanoid robot multimodal instruction parsing system according to claim 1, characterized in that: The multi-round dialogue clarification mechanism includes: An ambiguity detection unit, used to identify ambiguous points in instructions; A question generation unit generates clarification questions based on ambiguous points; The dialogue management unit is used to manage the multi-round dialogue status and integrate user feedback.

9. The humanoid robot multimodal instruction parsing system according to claim 1, characterized in that: The preset threshold is 0.8.

Images