A multi-modal human-robot interaction method and system for a humanoid robot

By recognizing multiple interactive objects and establishing independent information processing channels for humanoid robots, analyzing multimodal intents, and determining priority levels, the problem of lag and confusion in response of humanoid robots in multi-user interaction is solved, achieving more natural interaction fluency and accuracy.

CN122633035APending Publication Date: 2026-08-25SHANGHAI YUNFAN INTELLIGENT CONTROL ROBOT TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610782154.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-06-02
Publication Date
2026-08-25

AI Technical Summary

Technical Problem

Humanoid robots struggle to distinguish between primary and secondary interactions in multi-user interaction scenarios, leading to delayed, biased, and chaotic interaction responses.

Method used

By identifying multiple interactive objects, establishing independent information processing channels, performing intent parsing on multimodal information, determining the processing priority of interactive requests, and controlling the robot's output behavior based on the priority and preset strategies.

Benefits of technology

It achieves reasonable and orderly responses in multi-user interaction scenarios, avoiding delayed interaction responses and chaotic behavior, and improving the smoothness and accuracy of interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122633035A_ABST
    Figure CN122633035A_ABST
Patent Text Reader

Abstract

The application provides a multi-modal human-computer interaction method and system for a humanoid robot, and relates to the technical field of human-computer interaction. The technical solution points are: identifying a plurality of interaction objects in a current interaction scene, and establishing an independent information processing channel for each interaction object; performing intention analysis on multi-modal information of each information processing channel to obtain an intention confidence and an intention intensity corresponding to each interaction object; determining a processing priority level of an interaction request for each interaction object according to the current interaction scene and the intention confidence and the intention intensity of each interaction object; and controlling the output behavior of the humanoid robot according to the processing priority level and a preset interaction strategy. The application aims to improve the ability of the humanoid robot to interact naturally and smoothly with multiple users in a complex public environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and more specifically, to a multimodal human-computer interaction method and system for humanoid robots. Background Technology

[0002] In public interaction scenarios involving humanoid robots, when multiple users interact with the robot simultaneously with conflicting intentions, existing humanoid robot interaction solutions struggle to achieve smooth interaction. For example, when a family of three—parents and child—approaches the robot, the father inquires about technical parameters, the mother focuses on product design, and the child pulls on the robot's arm and points to a flashing light. The system prioritizes the father's verbal request, but the child's sudden physical contact and high-decibel shouting trigger a strong conflict signal. The robot cannot determine whether to continue responding to the father or the child, ultimately resulting in disjointed and chaotic behavior. In other words, existing humanoid robots lack the ability to understand complex scenarios, cannot distinguish primary and secondary interactions or shifts in attention within a group, and can only perform mechanical combinations of signals rather than truly understanding dynamic group interaction situations. Summary of the Invention

[0003] The purpose of this application is to provide a multimodal human-computer interaction method and system for humanoid robots, aiming to improve the ability of humanoid robots to interact naturally and smoothly with multiple users in complex public environments.

[0004] Firstly, this application provides a multimodal human-computer interaction method for humanoid robots, including: Identify multiple interactive objects in the current interactive scenario and establish an independent information processing channel for each of the interactive objects; Intent parsing is performed on the multimodal information of each of the information processing channels to obtain the intent confidence and intent strength of each corresponding interaction object; Based on the current interaction scenario and the intent confidence and intent strength of each interaction object, determine the processing priority level of the interaction request for each interaction object; The output behavior of the humanoid robot is controlled according to the processing priority level and the preset interaction strategy.

[0005] Furthermore, determining the processing priority of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object includes: Based on the current interaction scenario and the intent confidence and intent strength of each interaction object, the processing priority level of the interaction request of each interaction object is determined, and the first interaction object and the second interaction object among the plurality of interaction objects are determined. The first interaction object represents the main interaction object currently interacting with the humanoid robot to control the output behavior of the humanoid robot based on the main interaction object, and the second interaction object represents the user object with potential interaction intent.

[0006] Furthermore, after determining the first and second interaction objects among the plurality of interaction objects, the multimodal human-computer interaction method for humanoid robots further includes: The computing resources allocated to the first interaction object and the second interaction object for intent monitoring are determined, wherein the computing resources allocated to the first interaction object are greater than the computing resources allocated to the second interaction object.

[0007] Furthermore, the interaction strategy includes a first interaction strategy and a second interaction strategy. The first interaction strategy is used to control the interaction between the humanoid robot and the first interaction object, and the second interaction strategy is used to control the interaction between the humanoid robot and the first interaction object and the second interaction object. The step of controlling the output behavior of the humanoid robot based on the processing priority level and the preset interaction strategy includes: Adjust the processing priority level of each interaction object according to the intent confidence and intent strength of each interaction object; When the processing priority of the second interactive object is lower than that of the first interactive object, the output behavior of the humanoid robot is controlled based on the first interaction strategy. When the processing priority of the second interaction object is higher than that of the first interaction object, the output behavior of the humanoid robot is controlled based on the second interaction strategy.

[0008] Furthermore, the step of identifying multiple interactive objects in the current interactive scenario and establishing independent information processing channels for each of the interactive objects includes: Identify multiple interactive objects in the current interactive scene and determine the interactive role of each interactive object; Based on the interaction roles of each interaction object, an independent information processing channel is established for each interaction object, so as to process multimodal information through the analysis method corresponding to the interaction roles.

[0009] Furthermore, determining the processing priority of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object includes: According to the preset priority evaluation rules, the current interaction scenario, the confidence of the intent of each interaction object, the intensity of the intent of each interaction object, the interaction role of each interaction object, and the degree of matching between the multimodal information of each interaction object and the corresponding interaction role behavior pattern are converted into priority scores. Based on the priority score, the processing priority level of the interaction request for each of the interaction objects is determined.

[0010] Furthermore, the interaction strategy includes: When interacting with the first interactive object, if the highest priority interaction request does not correspond to the first interactive object, and the priority score of the highest priority interaction request is lower than a preset score threshold, the humanoid robot's output is controlled through a preset hierarchical response process to respond to the highest priority interaction request, and the interaction with the first interactive object continues after the hierarchical response process is completed.

[0011] Furthermore, the identification of multiple interactive objects in the current interactive scenario includes: The physical sensory information obtained by the physical sensors of the humanoid robot is used to determine multiple physical objects in the current interactive scene and to determine the feature points of multiple physical objects. The physical sensory information includes auditory sensory information and tactile sensory information. Multiple visual objects in the current interactive scene are determined by visual sensory information obtained by the visual sensors of the humanoid robot. Based on the feature points of the multiple physical objects and the multiple visual objects, multiple interactive objects in the current interactive scene are determined.

[0012] Further, determining multiple interactive objects in the current interactive scene based on the multiple physical object feature points and multiple visual objects includes: The position and spatial depth of each visual object are determined. A first condition is obtained by matching the position of each visual object with the feature points of multiple physical objects. A second condition is obtained by matching the spatial depth of each visual object with the preset human body spatial depth. A third condition is obtained by using the physical sensory information of the position of each visual object. Based on the first condition, the second condition, and the third condition, multiple interactive objects among the multiple visual objects are determined.

[0013] Secondly, this application provides a multimodal human-computer interaction system for humanoid robots, including: The object recognition module is used to identify multiple interactive objects in the current interactive scenario and to establish an independent information processing channel for each interactive object. The intent parsing module is used to perform intent parsing on the multimodal information of each of the information processing channels to obtain the intent confidence and intent strength of each of the interaction objects. The priority module is used to determine the processing priority level of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object. The control module is used to control the output behavior of the humanoid robot according to the processing priority level and the preset interaction strategy.

[0014] As can be seen from the above, the multimodal human-computer interaction method for humanoid robots disclosed in this application achieves parallel processing of information from different interaction objects by identifying multiple interaction objects in the current interaction scenario and establishing independent information processing channels for them. Based on this, intent parsing is performed on the multimodal information of each information processing channel to obtain the intent confidence and intent strength of each interaction object, thereby quantifying the interaction intent of each interaction object. According to the current interaction scenario and the intent confidence and intent strength of each interaction object, the processing priority level of the interaction requests of each interaction object is determined, enabling dynamic evaluation and sorting of interaction requests from different users. Finally, based on the processing priority level and the preset interaction strategy, the output behavior of the humanoid robot is controlled, ensuring that the robot can make reasonable, orderly, and socially compliant responses when facing complex interaction scenarios with multiple users and multiple intents. This effectively solves the problems of delayed, biased, and chaotic interaction responses caused by the lack of collaborative parsing capabilities of multimodal information when humanoid robots face multi-user interactions. Attached Figure Description

[0015] Figure 1 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in one embodiment of this application; Figure 2 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application; Figure 3 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application; Figure 4 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application; Figure 5 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application; Figure 6 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application; Figure 7This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application; Figure 8 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Detailed Implementation

[0016] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to characterize selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0017] like Figure 1 As shown, Figure 1 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in one embodiment of this application. The method may include, but is not limited to, steps S110 to S140.

[0018] Step S110: Identify multiple interactive objects in the current interactive scene and establish an independent information processing channel for each interactive object; Step S120: Perform intent parsing on the multimodal information of each information processing channel to obtain the intent confidence and intent strength of each corresponding interactive object; Step S130: Determine the processing priority level of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object. Step S140: Control the output behavior of the humanoid robot according to the processing priority level and the preset interaction strategy.

[0019] For example, a humanoid robot can continuously scan its surroundings using sensors such as visual and auditory sensors. When it detects multiple interactive entities in the environment—for instance, by recognizing human silhouettes or detecting human voices—it identifies these entities as interactive objects. Subsequently, each identified interactive object is assigned an independent information processing channel. This includes assigning a visual information processing channel to each detected human silhouette for tracking posture and facial expressions; and simultaneously assigning an auditory information processing channel to each detected sound source for analyzing speech content and intonation. The information processing channels for each interactive object are logically independent, ensuring that information from different interactive objects is not confused.

[0020] For each independent information processing channel, multimodal information of the corresponding interactive object can be continuously received and analyzed. For example, for visual information, facial expressions, eye direction, and body movements can be analyzed; for auditory information, speech content, speech rate, volume, and tone can be analyzed. A pre-trained intent recognition model can be used to input multimodal information into the model for processing, thereby resolving the potential intent of the interactive object. Simultaneously, the model also outputs an intent confidence score, representing the reliability of the intent judgment, and assessing the intent strength; for example, a louder voice volume and faster speech rate indicate a stronger intent.

[0021] For example, the current interaction scenario can include a guided tour scenario, a question-and-answer scenario, or an entertainment scenario. Based on the current interaction scenario and the confidence and intensity of intent of each interactive object, the processing priority of the interaction request of each interactive object is determined. For example, in a guided tour scenario, if an interactive object issues a clear, high-confidence instruction to explain the exhibits and has a high intent intensity, its interaction request will be set to a high processing priority; conversely, if another interactive object just randomly utters some low-confidence self-talk, its processing priority will be low.

[0022] After determining the processing priority of each interaction request, the humanoid robot can decide how to respond based on the priority level and preset interaction strategies. For example, if the highest priority request is to answer a question, the robot will activate the speech synthesis module to output the corresponding answer; if the highest priority request is to perform physical interaction, the robot will control the actuators to perform the corresponding action. Interaction strategies can include various rules, such as how to queue or switch between multiple high-priority requests when they exist simultaneously, and how to interrupt the current interaction and prioritize urgent requests in case of an emergency.

[0023] It is understood that the multimodal human-computer interaction method for humanoid robots proposed in this application effectively isolates and manages information flow from the source by identifying multiple interaction objects in the current interaction scenario and establishing independent information processing channels for each interaction object. Even if multiple users interact with the robot at the same time, each user's information is captured and processed independently, avoiding information confusion. On this basis, the information of each independent channel is analyzed in depth. The robot can not only identify the specific intention of each user, but also assess the reliability and intensity of the intention. Subsequently, based on the current interaction scenario and the confidence and intensity of the intention of each interaction object, the processing priority of the interaction request of each interaction object is determined. That is to say, the humanoid robot no longer simply determines the response order according to the order of arrival or the intensity of a single modality, but comprehensively considers the current specific scenario, the clarity of each user's intention, and the urgency of the expression, etc., so that the robot can judge which user's request is more important in the current situation and needs to be responded to first, thereby avoiding response errors caused by the inability to distinguish between primary and secondary requests. Finally, after the priorities are determined, the robot will generate voice, facial expressions, actions and other outputs according to the preset interaction strategies for different scenarios and priority combinations. Here, the interaction strategies may include how to politely interrupt the current interaction to respond to higher priority requests, how to take into account the needs of multiple users at the same time, and how to ignore low priority requests in specific situations.

[0024] like Figure 2 As shown, Figure 2 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Regarding step S130 above, the method may include, but is not limited to, step S210.

[0025] Step S210: Based on the current interaction scenario and the intent confidence and intent strength of each interaction object, determine the processing priority level of the interaction request of each interaction object, and determine the first interaction object and the second interaction object among multiple interaction objects. The first interaction object represents the main interaction object currently interacting with the humanoid robot to control the output behavior of the humanoid robot based on the main interaction object, and the second interaction object represents the user object with potential interaction intent.

[0026] It is understandable that, in some embodiments, although the processing priority of interaction requests can be determined based on the interaction scenario, intent confidence, and intent strength, in actual multi-user interaction scenarios, if the primary interaction object and potential interaction objects are not clearly distinguished, the humanoid robot may be unable to effectively focus on the core interaction or respond to potential interaction intents in a timely or accurate manner when handling multiple interaction requests, thereby affecting interaction efficiency and user experience. Therefore, the processing priority of interaction requests for each interaction object can be determined based on the current interaction scenario and the intent confidence and intent strength of each interaction object, and a first interaction object and a second interaction object can be identified among multiple interaction objects. The first interaction object represents the primary interaction object currently interacting with the humanoid robot to control its output behavior, and the second interaction object represents a user object with potential interaction intent.

[0027] Specifically, the first interaction object can be understood as the object that the humanoid robot is currently actively interacting with or primarily responding to, possessing a high degree of intent confidence and intensity, and whose interaction behavior may directly target the humanoid robot. The second interaction object can be understood as a user object that is not currently actively interacting with the humanoid robot, but whose multimodal information indicates that it has a potential interaction intent. For example, when a user observes the interaction between the humanoid robot and the first interaction object, or exhibits a posture indicating that they are about to participate in the interaction, that user can be identified as the second interaction object. The humanoid robot can continuously monitor the second interaction object, but will not immediately elevate the corresponding interaction request to the same processing priority level as the first interaction object.

[0028] In addition, when determining the first and second interaction objects, multiple factors can be considered. The interaction object with the highest intent confidence is given priority as the first interaction object. If there are multiple objects with similar intent confidence, the intent strength, interaction scenario, and historical interaction records can be combined for a comprehensive judgment. Other objects with lower intent confidence or intent strength but still showing a certain interaction tendency are normally identified as the second interaction object.

[0029] For example, suppose a humanoid robot is playing a game with a child in a home environment. The child is identified as the primary interaction target, and the robot prioritizes responding to the child's voice commands, gestures, or facial expressions, providing corresponding actions and voice feedback according to the game rules. Meanwhile, the child's parent observes nearby, occasionally offering comments or glancing at the robot; this parent is identified as the secondary interaction target. The robot continuously monitors the parent's voice and gaze but does not immediately interrupt its interaction with the child to respond to the parent. For instance, if the parent only makes a soft comment, the robot continues interacting with the child; however, if the parent suddenly issues an urgent command or shows a clear intention to intervene, the robot can adjust the parent's processing priority based on the confidence and strength of that intention, or elevate the parent to the new primary interaction target, thus achieving a smooth switching and response between interactions.

[0030] like Figure 3 As shown, Figure 3 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Regarding the above method, the method may also include, but is not limited to, step S310.

[0031] Step S310: Determine the computing resources allocated to the first interaction object and the second interaction object for intent monitoring, wherein the computing resources allocated to the first interaction object are more than the computing resources allocated to the second interaction object.

[0032] In some embodiments, based on the first and second interaction objects, the processing priority of interaction requests for each interaction object can be determined according to the current interaction scenario and the intent confidence and intent strength of each interaction object, and the main interaction object currently interacting and the user object with potential interaction intent can be further identified. However, continuous multimodal information processing and intent monitoring of multiple interaction objects consume a large amount of computing resources. If resources are not allocated reasonably, the system may experience slow response when processing interaction requests from the main interaction object, or may be unable to effectively monitor changes in the intent of potential interaction objects, thereby affecting the overall smoothness and efficiency of the interaction.

[0033] Based on this, after identifying the first and second interaction objects among multiple interaction objects, the computing resources allocated to the first and second interaction objects for intent monitoring can be determined. Here, the computing resources for intent monitoring refer to the processor time, memory, storage space, and access permissions for specific algorithm modules required for real-time processing, analysis, and intent parsing of the multimodal information of the interaction objects. Computing resources are allocated to the first and second interaction objects to support the continuous tracking and evaluation of their respective intents. The first interaction object is identified as the primary interaction object currently interacting, and its corresponding interaction requests require more timely and accurate responses. The second interaction object represents a user object with potential interaction intents, and its priority is usually lower than that of the primary interaction object. Therefore, more computing resources are allocated to the first interaction object than to the second interaction object.

[0034] In this way, differentiated resource allocation ensures sufficient computational support for the interpretation and response of the primary interaction object, while also effectively monitoring the intentions of potential interaction objects, avoiding resource waste and effectively solving the problem of computational resource constraints in multi-interaction object scenarios. Furthermore, by allocating more computational resources to the primary interaction object, the interpretation of multimodal information receives more comprehensive computational support, ensuring rapid and accurate identification and response to the primary interaction object's intentions. Simultaneously, although relatively fewer computational resources are allocated to the secondary interaction object, it is still sufficient to continuously monitor its potential interaction intentions, avoiding intent omissions due to insufficient resources. This allows the humanoid robot to effectively address other potential interaction objects while maintaining high-quality interaction with the primary interaction object, improving overall interaction efficiency and user experience.

[0035] For example, suppose a humanoid robot is conversing with a user, identified as the first interaction object. Simultaneously, another user is observing in the room, identified as the second interaction object. To ensure smooth and timely responses in the conversation with the first interaction object, more computing resources, including CPU cores, GPU computing power, and memory bandwidth, can be allocated to processing the multimodal information flow of the first interaction object. This allows the neural network model used to interpret the first interaction object's intent to run at a higher frequency or use a more complex model for analysis. Meanwhile, relatively fewer computing resources are allocated to the second interaction object, which can be used only for periodically monitoring for significant voice input, large body movements, or specific facial expression changes to determine if a potential interaction intent has been generated.

[0036] In another embodiment of this application, a multimodal human-computer interaction method for humanoid robots is provided, wherein the interaction strategy includes a first interaction strategy and a second interaction strategy. The first interaction strategy is used to control the interaction between the humanoid robot and the first interaction object, and the second interaction strategy is used to control the interaction between the humanoid robot and the first interaction object and the second interaction object. like Figure 4 As shown, Figure 4 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Regarding step S140 above, the method may include, but is not limited to, steps S410 to S430.

[0037] Step S410: Adjust the processing priority level of each interaction object according to the intent confidence and intent strength of each interaction object; Step S420: When the processing priority of the second interactive object is lower than that of the first interactive object, control the output behavior of the humanoid robot based on the first interaction strategy. Step S430: When the processing priority of the second interactive object is higher than that of the first interactive object, control the output behavior of the humanoid robot based on the second interaction strategy.

[0038] For example, the first interaction strategy can be understood as an interaction mode focused on the primary interaction object. In this mode, the humanoid robot mainly concentrates its resources and attention on interacting with the primary interaction object, such as maintaining eye contact, responding to its voice commands, and executing its requests, to ensure the smooth progress of the main interaction process and guarantee the quality and consistency of the core interaction. The second interaction strategy is a more complex interaction mode that can simultaneously consider both the primary and secondary interaction objects. In this mode, the humanoid robot not only maintains interaction with the primary interaction object but also allocates some resources and attention to monitor and appropriately respond to the potential or explicit intentions of the secondary interaction object. For example, when the secondary interaction object shows an urgent or high-intensity intention, the humanoid robot may briefly shift some attention to the secondary interaction object for a brief confirmation or response before returning to interaction with the primary interaction object, or, if necessary, completely switch the interaction focus to improve the humanoid robot's adaptability to multi-user environments and avoid ignoring other important interaction requests.

[0039] For example, adjusting the processing priority of each interaction object based on its intent confidence and intensity means continuously monitoring and evaluating the intent confidence and intensity of each interaction object, and dynamically updating the processing priority of each interaction object in real time. When the processing priority of the second interaction object is lower than that of the first interaction object, the current main interaction should still be focused on the first interaction object. In this case, the humanoid robot controls its output behavior based on the first interaction strategy to maintain stable interaction with the first interaction object. Conversely, when the processing priority of the second interaction object is higher than that of the first interaction object, it indicates that the second interaction object may have a more urgent or important interaction need. In this case, the humanoid robot will control its output behavior based on the second interaction strategy, adjust the interaction focus, and may temporarily interrupt or partially transfer the interaction with the first interaction object to prioritize the processing of the second interaction object's request, thereby ensuring timely response to urgent or important situations.

[0040] Based on this, humanoid robots can flexibly adjust the processing priority of different interactive objects and select appropriate interaction strategies according to the dynamic changes in the confidence and intensity of the intentions of each interactive object in the interactive scenario. This allows humanoid robots to no longer be limited to a single preset interaction mode, but to intelligently switch between focusing on the main interactive object and taking into account other potential interactive objects.

[0041] For example, suppose in a family setting, a humanoid robot is telling a story to the child (the primary interaction partner). Suddenly, the child's mother (the secondary interaction partner) issues an urgent voice command, such as requesting medication. Because the humanoid robot continuously monitors the confidence and intensity of the child's and mother's intentions, these values ​​rapidly increase when the mother issues the urgent command, leading to a higher processing priority. If the priority score of the mother's interaction request is determined to be higher than the priority score of the child's current storytelling, meaning the secondary interaction partner has a higher processing priority than the primary one, the humanoid robot will switch from its previous control mode focused on storytelling based on the primary interaction strategy to a control mode based on the secondary interaction strategy. For example, the humanoid robot might immediately pause storytelling, turn to the mother and respond verbally, then perform the task of fetching the medication. After fulfilling the mother's request, the humanoid robot can choose, based on a preset strategy, whether to continue storytelling or wait for a new interaction command.

[0042] like Figure 5 As shown, Figure 5 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Regarding step S110 above, the method may include, but is not limited to, steps S510 to S520.

[0043] Step S510: Identify multiple interactive objects in the current interactive scene and determine the interactive role of each interactive object; Step S520: Establish independent information processing channels for each interactive object according to its interactive role, so as to process multimodal information through the analysis method of the corresponding interactive role.

[0044] For example, identifying multiple interactive objects in the current interactive scenario and determining the interactive role of each object refers to further analyzing the behavioral patterns, location, voice features, body language, and other multimodal information of each interactive object after detecting multiple potential interactive objects to determine the possible role each interactive object may play in the current interactive scenario. For example, interactive roles may include speaker, audience, observer, questioner, child, adult, etc. Here, interactive roles can be identified and classified according to preset rules or through machine learning models. An independent information processing channel is established for each interactive object based on its interactive role, so that multimodal information is processed through the analysis method corresponding to the interactive role. That is, different information processing strategies and resources are configured for different interactive roles. For example, for an interactive object identified as a speaker, the information processing channel may focus more on the accuracy of voice recognition and the depth of semantic understanding; while for an observer, the information processing channel may focus more on posture recognition and monitoring of potential intentions to ensure more accurate and efficient parsing of the multimodal information of each interactive object.

[0045] Understandably, by introducing interactive roles, humanoid robots can more precisely identify and understand the behavioral patterns and intentions of different interactive objects in specific scenarios. By establishing an independent information processing channel matching a role for each interactive object and adopting the corresponding analysis method for that role, a universal processing mode for all interactive objects can be avoided. This allows for dynamic adjustment of the focus of information processing and resource allocation based on the actual needs and behavioral characteristics of the interactive objects, thereby improving the accuracy and efficiency of multimodal information intent parsing. Simultaneously, humanoid robots can more intelligently adapt to complex multi-object interaction environments, effectively distinguish and respond to the needs of different interactive roles, improve the accuracy of intent parsing, and reduce the possibility of misjudgments and omissions.

[0046] For example, suppose a humanoid robot is interacting with a child in a family setting. The child is identified as the primary interaction target, and the interaction role is determined to be the child user. Meanwhile, the child's parent is observing nearby and is identified as the secondary interaction target, with the interaction role determined to be the adult observer. Subsequently, an information processing channel can be established for the child user. This channel can be configured to give higher weight and lower intent confidence thresholds to children's unique voice intonation, simple body movements, and facial expressions, to accommodate the characteristics of unclear expression or rapidly changing intents in children. Furthermore, this channel may also integrate a database of children's behavior patterns to predict the child's next action or need. Simultaneously, another information processing channel is established for the adult observer. This channel focuses more on monitoring overall posture, eye direction, and subtle voice cues, and sets a higher intent confidence threshold to avoid overreacting to the observer's unintentional behavior. The analysis method of this channel can be more inclined to identify potential intervention or guidance intentions. In this way, even if children and parents send multimodal information at the same time, the humanoid robot can process it using customized analysis methods according to their respective interaction roles, thereby more accurately understanding the child's interaction intentions, while also appropriately monitoring the parent's potential intervention needs, without causing interaction confusion due to the mixing of information from different roles.

[0047] like Figure 6 As shown, Figure 6 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Regarding step S130 above, the method may include, but is not limited to, steps S610 to S620.

[0048] Step S610: According to the preset priority evaluation rules, convert the current interaction scenario, the confidence of the intent of each interaction object, the intensity of the intent of each interaction object, the interaction role of each interaction object, and the degree of matching between the multimodal information of each interaction object and the corresponding interaction role behavior pattern into priority scores. Step S620: Determine the priority level for processing interaction requests of each interactive object based on priority scoring.

[0049] In some embodiments, the processing priority of an interaction request can be determined based on the current interaction scenario, intent confidence, and intent strength. However, in scenarios with multiple interaction objects, relying solely on these factors may not fully reflect the actual importance, urgency, and rationality of each interaction object's behavior patterns. Furthermore, by introducing interaction roles, role information can be effectively utilized to more accurately assess priorities.

[0050] Based on this, according to preset priority evaluation rules, the current interaction scenario, the intent confidence of each interaction object, the intent strength of each interaction object, the interaction role of each interaction object, and the degree of matching between the multimodal information of each interaction object and the corresponding interaction role's behavioral pattern can be converted into a priority score. Here, the preset priority evaluation rules refer to the standards and algorithms used to quantitatively evaluate the priority of interaction requests. The priority evaluation rules can be customized according to actual application needs, and different weights or scoring functions can be set for different interaction scenarios, interaction roles, intent confidence ranges, intent strength ranges, and behavioral pattern matching degrees.

[0051] For example, the priority score, which converts the current interaction scenario, the intent confidence of each interaction object, the intent strength of each interaction object, the interaction role of each interaction object, and the degree of matching between the multimodal information of each interaction object and the corresponding behavioral pattern of the interaction role, is a comprehensive numerical value calculated by considering data from multiple dimensions. The current interaction scenario may include time, location, and environmental state, such as whether it is a dangerous situation or whether an emergency response is needed; intent confidence represents the accuracy of recognizing the user's intent; and intent strength represents the intensity of the user's expressed intent. Interaction roles refer to the roles identified when recognizing interaction objects; interaction role behavior patterns refer to the typical behavior patterns or expected behavior norms associated with a specific interaction role. For example, children may exhibit more direct and emotional interaction behaviors, while adults may exhibit more rational and complex interaction behaviors; the degree of matching between multimodal information and the corresponding interaction role behavior patterns refers to whether the interaction object's behavior in multiple modalities such as vision, hearing, and touch conforms to the expected behavior patterns of the corresponding interaction role. For example, if an interaction object identified as a child exhibits behaviors such as crying and shouting urgently, it highly matches the behavior patterns of children in specific scenarios, thus affecting its priority score.

[0052] Determining the processing priority level of interaction requests for each interactive object based on the priority score means mapping the calculated priority score to a specific processing priority level. This can be achieved by setting a series of score thresholds, or by sorting the priority scores of all interactive objects to determine the relative processing priority level.

[0053] Understandably, intent confidence and intent strength reflect the clarity and urgency of the user's intent. The interaction scenario provides contextual information, enabling priority assessment to adapt to different environmental needs. The interaction role introduces considerations of user identity and expected behavior, allowing for differentiated processing based on the characteristics of different roles. The degree of matching between multimodal information and the corresponding interaction role's behavioral patterns further verifies the rationality and authenticity of the interaction behavior. Only through the comprehensive use of this multi-dimensional information can a more accurate and objective priority score be generated. Humanoid robots can more intelligently and accurately identify and respond to user interaction requests in a complex environment with multiple interaction objects.

[0054] For example, suppose in a family setting, a humanoid robot is playing and interacting with a child as the primary interaction target. At this moment, an adult as the secondary interaction target suddenly issues a voice command, asking the robot to fetch water from the kitchen. Meanwhile, the child accidentally falls while playing, cries, and tries to grab the robot's arm.

[0055] In this scenario, based on preset priority assessment rules, the current interaction environment is determined to be a home setting. The child's fall involves safety or comfort issues, indicating a high degree of urgency. The adult's voice commands are clear, with high confidence and moderate intensity, while the child's crying and grasping movements suggest a high degree of confidence and very high intensity of need for help. The child is the primary interaction target, and the adult is the secondary. The adult's commands conform to typical adult behavior patterns in a home setting, showing a high degree of match. The child's crying and seeking physical contact closely matches typical behavior patterns when experiencing discomfort or seeking help, showing a very high degree of match. Therefore, through priority assessment rules, the weight of the child's behavior patterns when experiencing discomfort is set higher than that of the adult's daily commands, and the weight of high-intensity emotional expressions is higher than that of medium-intensity commands. Considering all factors, the priority score for the child's interaction request is calculated to be significantly higher than that of the adult's interaction request. Thus, the child's interaction request is determined to have a higher processing priority. The humanoid robot will prioritize stopping playing with the child and focusing on the child's condition instead of immediately going to the kitchen to get water. After confirming that the child is unharmed and has been calmed down, the robot decides whether to carry out the task of fetching water based on the priority of the adult's request.

[0056] In another embodiment of this application, a multimodal human-computer interaction method for humanoid robots is provided, the interaction strategy includes: When interacting with the first interaction object, if the highest priority interaction request does not correspond to the first interaction object, and the priority score of the highest priority interaction request is lower than the preset score threshold, the humanoid robot's output is controlled through a preset hierarchical response process to respond to the highest priority interaction request, and the interaction with the first interaction object continues after the hierarchical response process is completed.

[0057] Understandably, in real-world multi-user interaction scenarios, when a humanoid robot is deeply interacting with a primary interaction object, if requests from other secondary interaction objects are determined to be of the highest priority, but their importance or urgency is not high enough to require an immediate interruption of the primary interaction, then simply processing them according to the highest priority principle may unnecessarily interrupt the primary interaction, affecting the smoothness of the interaction and the user experience; while completely ignoring such requests may cause the needs of secondary interaction objects to be overlooked.

[0058] Therefore, the interaction strategy provided in this application includes: when interacting with a first interaction object, if the interaction request with the highest priority does not correspond to the first interaction object, and the priority score of the interaction request with the highest priority is lower than a preset score threshold, the humanoid robot's output is controlled through a preset hierarchical response process to respond to the interaction request with the highest priority, and the interaction with the first interaction object continues after the hierarchical response process is completed. The highest priority interaction request is the one that has been evaluated and determined to have the highest processing priority among all requests from the current interaction objects. The preset scoring threshold is a pre-set value used to determine whether the highest priority interaction request is a secondary or tiered response. When the priority score of the highest priority request is lower than the preset scoring threshold, it indicates that although the request has the highest priority, its importance or urgency has not reached the point where the current main interaction needs to be interrupted immediately. The tiered response process is a preset non-disruptive lightweight response mechanism used to respond appropriately to secondary but high priority requests without completely interrupting the interaction with the first interaction object. For example, it may include short voice prompts, small body movements, eye contact, etc., to indicate that the robot has perceived the request, but will not immediately devote all resources to processing it.

[0059] Understandably, by introducing a priority score for the highest priority request and comparing it with a preset threshold, combined with a tiered response process, the humanoid robot can achieve more intelligent and flexible interaction management in complex multi-user interaction environments. This avoids unnecessary interruptions to primary interactions by simply following priority rules when faced with secondary but high-priority requests from non-primary interaction objects, thus improving the smoothness of the interaction and the user experience. Simultaneously, the tiered response mechanism ensures that even secondary requests receive timely and appropriate feedback, preventing user dissatisfaction due to ignored requests.

[0060] For example, suppose a humanoid robot is engaged in an in-depth conversation with a first interaction subject, discussing a complex technical issue. At this moment, a second interaction subject nearby softly calls out to the robot and points to a water glass on the table, indicating a request for water. The humanoid robot recognizes the second interaction subject's call and gesture, interpreting it as a request for water. After priority evaluation, the water request is determined to be the highest priority request due to its urgency. However, further priority evaluation reveals that the priority score is below a preset threshold because it does not involve safety issues, and the robot is currently engaged in an important conversation with the first interaction subject. In this situation, the humanoid robot does not immediately interrupt its conversation with the first interaction subject. Instead, it initiates a preset tiered response process. For instance, the robot might briefly turn its head towards the second interaction subject, making brief eye contact and issuing a short voice prompt. After completing the response, the robot immediately returns its attention to the first interaction subject and continues the previous conversation. When the conversation with the first interaction subject concludes or a gap occurs, the robot then processes the second interaction subject's water request according to its internal scheduling mechanism.

[0061] like Figure 7 As shown, Figure 7 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Regarding step S110 above, the method may include, but is not limited to, steps S710 to S720.

[0062] Step S710: Determine multiple physical objects in the current interactive scene by using the physical sensory information obtained by the physical sensors of the humanoid robot, and determine the feature points of multiple physical objects. The physical sensory information includes auditory sensory information and tactile sensory information. Step S720: Determine multiple visual objects in the current interactive scene by using the visual sensory information obtained by the humanoid robot's visual sensor. Based on the feature points of multiple physical objects and the multiple visual objects, determine multiple interactive objects in the current interactive scene.

[0063] For example, a humanoid robot is equipped with various sensors to perceive its surrounding environment. These physical sensors may include, but are not limited to, microphone arrays, tactile sensors, and force sensors, used to acquire physical sensory information. Auditory sensory information refers to sound data acquired through auditory sensors such as microphone arrays, including speech, ambient noise, and sounds of objects colliding. Tactile sensory information refers to data related to contact or force applied to objects, such as pressing, rubbing, and grasping, acquired through tactile sensors or force sensors. By processing and analyzing the physical sensory information, multiple physical objects in the current interaction scenario can be identified, and feature points of these objects can be further extracted, such as the location of the sound source, contact points, and areas of force application.

[0064] In addition, humanoid robots are also equipped with visual sensors, such as cameras and depth cameras, to acquire visual sensory information. Visual sensory information includes images, video streams, depth maps, etc. By performing image processing and computer vision algorithm analysis on visual sensory information, multiple visual objects in the current interactive scene can be identified, such as faces, human bodies, gestures, and object shapes.

[0065] Understandably, in complex and ever-changing interaction scenarios, a single perceptual modality may face limitations such as insufficient lighting, occlusion, background noise interference, or incomplete information, leading to insufficient accuracy and comprehensiveness in the recognition of interactive objects, which in turn affects the effectiveness of subsequent intent parsing and interaction strategies. Therefore, after acquiring the physical object feature points and the visual object, the information from these two different modalities is fused and correlated. For example, by matching the physical object feature points with the visual object in space and time, multiple interactive objects in the current interaction scenario can be more accurately identified. In other words, physical sensory information can provide clues about the existence, location, activity, or contact with the robot, while visual sensory information can provide the object's shape, color, posture, and more precise spatial positioning. The combination of the two can form a complementary advantage, associating and fusing the feature points of physical objects with visual objects. For example, when a sound is heard, the visual system can prioritize the area in the direction of the sound source for identification; when the robot touches an object, the visual system can confirm the identity and state of the object being touched. This allows humanoid robots to perceive and understand multiple potential interactive objects in the interactive scene more comprehensively and accurately, thus providing a solid foundation for subsequent intent parsing and interaction strategy formulation.

[0066] For example, suppose a humanoid robot is interacting with a user in a home environment, while other family members may be engaged in their own activities. The robot continuously listens to sounds in the environment through its microphone array. When it detects a child's laughter coming from the other side of the room, the robot can initially determine that a physical object exists in that direction and extract the sound source location as a feature point of the physical object. Simultaneously, the robot scans the room with its camera, identifying multiple visual objects, including the user interacting with it, a pet on the sofa, and the child making the laughter. Then, it matches and associates the feature points of the physical object determined by auditory sensory information with the visual objects identified by visual sensory information. For example, if the location of the laughter source is at the same height as the visually identified child, the child can be confirmed as an interactive object. Furthermore, if the robot accidentally touches a cup on the table, the tactile sensor will immediately provide tactile sensory information. Combined with visual information, it can accurately identify the touched physical object as a cup and recognize it as a non-human interactive object.

[0067] like Figure 8 As shown, Figure 8 This is a flowchart of a multimodal human-computer interaction method for humanoid robots provided in another embodiment of this application. Regarding step S720 above, the method may include, but is not limited to, steps S810 to S820.

[0068] Step S810: Determine the position and spatial depth of each visual object. Obtain the first condition by matching the position of each visual object with the feature points of multiple physical objects. Obtain the second condition by matching the spatial depth of each visual object with the preset human body spatial depth. Obtain the third condition by the physical sensory information of the position of each visual object. Step S820: Based on the first condition, the second condition, and the third condition, determine multiple interactive objects among multiple visual objects.

[0069] For example, determining the position and spatial depth of each visual object can be achieved using image data acquired by the visual sensors of a humanoid robot. Image processing and object detection algorithms are then used to identify the visual objects in the image and determine their positions in the two-dimensional image. Simultaneously, depth perception technologies such as binocular vision, structured light, or time-of-flight sensors can be used to acquire the depth information of each visual object in three-dimensional space. A first condition is obtained by comparing the position of each visual object with the feature points of multiple physical objects. The position information of the visual object in two-dimensional or three-dimensional space can be compared with the feature points of the physical objects acquired by physical sensors. If the position of the visual object and a certain physical object feature point are within a preset distance threshold, the matching degree is considered high, and the first condition is met.

[0070] The second condition is obtained by matching the spatial depth of each visual object with the preset human body spatial depth. This can be achieved by comparing the spatial depth information occupied by the visual object with the preset size range of a typical human body in three-dimensional space. For example, the height range of an adult can be preset as a reference for human body spatial depth. If the depth range of the visual object is within this preset range, the second condition is met, which helps to exclude non-human visual objects.

[0071] The third condition is derived from the physical sensory information of the location of each visual object. This could be determining whether corresponding physical sensory information exists in the area where the visual object is located, such as whether sound or touch is detected in the area where the visual object is located. The third condition further strengthens the recognition of interactive objects, ensuring that they are not only visually visible but also physically perceptible.

[0072] Based on the first, second, and third conditions, multiple interactive objects among multiple visual objects are identified. Logical combinations can be set, such as first condition AND (second condition OR third condition) or first condition AND second condition AND third condition, to determine which visual objects are the true interactive objects. In this way, by utilizing multimodal information for cross-validation, the accuracy and robustness of interactive object recognition are improved. It can more accurately determine which visual objects are truly humanoid interactive objects, avoiding the misidentification or omission problems that may occur when relying on only a single modality of information. This enables humanoid robots to more accurately perceive and understand their surrounding interactive environment.

[0073] This application also discloses a multimodal human-computer interaction system for humanoid robots, comprising: The object recognition module is used to identify multiple interactive objects in the current interactive scenario and to establish an independent information processing channel for each interactive object. The intent parsing module is used to perform intent parsing on the multimodal information of each of the information processing channels to obtain the intent confidence and intent strength of each of the interaction objects. The priority module is used to determine the processing priority level of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object. The control module is used to control the output behavior of the humanoid robot according to the processing priority level and the preset interaction strategy.

[0074] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multimodal human-computer interaction method for humanoid robots, characterized in that, include: Identify multiple interactive objects in the current interactive scenario and establish an independent information processing channel for each of the interactive objects; Intent parsing is performed on the multimodal information of each of the information processing channels to obtain the intent confidence and intent strength of each corresponding interaction object; Based on the current interaction scenario and the intent confidence and intent strength of each interaction object, determine the processing priority level of the interaction request for each interaction object; The output behavior of the humanoid robot is controlled according to the processing priority level and the preset interaction strategy.

2. The multimodal human-computer interaction method for humanoid robots according to claim 1, characterized in that, The step of determining the processing priority level of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object includes: Based on the current interaction scenario and the intent confidence and intent strength of each interaction object, the processing priority level of the interaction request of each interaction object is determined, and the first interaction object and the second interaction object among the plurality of interaction objects are determined. The first interaction object represents the main interaction object currently interacting with the humanoid robot to control the output behavior of the humanoid robot based on the main interaction object, and the second interaction object represents the user object with potential interaction intent.

3. The multimodal human-computer interaction method for humanoid robots according to claim 2, characterized in that, After determining the first and second interaction objects among the plurality of interaction objects, the multimodal human-computer interaction method for humanoid robots further includes: The computing resources allocated to the first interaction object and the second interaction object for intent monitoring are determined, wherein the computing resources allocated to the first interaction object are greater than the computing resources allocated to the second interaction object.

4. The multimodal human-computer interaction method for humanoid robots according to claim 2, characterized in that, The interaction strategy includes a first interaction strategy and a second interaction strategy. The first interaction strategy is used to control the interaction between the humanoid robot and the first interaction object, and the second interaction strategy is used to control the interaction between the humanoid robot and the first interaction object and the second interaction object. The step of controlling the output behavior of the humanoid robot based on the processing priority level and the preset interaction strategy includes: Adjust the processing priority level of each interaction object according to the intent confidence and intent strength of each interaction object; When the processing priority of the second interactive object is lower than that of the first interactive object, the output behavior of the humanoid robot is controlled based on the first interaction strategy. When the processing priority of the second interaction object is higher than that of the first interaction object, the output behavior of the humanoid robot is controlled based on the second interaction strategy.

5. The multimodal human-computer interaction method for humanoid robots according to claim 2, characterized in that, The process of identifying multiple interactive objects in the current interactive scenario and establishing an independent information processing channel for each interactive object includes: Identify multiple interactive objects in the current interactive scene and determine the interactive role of each interactive object; Based on the interaction roles of each interaction object, an independent information processing channel is established for each interaction object, so as to process multimodal information through the analysis method corresponding to the interaction roles.

6. The multimodal human-computer interaction method for humanoid robots according to claim 5, characterized in that, The step of determining the processing priority level of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object includes: According to the preset priority evaluation rules, the current interaction scenario, the confidence of the intent of each interaction object, the intensity of the intent of each interaction object, the interaction role of each interaction object, and the degree of matching between the multimodal information of each interaction object and the corresponding interaction role behavior pattern are converted into priority scores. Based on the priority score, the processing priority level of the interaction request for each of the interaction objects is determined.

7. The multimodal human-computer interaction method for humanoid robots according to claim 5, characterized in that, The interaction strategy includes: When interacting with the first interactive object, if the highest priority interaction request does not correspond to the first interactive object, and the priority score of the highest priority interaction request is lower than a preset score threshold, the humanoid robot's output is controlled through a preset hierarchical response process to respond to the highest priority interaction request, and the interaction with the first interactive object continues after the hierarchical response process is completed.

8. The multimodal human-computer interaction method for humanoid robots according to claim 5, characterized in that, The identification of multiple interactive objects in the current interactive scenario includes: The physical sensory information obtained by the physical sensors of the humanoid robot is used to determine multiple physical objects in the current interactive scene and to determine the feature points of multiple physical objects. The physical sensory information includes auditory sensory information and tactile sensory information. Multiple visual objects in the current interactive scene are determined by visual sensory information obtained by the visual sensors of the humanoid robot. Based on the feature points of the multiple physical objects and the multiple visual objects, multiple interactive objects in the current interactive scene are determined.

9. The multimodal human-computer interaction method for humanoid robots according to claim 8, characterized in that, The step of determining multiple interactive objects in the current interactive scene based on the multiple physical object feature points and multiple visual objects includes: The position and spatial depth of each visual object are determined. A first condition is obtained by matching the position of each visual object with the feature points of multiple physical objects. A second condition is obtained by matching the spatial depth of each visual object with the preset human body spatial depth. A third condition is obtained by using the physical sensory information of the position of each visual object. Based on the first condition, the second condition, and the third condition, multiple interactive objects among the multiple visual objects are determined.

10. A multimodal human-computer interaction system for humanoid robots, characterized in that, include: The object recognition module is used to identify multiple interactive objects in the current interactive scenario and to establish an independent information processing channel for each interactive object. The intent parsing module is used to perform intent parsing on the multimodal information of each of the information processing channels to obtain the intent confidence and intent strength of each of the interaction objects. The priority module is used to determine the processing priority level of the interaction requests of each interaction object based on the current interaction scenario and the intent confidence and intent strength of each interaction object. The control module is used to control the output behavior of the humanoid robot according to the processing priority level and the preset interaction strategy.