A robot interaction method, device and robot

Abstract

This invention provides a robot interaction method, device, and robot, relating to the field of artificial intelligence technology. The method includes: acquiring sensor data from the robot and outputting multimodal features corresponding to the sensor data to a first model and a second model; determining first result information based on the multimodal features using the first model; controlling the robot based on the first result information, wherein the first result information includes a first response content to an inquiry, a first identifier, and prior inference features; determining second result information based on the multimodal features and prior inference features using the second model, and sending target feedback information to the first model during the determination of the second result information, wherein the second result information includes a second response content and an action sequence; controlling the robot to output the second response content and execute the action corresponding to the action sequence. This method solves the problems related to interaction efficiency and flexibility, and can improve interaction efficiency and flexibility.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robot interaction method, device, and robot. Background Technology

[0002] With the rapid development of service robots, smart terminals, and AI systems with multimodal interaction capabilities, users are placing higher demands on robot interaction experiences. On the one hand, robots need to possess human-like instant feedback capabilities, responding quickly to user input and reducing the feeling of waiting. On the other hand, robots also need to have complex semantic understanding, multi-turn task planning, knowledge retrieval, and action execution capabilities to complete intelligent interactions in complex scenarios. However, existing robot interaction methods have shortcomings in terms of interaction efficiency and flexibility. Summary of the Invention

[0003] This invention provides a robot interaction method, device, and robot, which can improve interaction efficiency and flexibility.

[0004] According to one aspect of the present invention, a robot interaction method is provided, comprising:

[0005] The robot's sensor data is acquired, and the corresponding multimodal features of the sensor data are output to the first model and the second model. The sensor data includes input voice, visual images, and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than that of the second model.

[0006] The first model determines first result information based on the multimodal features, and controls the robot based on the first result information. The first result information includes a first response content of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the features generated by the first model in the process of determining the first result information.

[0007] The second model determines the second result information based on the multimodal features and the prior inference features, and sends target feedback information to the first model during the process of determining the second result information. The target feedback information includes information generated by the second model during the process of determining the second result information, and the second result information includes second response content and action sequence.

[0008] The robot is controlled to output the second response content and execute the action corresponding to the action sequence.

[0009] According to another aspect of the present invention, a robot interaction device is provided, the device comprising:

[0010] The data acquisition module is used to acquire sensor data of the robot and output the multimodal features corresponding to the sensor data to the first model and the second model. The sensor data includes input voice, visual images and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than that of the second model.

[0011] The first response module is used to determine first result information based on the multimodal features through the first model, and control the robot based on the first result information. The first result information includes the first response content of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the feature generated by the first model in the process of determining the first result information.

[0012] The second response module is used to determine second result information based on the multimodal features and the prior inference features through the second model, and to send target feedback information to the first model during the process of determining the second result information. The target feedback information includes information generated by the second model during the process of determining the second result information, and the second result information includes second response content and action sequence.

[0013] The output execution module is used to control the robot to output the second response content and execute the actions corresponding to the action sequence.

[0014] According to another aspect of the present invention, a robot is provided, the robot comprising:

[0015] Sensor, used to collect the sensor data;

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that is executed by the at least one processor, such that the at least one processor can perform the robot interaction method described in any embodiment of the present invention.

[0019] The technical solution of this invention determines multimodal features based on robot sensor data, quickly determines first response content and prior inference features using a first model based on the multimodal features, and controls the robot based on the first response content. A second model determines second response content and action sequence based on the multimodal features and prior inference features, and controls the robot based on the second response content and action sequence. During the generation of the second response content and action sequence, target feedback information is sent to the first model. This enables the second model to generate second result information by combining multimodal features and prior inference features, and the first model can perceive the reasoning process of the second model, achieving deep information fusion between the two models. This balances the response speed and deep inference accuracy of robot interaction, solving the problems of interaction efficiency and flexibility in robot interaction methods, and improving both interaction efficiency and flexibility.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a robot interaction method is provided as an embodiment of the present invention;

[0023] Figure 2 A flowchart of another robot interaction method is provided for embodiments of the present invention;

[0024] Figure 3 A schematic diagram of the architecture of a robot interaction system provided by the present invention;

[0025] Figure 4 A flowchart illustrating yet another robot interaction method provided in an embodiment of the present invention;

[0026] Figure 5 A flowchart illustrating another robot interaction method provided by the present invention;

[0027] Figure 6 This is a schematic diagram of the structure of a robot interaction device provided in an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the structure of a robot that can be used to implement an embodiment of the present invention is shown. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Figure 1 This invention provides a flowchart of a robot interaction method, applicable to user-service robot interactions. The method can be executed by a robot interaction device, which can be implemented in hardware and / or software and configured within the robot. Figure 1 As shown, the method includes:

[0032] S110. Acquire the robot's sensor data and output the multimodal features corresponding to the sensor data to the first model and the second model. The sensor data includes input voice, visual images, and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than that of the second model.

[0033] The robot can include body-worn robots, for example, a robot may have components such as a head and / or arms. Optionally, the robot can be applied in hotels, tourist attractions, or other scenarios requiring human-computer interaction.

[0034] In one scenario, the robot is equipped with various types of sensors, such as voice sensors, image sensors, and posture sensors. Sensor data represents the data collected by the robot's sensors. For example, sensor data includes input voice, visual image data, and robot posture. Robot posture can include at least one of the following: the orientation angle of the robot's head and the coordinate sequence of preset points on the robot arm.

[0035] For example, the robot collects input speech, visual images, and robot posture in real time through sensors, and encodes these data to obtain multimodal features. In some cases, the multimodal features are output to the inference entry point of the first model; the multimodal features are also output to the data buffer of the second model, providing a data foundation for the first and second models to perform inference.

[0036] The first model can be a lightweight multimodal large model used to receive streaming text and image information units (tokens) and generate candidate outputs with extremely low latency, covering subconscious behaviors such as simple question and answer, gaze orientation, listening posture, and interruption pauses.

[0037] The second model can be a high-parameter multimodal model, which performs deliberate reasoning with the support of an intelligent system. The intelligent system includes a business knowledge base, a behavior base, and a skill base, used to achieve complex intent understanding, long-context reasoning, knowledge retrieval, and tool invocation.

[0038] Because the second model has more parameters than the first model, and requires a lot of reasoning to generate the response content, the response speed of the second model is slower than that of the first model.

[0039] In some cases, in response to receiving user input queries, the robot acquires sensor data and encodes it into multimodal features. These multimodal features are then sent to the inference entry point of the first model and the data buffer of the second model, respectively. When the second model is enabled, the multimodal features are read from the data buffer into the second model. This embodiment of the invention achieves rapid response through the first model and complex inference through the second model. Through information interaction between the two models, the second model utilizes the pre-reasoning features of the first model, while the first model can perceive the inference information of the second model and adjust its interaction strategy. This solves the problems of insufficient flexibility and wasted system resources caused by the two models operating independently.

[0040] S120. The first model determines first result information based on the multimodal features, and the robot is controlled based on the first result information. The first result information includes the first response content of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the feature generated by the first model in the process of determining the first result information.

[0041] The first response includes the answer text and a preset concluding statement. In some cases, if the query information does not conform to the preset scenario, the first response includes the answer text to the query information and a first action sequence. In other cases, if the query information conforms to the preset scenario, the first response includes a preset concluding statement and a second action sequence.

[0042] The preset scenarios can include pre-defined scenarios that require a response from the second model. Optionally, the preset scenarios include scenarios involving visual tasks and continuation tasks. Visual tasks include user questions related to on-screen content, such as identifying objects, viewing text, or judging a scene. Since the first model lacks deep image understanding capabilities, when a user question is detected to involve on-screen content, the second model needs to be invoked to generate the answer text for the user's question.

[0043] The continuation task includes user questions containing continuation commands such as "continue," "go on," and "you didn't finish speaking." Continuing the conversation requires restoring the interrupted historical dialogue context. Since this context is stored in the second model, the second model needs to be invoked to generate the answer text to the user's question.

[0044] In some cases, if the query indicates a simple user question, the first model outputs a complete answer text and a first action sequence. If the query indicates a complex user question, the first model outputs a continuation statement (without an answer text) and a second action sequence. Both the first and second action sequences include pre-defined basic action instructions. For example, "Let me think about this carefully," to avoid conflicts with subsequent outputs from the second model. Simple scenarios include everyday greetings, simple fact-finding, and basic action sequences. Complex scenarios include professional knowledge, online searches, multi-step tasks, and long-form logical question-and-answer sessions.

[0045] In this invention, the pre-inference features include preliminary semantic labels, intent classification results, and action baseline parameters. These pre-inference features are generated by the first model during the process of determining the first result information. Optionally, the pre-inference features are input into the data buffer of the second model as supplementary input. For example, the pre-inference features include a unique identifier for the current dialogue turn, response status, immediate response text, a first identifier, whether the second model needs to enable visual understanding, and stable transcription of the user text. The response status includes ignore, respond, and interrupt.

[0046] For example, the first model performs incremental inference on multimodal features and outputs the first response content, the first identifier, and the preceding inference features. The first model has advantages such as low latency optimization, small number of parameters, and fast inference speed. It can start generating by receiving a portion of streaming multimodal data, ensuring a fast response speed.

[0047] In some cases, the interaction strategy is adjusted by the first model based on the processing progress status and / or the tool information. For example, the first model receives target feedback information from the second model in real time, where the target feedback information includes processing progress status and / or tool information, etc. The processing progress status represents the processing progress of all nodes in the inference process of the second model. For example, the processing progress status includes waiting to start, deep inference in progress, text generation in progress, tool invocation in progress, and processing completed, etc. For the tool invocation in progress status, the tool name and the success or failure flag of the tool invocation are also sent to the first model. Optionally, the tool information includes a tool result summary, etc. The tool result summary is a summary of the tool invocation result, and the tool result summary is pushed to the first model so that the first model can quickly determine the answer text based on the tool result summary.

[0048] Optionally, the target feedback information is a structured information, including a unique identifier for this round of dialogue, processing progress status, tool name, tool call status, success or failure markers for the tool call, and a summary of the tool results.

[0049] S130. The second model determines the second result information based on the multimodal features and the prior inference features, and sends target feedback information to the first model during the process of determining the second result information. The target feedback information includes information generated by the second model during the process of determining the second result information, and the second result information includes the second response content and the action sequence.

[0050] The second response content can be a complete streaming text output after deep inference is performed by the second model. The action sequence is a set of action instructions corresponding to the second response content, output by the second model after deep inference is performed. The robot outputs the speech corresponding to the second response content and also parses the action sequence into robot joint control instructions, achieving synchronous execution of language and action.

[0051] For example, the multimodal features and the pre-inference features are input into the second model, and the second response content and the action sequence are output through the second model; during the generation of the second response content and the action sequence, the processing progress status and tool information are sent to the first model.

[0052] In some embodiments, multimodal features and pre-inference features corresponding to the current query information are read from the data cache of the second model and sent to the inference entry point of the second model. The second model performs deep inference on the multimodal features and pre-inference features, outputting streaming text content and choreographed head and limb action sequences. Throughout the inference process of the second model, target feedback information is pushed to the first model through a feedback channel. The feedback channel is a virtual channel for transmitting information between the second and first models. It should be noted that the feedback channel can be a bidirectional interactive channel; the second model can send target feedback information to the first model through the feedback channel, and the first model can send pre-inference features, etc., to the second model through the feedback channel.

[0053] S140. Control the robot to output the second response content and execute the action corresponding to the action sequence.

[0054] The execution of the responses output by the first and second models is controlled via a speech synthesis queue. Due to the rapid response of the first model, its first response enters the speech synthesis queue earlier than the second response. Text-to-speech operations are executed sequentially based on the order of the responses in the queue, and the robot is controlled to play the corresponding speech. During speech playback, basic action commands output by the first model, or action commands corresponding to the action sequences output by the second model, are also sent to the robot's action execution end. This action execution end includes components such as the robot's head and limbs.

[0055] By implementing the embodiments of the present invention, the following problems can be avoided:

[0056] On the one hand, robots use large-parameter language models or visual language models to complete interactive tasks. Due to the large number of parameters in these models and the complexity of the reasoning process, real-time voice interaction usually requires a long computation time, causing the robot to be unable to generate feedback quickly, and users are prone to experiencing a significant sense of waiting.

[0057] On the other hand, robots use lightweight models to achieve rapid response, but due to the limitation of model size, their semantic understanding ability, complex task planning ability and context association ability are limited. They are only suitable for simple question and answer, basic action control and other scenarios, and cannot meet the needs of complex tasks.

[0058] In some situations, combining a lightweight model and a complex model—using the lightweight model to quickly respond to simple interactions and the complex model to perform complex reasoning to respond to complex interactions—is problematic. However, since both models operate as independent processing modules, the lack of an effective information exchange mechanism between them prevents the complex model from utilizing the reasoning results already generated by the lightweight model. Furthermore, the lightweight model is unaware of the reasoning progress of the complex model, leading to prolonged periods of unresponsiveness and user confusion. Therefore, this approach suffers from insufficient interactive flexibility and wasted system resources.

[0059] This invention constructs a collaborative architecture with bidirectional communication, state alignment, and dynamic closed-loop, effectively balancing the model's rapid response capability with deep inference accuracy, and achieving seamless connection between the outputs of the first and second models. For example, in simple interaction scenarios, the first model quickly outputs the first response content, while in complex scenarios, the second model combines the pre-inference features of the first model to output the second response content. This achieves alignment of the semantics and action states of the first model during the inference process of the second model, ensuring a natural connection between the output of the second response content of the second model and the output of the first response content of the first model.

[0060] The technical solution of this invention determines multimodal features based on robot sensor data, quickly determines first response content and prior inference features using a first model based on the multimodal features, and controls the robot based on the first response content. A second model determines second response content and action sequence based on the multimodal features and prior inference features, and controls the robot based on the second response content and action sequence. During the generation of the second response content and action sequence, target feedback information is sent to the first model. This enables the second model to generate second result information by combining multimodal features and prior inference features, and the first model can perceive the reasoning process of the second model, achieving deep information fusion between the two models. This balances the response speed and deep inference accuracy of robot interaction, solving the problems of interaction efficiency and flexibility in robot interaction methods, and improving both interaction efficiency and flexibility.

[0061] Figure 2 This document provides a flowchart of another robot interaction method for embodiments of the present invention. The technical solution of the present invention can be combined with implementations of other embodiments. For the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 2As shown, the method in this embodiment may specifically include:

[0062] S210. Acquire the robot's sensor data and output the multimodal features corresponding to the sensor data to the first model and the second model. The sensor data includes input voice, visual images, and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than that of the second model.

[0063] S220. The first model determines first result information based on the multimodal features, and the robot is controlled based on the first result information. The first result information includes a preset concluding statement of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the feature generated by the first model in the process of determining the first result information.

[0064] S230. Input the multimodal features and the pre-inference features into the second model, and output the second response content and the action sequence through the second model.

[0065] For example, the multimodal features and pre-inference features in the data cache are input into the second model, and deep reasoning is performed through the second model to output complete streaming text content and an arranged head and limb movement sequence, which are then sent to the arbitration module.

[0066] The arbitration module maintains a global session version number, a queue of pending rounds, a speech synthesis queue, and an interruption context cache. It uniformly manages the execution order of the output results of the first and second models, the priority of dialogue rounds, and conflict resolution. It covers three core scenarios: output timing alignment, multi-round concurrent conflict handling, and full-process management of interruption and continuation. It fundamentally solves the problems of out-of-order, mixed playback, or context loss in streaming interaction.

[0067] S240. Add the preset continuation statement output by the first model and the second response content to the speech synthesis queue according to the generation time, wherein the speech synthesis queue is used to control the playback order of the speech corresponding to the second response content.

[0068] This invention relies on the first-in-first-out (FIFO) characteristic of speech synthesis queues. The preset continuation sentences output by the first model are queued first, and the response text output by the second model is queued later. This achieves the playback order of first broadcasting the continuation sentences and then providing detailed explanations, avoiding disordered output due to varying speeds. At the same time, during the reasoning process of the second model, the preset continuation sentences are played first to prompt the user to wait, avoiding long periods of no response and improving the interactive experience.

[0069] S250: In response to user input during voice playback, stop voice playback and interrupt action execution, record the interruption context of the second model, and control the robot to switch to a listening posture.

[0070] For example, if a user interruption is detected during voice playback, the voice playback is stopped, the robot's actions are interrupted, the interruption context of the second model for this round of dialogue is recorded, and the robot is simultaneously controlled to switch to a listening posture to provide data support for subsequent user instructions.

[0071] S260. During the generation of the second response content and the action sequence, a new round of dialogue is detected based on the global version number of the dialogue. The unfinished tasks of the old round of dialogue in the second model are abandoned, and the unplayed audio of the old round of dialogue is deleted.

[0072] For example, during the inference process of the second model, if the user initiates a new round of valid interaction, the global version number is updated based on the version number of the new round of dialogue. The new round of dialogue is confirmed to be effective by incrementing the global version number. The inference task of the second model for the unfinished old round of dialogue is canceled, the old round of voice to be played is cleared, and the subsequent output of the old round is automatically discarded due to the version number mismatch, so as to ensure that the user's new intention takes priority.

[0073] It should be noted that S240-S260 in the above embodiments are conditionally triggered. The corresponding steps will only be executed when the triggering conditions of the steps are met. Furthermore, the three steps are not executed sequentially. Their execution order can be the order shown in the above embodiments or the execution order can be changed. The present invention does not specifically limit this.

[0074] S270. During the process of generating the second response content and the action sequence, the processing progress status and tool information are sent to the first model.

[0075] In this embodiment of the invention, a low-latency first model rapidly generates structured decision data such as instant responses, action sequences, and contextual features, and synchronizes the pre-existing features of the first model to the data cache of the second model. When the second model performs deep inference by combining multimodal features and the pre-existing inference features of the first model, it aligns the semantics and action states of the first model to ensure a natural connection between the outputs of the second and first models. On one hand, during the deep inference process, the second model pushes the processing progress status of the inference flow to the first model. Based on the processing progress status, the first model adaptively adjusts the interaction logic, adapts to the tool call scenario, or reasonably interrupts the second model's inference task based on irrelevant new input dialogue (e.g., interrupting the second model's inference task when a new round of dialogue is detected), optimizing system resource utilization. On the other hand, during the deep inference process, the second model pushes a tool result summary to the first model. The first model can quickly generate answer text based on the tool result summary, which can be quickly applied to scenarios such as user follow-up questions or instant notification of the cause of call failure. By adopting the above methods, the efficiency of interactive response can be significantly improved.

[0076] S280. Control the robot to output the second response content and execute the action corresponding to the action sequence.

[0077] For example, the robot is controlled to broadcast the voice corresponding to the second response content, and the action sequence is parsed into control commands for the robot joints, so as to realize the synchronous execution of voice and action.

[0078] Figure 3 This is a schematic diagram of the architecture of a robot interaction system provided by the present invention. Figure 3As shown, this embodiment of the invention constructs a robot interaction architecture with dual-path parallelism, on-demand scheduling, and ordered output. The core consists of two independent inference paths: a fast path 310 and a slow path 320, complemented by a streaming multimodal input module 330, an arbitration module 340, and an output module 350. The streaming multimodal input module 330 serves as the data input entry point, connecting to the robot's audio and visual sensors, as well as the body control system. The body control system provides the robot's posture information, encoding this information into multimodal features with round-specific identifiers. These multimodal features can be presented as streaming text-to-image tokens 360. The streaming text-to-image tokens 360 are synchronously distributed to the inference entry point of the fast path 310 and the data buffer of the slow path 320. The fast path 310 deploys a lightweight multimodal model, receives the streaming text-to-image tokens 360, and generates candidate outputs with extremely low latency. For example, candidate outputs cover simple question-and-answer, gaze orientation, listening posture, and subconscious behaviors such as interruptions and pauses. The slow path 320 deploys a high-parameter multimodal model, performing deliberate reasoning with the support of an intelligent system. The intelligent system includes a business knowledge base, a behavior base, and a skill base, used for complex intent understanding, long-context reasoning, knowledge retrieval, and tool invocation. The arbitration module 340 maintains at least one of the following: a global session version number, a queue of pending rounds, a speech synthesis queue, resolves new and old rounds, handles interruptions, handles concurrent conflicts of multiple candidate rounds, and controls the playback order of fast and slow path outputs. The output module 350 parses the results processed by the arbitration module 340 into speech data and head and limb movement sequences, and sends them to the robot's execution end to achieve synchronized playback of speech and execution of actions corresponding to the movement sequences.

[0079] The technical solution of this invention, after outputting the second response content and action sequence through the second model, arbitrates the output results of the first and second models based on the global version number, thereby achieving output timing control, multi-round conflict resolution, and end-to-end management of interruption and continuation. This enables low latency, instant response, high-precision deep reasoning, and a smooth and natural streaming dialogue experience. It solves the problem that related technologies lack an output arbitration mechanism for real-time interaction, making it difficult to handle complex scenarios such as multi-round dialogue, user-initiated interruption, voice playback and action execution control, and multi-task concurrency. It achieves unified scheduling and dynamic adjudication of multiple output streams.

[0080] Figure 4 This is a flowchart illustrating another robot interaction method provided by an embodiment of the present invention. The technical solution of the present invention can be combined with the implementation methods of other embodiments. For the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 4 As shown, the method in this embodiment may specifically include:

[0081] S410. Acquire the robot's sensor data and output the multimodal features corresponding to the sensor data to the first model and the second model. The sensor data includes input voice, visual images, and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than that of the second model.

[0082] S420. The first model determines first result information based on the multimodal features, and the robot is controlled based on the first result information. The first result information includes the first response content of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the feature generated by the first model in the process of determining the first result information.

[0083] S430: Determine whether to call the second model. If yes, execute S440; otherwise, execute S460.

[0084] For simple interactive scenarios such as daily greetings or basic fact queries, the first model can generate and output the first response content without calling the second model. However, for complex interactive scenarios, it is necessary to combine the first and second models to output the second result information.

[0085] In some cases, the second model needs to be invoked when the following situations occur:

[0086] First, visual task triggering: The first model does not have deep image understanding capabilities. When it detects that the user's question involves the content of the screen, such as recognizing objects, viewing text, or judging the scene, the second model must be activated regardless of the first model's original judgment on whether the second model should be activated.

[0087] Second, continuation command triggering: When user input containing continuation commands such as "continue," "go on," or "didn't finish speaking" is detected, the second model needs to be activated. Since continuation requires restoring the interrupted historical dialogue context, this type of context is only stored in the second model; the first model cannot independently complete the continuation.

[0088] Third, verify the validity of the output of the first model. If any anomalies occur, such as empty text, multi-line output, inclusion of meta-description prefixes (e.g., "reply" or "analysis"), or concluding phrases exceeding the character limit, and it is determined that the second model is enabled, replace the output of the first model with preset concluding phrases. For example, preset concluding phrases include "Let me think about it first." If it is determined that the second model is not enabled, discard the output of the first model and disable the second model to avoid sending invalid content to users.

[0089] Optionally, the second model is activated if all of the following conditions are met: the current task has not been cancelled (the version number of the current dialogue round is consistent with the global version number), stable transcription of user speech is completed, the decision result of the first model has been returned, and the first flag is true. If the first flag is true, the second model is activated. If the first flag is false, the current round's data in the second model's data buffer is immediately released, the second model is not activated, and the robot is controlled based on the output result of the first model.

[0090] S440. The second model determines the second result information based on the multimodal features and the prior inference features, and sends target feedback information to the first model during the process of determining the second result information. The target feedback information includes information generated by the second model during the process of determining the second result information, and the second result information includes the second response content and the action sequence.

[0091] S450: Control the robot to output the second response content and execute the action corresponding to the action sequence.

[0092] S460. Control the robot to output the first response content and execute the preset action command.

[0093] The technical solution of this invention uses the first identifier output by the first model combined with multi-layer rule verification as the core basis for determining whether to enable the second model, thus avoiding misjudgment.

[0094] Figure 5 This is a flowchart illustrating another robot interaction method provided by the present invention. Figure 5 As shown, the method includes:

[0095] S510, Begin.

[0096] S520, System Initialization.

[0097] For example, load the weights of the first model and the second model, configure decision rules, output collaborative rules, and initialize the business knowledge base, behavioral tool interface and skill template library that accompany the second model.

[0098] S530, multimodal feature acquisition and splitting.

[0099] For example, the robot collects user voice, visual images and robot body state in real time through sensors, encodes them into multimodal features, and distributes the multimodal features to the inference entry of the first model and the data cache of the second model, respectively, to provide a data foundation for dual-path parallel inference.

[0100] S540, Multi-level decision verification.

[0101] For example, based on the first identifier output by the first model, multiple layers of rule verification are superimposed to generate the start flag of the second model, so as to determine whether the second model is enabled by the truth or falsehood of the start flag.

[0102] S550. Determine if the start flag of the second model is true. If it is, execute S560; otherwise, execute S570.

[0103] S560, Perform deep inference through the second model.

[0104] S570. Arbitrate the output results of the first model and the second model to determine the model output information.

[0105] The first response content output by the first model and the second response content output by the second model are added to the speech synthesis queue. Relying on the first-in-first-out characteristic of the speech synthesis queue, the first response content is dequeued before the second response content to avoid out-of-order output.

[0106] When a user initiates a new round of effective interaction, the new round is announced by incrementing the global version number. Incomplete reasoning tasks from the old round are canceled, and the audio from the old round to be played is cleared. Subsequent outputs from the old round are automatically discarded due to version number mismatch, ensuring that the user's new intent takes priority.

[0107] When a user interrupts while the robot is playing a voice message, the current voice playback stops, the current robot action is interrupted, the context of the interruption in the second model is recorded, and the robot is simultaneously controlled to switch to a listening posture to provide data support for subsequent commands.

[0108] S580, Execution output information.

[0109] By converting the first response output of the first model into speech and playing it through the robot, the robot is controlled to perform corresponding actions based on preset action commands. By converting the second response output of the second model into speech and playing it through the robot, the action sequence is parsed into robot joint control commands, achieving synchronous execution of language and actions.

[0110] S590, End.

[0111] Figure 6 This is a schematic diagram of the structure of a robot interaction device provided in an embodiment of the present invention. Figure 6 As shown, the device includes:

[0112] The data acquisition module 610 is used to acquire sensor data of the robot and output the multimodal features corresponding to the sensor data to the first model and the second model. The sensor data includes input voice, visual image and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than the response speed of the second model.

[0113] The first response module 620 is used to determine first result information based on the multimodal features through the first model, and control the robot based on the first result information. The first result information includes the first response content of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the feature generated by the first model in the process of determining the first result information.

[0114] The second response module 630 is used to determine second result information based on the multimodal features and the prior inference features through the second model, and to send target feedback information to the first model during the process of determining the second result information. The target feedback information includes information generated by the second model during the process of determining the second result information, and the second result information includes second response content and action sequence.

[0115] The output execution module 640 is used to control the robot to output the second response content and execute the action corresponding to the action sequence.

[0116] Optionally, the data acquisition module 610 is specifically used for:

[0117] Output the multimodal features to the inference entry point of the first model;

[0118] The multimodal features are output to the data buffer of the second model.

[0119] Optional, including:

[0120] If the query information does not conform to the preset scenario, the first response content includes the answer text to the query information and the first action sequence;

[0121] If the query information matches a preset scenario, the first response content includes a preset concluding statement and a second action sequence.

[0122] Optionally, the second response module 630 is specifically used for:

[0123] The multimodal features and the pre-inference features are input into the second model, and the second response content and the action sequence are output through the second model;

[0124] During the generation of the second response content and the action sequence, processing progress status and tool information are sent to the first model.

[0125] Optionally, after outputting the second response content and the action sequence through the second model, at least one of the following is also included:

[0126] The preset continuation statement output by the first model and the second response content are added to the speech synthesis queue according to the generation time, wherein the speech synthesis queue is used to control the playback order of the speech corresponding to the second response content;

[0127] In response to user input during voice playback, the voice playback is stopped and the action execution is interrupted. The interruption context of the second model is recorded, and the robot is controlled to switch to a listening posture.

[0128] Optionally, the process of generating the second response content and the action sequence further includes:

[0129] A new round of dialogue is detected based on the global version number of the dialogue. Unfinished tasks from the old round of dialogue in the second model are abandoned, and unplayed audio from the old round of dialogue is deleted.

[0130] Optional, also includes:

[0131] The interaction strategy is adjusted by the first model based on the processing progress status and / or the tool information.

[0132] Optionally, the following conditions may also be used to determine whether to enable the second model:

[0133] The version number of the dialog to which the query information belongs is consistent with the global version number;

[0134] User's voice transcription is now complete and stable.

[0135] The first result information of the first model has been returned;

[0136] The first identifier indicates that the second model is enabled.

[0137] The robot interaction device provided in the embodiments of the present invention can execute the robot interaction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0138] Figure 7 A schematic diagram of a robot that can be used to implement embodiments of the present invention is shown. A robot 700 includes:

[0139] Sensor 710 is used to collect the sensor data;

[0140] At least one processor 720; and

[0141] Memory 730 communicatively connected to the at least one processor; wherein,

[0142] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the robot interaction method described in the embodiments of the present invention.

[0143] The robot execution component 740 is used to execute actions corresponding to the action sequence.

[0144] In some embodiments, the robot interaction method is implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the robot via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random access memory (RAM) and executed by a processor, one or more steps of the robot interaction method described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform the robot interaction method by any other suitable means (e.g., by means of firmware).

[0145] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), system-on-a-chip (SoCs), complex programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0149] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0150] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0151] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A robot interaction method, characterized in that, include: The robot's sensor data is acquired, and the corresponding multimodal features of the sensor data are output to the first model and the second model. The sensor data includes input voice, visual images, and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than that of the second model. The first model determines first result information based on the multimodal features, and controls the robot based on the first result information. The first result information includes a first response content of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the features generated by the first model in the process of determining the first result information. The second model determines the second result information based on the multimodal features and the prior inference features, and sends target feedback information to the first model during the process of determining the second result information. The target feedback information includes information generated by the second model during the process of determining the second result information, and the second result information includes second response content and action sequence. The robot is controlled to output the second response content and execute the action corresponding to the action sequence.

2. The method according to claim 1, characterized in that, The step of outputting the multimodal features corresponding to the sensor data to the first and second models includes: Output the multimodal features to the inference entry point of the first model; The multimodal features are output to the data buffer of the second model.

3. The method according to claim 1, characterized in that, include: If the query information does not conform to the preset scenario, the first response content includes the answer text to the query information and the first action sequence; If the query information matches a preset scenario, the first response content includes a preset concluding statement and a second action sequence.

4. The method according to claim 1, characterized in that, The step of determining the second result information based on the multimodal features and the prior inference features through the second model, and sending target feedback information to the first model during the process of determining the second result information, includes: The multimodal features and the pre-inference features are input into the second model, and the second response content and the action sequence are output through the second model; During the generation of the second response content and the action sequence, processing progress status and tool information are sent to the first model.

5. The method according to claim 4, characterized in that, After outputting the second response content and the action sequence through the second model, it also includes at least one of the following: The preset continuation statement output by the first model and the second response content are added to the speech synthesis queue according to the generation time, wherein the speech synthesis queue is used to control the playback order of the speech corresponding to the second response content; In response to user input during voice playback, the voice playback is stopped and the action execution is interrupted. The interruption context of the second model is recorded, and the robot is controlled to switch to a listening posture.

6. The method according to claim 4, characterized in that, The process of generating the second response content and the action sequence also includes: A new round of dialogue is detected based on the global version number of the dialogue. Unfinished tasks from the old round of dialogue in the second model are abandoned, and unplayed audio from the old round of dialogue is deleted.

7. The method according to claim 4, characterized in that, Also includes: The interaction strategy is adjusted by the first model based on the processing progress status and / or the tool information.

8. The method according to claim 1, characterized in that, It also includes using the following conditions to determine whether to enable the second model: The version number of the dialog to which the query information belongs is consistent with the global version number; User's voice transcription is now complete and stable. The first result information of the first model has been returned; The first identifier indicates that the second model is enabled.

9. A robot interaction device, characterized in that, include: The data acquisition module is used to acquire sensor data of the robot and output the multimodal features corresponding to the sensor data to the first model and the second model. The sensor data includes input voice, visual images and robot posture. The input voice includes the query information input by the user to the robot. The response speed of the first model is greater than that of the second model. The first response module is used to determine first result information based on the multimodal features through the first model, and control the robot based on the first result information. The first result information includes the first response content of the query information, a first identifier, and a prior inference feature. The first identifier is used to indicate whether the second model is enabled, and the prior inference feature represents the feature generated by the first model in the process of determining the first result information. The second response module is used to determine second result information based on the multimodal features and the prior inference features through the second model, and to send target feedback information to the first model during the process of determining the second result information. The target feedback information includes information generated by the second model during the process of determining the second result information, and the second result information includes second response content and action sequence. The output execution module is used to control the robot to output the second response content and execute the actions corresponding to the action sequence.

10. A robot, characterized in that, The robot includes: Sensors are used to collect sensor data; At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the robot interaction method according to any one of claims 1-8.

Images