An active multi-round dialogue system and method for an embodied intelligent robot

Abstract

The present invention discloses an active multi-round dialogue system for an embodied intelligent robot, comprising: a data acquisition module for acquiring environmental information within the robot's service range; a history recording module for storing interaction information of multi-round human-machine dialogues based on the user's identity ID, the interaction information including dialogue content and interaction results; a dialogue generation module for generating reply text; an embodied intelligent algorithm module for generating action instructions to control the robot to approach the user and enter an interactive state; an active dialogue strategy module for generating corresponding active dialogue strategies; the robot performs dialogue tasks in the interactive state based on the generated reply text and active dialogue strategy. The present invention also provides an active multi-round dialogue method. The system provided by the present invention can effectively promote interaction tasks between the robot and the user in the form of multi-round dialogues, thereby improving the robot's dialogue achievement rate and service intelligence.

Description

Technical Field

[0001] The present invention belongs to the field of robot interaction control, and in particular relates to an active multi-round dialogue system and method for an embodied intelligent robot. Background Art

[0002] Common conversational robots have been widely used in intelligent customer service, smart speakers and other fields. They mainly include the following:

[0003] Speech Recognition (ASR): Converts user commands from speech into text. Semantic Understanding (NLU): Parses the text of user commands into understandable content for the robot. Dialogue Management (DM): Executes user intent, generates feedback signals, and determines the response strategy. Language Generation (NLG): Converts the dialogue management strategy into fluent text that the user can understand. Text-to-Text (TTS): Converts the text response into speech for the user.

[0004] For common voice chatbots, the multi-turn conversation management module generally consists of three main modules: conversation templates, conversation tracking, and conversation strategies. However, conversation scripts are usually written in advance, which can easily lead to multi-turn conversation failures. Furthermore, because most existing voice chatbots (such as smart speakers) lack active vision and mobility, and lack embodied intelligence, they primarily rely on microphones combined with some voiceprint recognition capabilities. However, in complex or highly variable acoustic environments, voiceprint recognition accuracy is not very high. As a result, the entire conversation management module often only remembers the first N rounds of conversation (typically N = 5) and cannot accurately distinguish between subject IDs. Voiceprint errors can often cause conversations between different people to be mixed together, or separate conversations with the same person, making it impossible for the conversation system to accurately record the conversation history of each service recipient.

[0005] Patent document CN116361442A discloses an AI-based business hall data analysis method and system. This method uses a pre-built intent recognition model to merge multiple input commands and classify related services based on the resulting sentence vectors, thereby improving service processing efficiency. This method proposes that multi-round conversation information is directly input via text, which can lead to classification errors due to the influence of ambient noise in real-world scenarios.

[0006] Patent document CN116364060A discloses a method, apparatus, device, and medium for testing an in-vehicle robot. The method includes constructing a conversation flow based on a conversation scenario; obtaining vehicle computer simulation instructions, configuring the vehicle computer simulation instructions and the in-vehicle robot interface to obtain robot simulation instructions; configuring the robot simulation answers for each round of conversation in the conversation flow corresponding to different vehicle models and vehicle computer versions, and configuring the corresponding robot simulation questions for the robot simulation answers after each completed conversation round to obtain a conversation test flow; and executing the conversation test flow, starting with the first round of conversation questions and making corresponding simulated responses, until the last round of conversation is completed or interrupted. This method is designed for vehicle interiors with low ambient noise, and the multi-round conversation proposed in the article is only executed in the order of user-initiated commands, which may cause command recognition errors to prevent subsequent commands from being executed. Summary of the Invention

[0007] The purpose of the present invention is to provide an active multi-round dialogue system and method for an embodied intelligent robot, which can effectively promote the interactive tasks between the robot and the user in the form of multi-round dialogue, thereby improving the robot's dialogue achievement rate and service intelligence.

[0008] To achieve the first objective of the present invention, an active multi-round dialogue system of an embodied intelligent robot is provided, comprising the following modules:

[0009] The data acquisition module is used to obtain environmental information within the robot's service range, including the relative position of the user and the robot, and the user's conversation status and conversation content.

[0010] The history record module stores the interaction information of multiple rounds of human-computer dialogues based on the user's identity ID. The interaction information includes the dialogue content and interaction results.

[0011] The dialogue generation module generates corresponding reply text based on the input dialogue content and the dialogue content retained by the user in the history module.

[0012] The embodied intelligence algorithm module is used to generate action instructions based on the acquired relative posture to control the robot to approach the user and enter an interactive state.

[0013] The active dialogue strategy module generates the corresponding active dialogue strategy based on the user's dialogue status and the interaction results in the history record module.

[0014] The robot performs conversation tasks in the interaction state based on the generated reply text and active conversation strategy.

[0015] The present invention redesigns the multi-round dialogue strategy and introduces the characteristics of embodied intelligent robots, allowing the robots to effectively use active vision and mobility capabilities to manage dialogue states and control the nodes and rhythm of active questioning, thereby effectively promoting the robot's ability to provide services to users in the form of multi-round dialogues.

[0016] Specifically, the data acquisition module includes an auditory unit, a visual unit and an analysis unit.

[0017] The auditory unit is used to collect sound information containing the user and / or other things in the environment.

[0018] The visual unit is used to collect image information containing users and / or other things in the environment.

[0019] The analysis unit is used to generate the relative posture of the user and the robot, and the user's dialogue status based on the sound information and image information.

[0020] Specifically, the analysis unit also includes the maintenance and update of the conversation state, and the expressions for the maintenance and update are as follows:

[0021] s t+1 ←s t +a t +o t+1

[0022] Where s t+1 represents the conversation state at time t+1, s t represents the dialogue state at the previous moment t, a t represents the system row at the previous time t, o t+1 Indicates the user action at time t+1.

[0023] Specifically, the relative posture includes the relative distance between the robot and the user, the horizontal angle, and the line of sight direction.

[0024] Specifically, the dialog state includes user actions and user action objects.

[0025] Specifically, the identity ID includes the user's face recognition information and / or voiceprint recognition information.

[0026] Specifically, the dialogue generation module includes a pre-built large language model, and the large language model generates relevant reply text according to the input dialogue content.

[0027] Specifically, the action instructions include body turning instructions and body movement instructions of the robot.

[0028] The body turning instruction is used to adjust the movement direction of the robot according to the source direction of the user's interactive voice.

[0029] The body movement instruction is used to drive the robot to move to the user's location based on the relative distance between the robot and the user.

[0030] Specifically, the active dialogue strategy includes selecting whether to actively initiate a dialogue according to the dialogue state of the target user when the target user has a failed interaction result.

[0031] To achieve the second objective of the present invention, a method for active multi-round dialogue is provided, which is implemented based on the active multi-round dialogue system of the embodied intelligent robot, and includes the following steps:

[0032] The robot generates a corresponding identity ID for the user with whom it has interacted within the service scope, and based on the identity ID, records the conversation content with the user, the execution time of the interactive task, and the interaction results.

[0033] The robot retrieves historical interaction information based on the identity ID of each user within the service scope. When a target user has a failed interaction result, the robot locks the target user and generates a corresponding reply text based on the historical conversation content and the current conversation content.

[0034] And based on the target user's conversation status, the corresponding active conversation strategy is generated.

[0035] Based on the generated active dialogue strategy and the corresponding reply text, perform the interaction task with the target user:

[0036] When the active conversation receives a response from the target user, the failed interaction task in the history record is advanced until it is completed;

[0037] When the active dialogue does not receive a response from the target user, the dialogue status of the target user is updated and the interactive task is retried.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] Based on the traditional voice dialogue robot, the embodied intelligent algorithm module is introduced and combined with a redesigned multi-round dialogue system, thus solving the problem that the existing voice robots cannot remedy in situations such as dialogue failure / task execution failure. By initiating dialogues and actively remedying problems through the robot's intelligent ability, the robot's dialogue completion rate and task completion rate are greatly improved, thereby improving service capabilities and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of an active multi-round dialogue system of an embodied intelligent robot provided in this embodiment;

[0041] Figure 2 A schematic diagram of the flow of the data acquisition module provided in this embodiment;

[0042] Figure 3 A schematic diagram of the active dialogue strategy after multiple rounds of dialogue fail provided in this embodiment;

[0043] Figure 4 A schematic diagram of the active dialogue strategy after a single-round dialogue fails provided in this embodiment;

[0044] Figure 5 A schematic diagram of the active dialogue strategy in the event of task failure provided in this embodiment. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] like Figure 1 As shown in FIG, an active multi-round dialogue system of an embodied intelligent robot includes the following modules:

[0047] The data acquisition module is used to obtain environmental information within the robot's service range, including the relative position of the user and the robot, and the user's conversation status and conversation content.

[0048] The history record module stores the interaction information of multiple rounds of human-computer dialogues based on the user's identity ID. The interaction information includes the dialogue content and interaction results.

[0049] The dialogue generation module generates corresponding reply text based on the input dialogue content and the dialogue content retained by the user in the history module.

[0050] The embodied intelligence algorithm module is used to generate action instructions based on the acquired relative posture to control the robot to approach the user and enter an interactive state.

[0051] The active dialogue strategy module generates the corresponding active dialogue strategy based on the user's dialogue status and the interaction results in the history record module.

[0052] The robot performs conversation tasks in the interaction state based on the generated reply text and active conversation strategy.

[0053] In addition, the history record module, dialogue generation module and active dialogue strategy module are deployed in cloud servers with greater computing power, while the embodied intelligence algorithm module is deployed locally in the robot, thereby improving the execution efficiency of dialogue tasks.

[0054] More specifically, the data acquisition module includes an auditory unit, a visual unit, and an analysis unit.

[0055] like Figure 2 As shown, the auditory unit includes a robot microphone array to obtain six-way acoustic signals within the robot's service range.

[0056] The vision unit uses an RGB camera and a depth camera to obtain planar image video streams and depth image video streams within the robot's service range.

[0057] The analysis unit generates corresponding relative posture and dialogue status based on the sound information and image information.

[0058] The relative posture is obtained by analyzing the robot's absolute coordinates and body angle (with due north as 0 degrees), the user's relative coordinates (relative to the robot or absolute coordinates), the straight-line distance between the user and the robot (obtained using a lidar or depth camera), the user's body orientation (relative to the robot's absolute coordinate system), and the user's facial orientation (captured based on facial recognition and based on the robot's absolute coordinate system).

[0059] The conversation status locks the user through the source of sound information and determines whether the user is using a communication device (such as a mobile phone, microphone) or whether there are other users around the user who are in a communication state. If the user is not using a communication device and communicating with other users, the user's voice content is analyzed to generate information about the user's actions and the objects of the user's actions.

[0060] For example, if a user moves a large angle during an exhibition, such as taking a few steps left or right to view an exhibit, causing the user to leave the camera's optimal field of view, the camera needs to be rotated to capture the best image data.

[0061] Adjust the camera focus distance - If the user moves a long distance during the exhibition, such as moving between exhibition points much faster than the robot, causing the user to leave the camera's optimal focus distance, the camera focus distance needs to be adjusted to collect the user's best image data.

[0062] The mobile chassis follows the customer. Users will move around the exhibition hall, so the robot needs to move its chassis to follow the user and provide continuous service, while ensuring that the user is within the robot's optimal field of view so that it can clearly see the user's movements and hear the user's voice.

[0063] Mobile chassis to search for customers. If the robot and the user become lost, for example, because the user suddenly goes outside to answer a phone call, and the robot cannot find the user by rotating the camera, the robot needs to move the chassis to actively search for the service object outside its current field of view so that it can continue to provide service after finding the user.

[0064] The history record module obtains the user's face recognition information and voiceprint recognition information to generate a unique identity ID for the user, and stores the user's historical conversation content and interaction results.

[0065] The dialogue generation module uses a pre-trained large language model to generate reply text. Based on the input dialogue content, it outputs the corresponding text, making the robot's multi-round dialogue more flexible and intelligent, without being restricted by fixed templates. This embodiment uses chatGPT as the framework for model construction.

[0066] This embodiment also provides an active multi-round dialogue method, which is based on the active multi-round dialogue system of the embodied intelligent robot proposed in the above embodiment. The specific process is as follows:

[0067] The robot generates a corresponding identity ID for the user with whom it has interacted within the service scope, and based on the identity ID, records the conversation content with the user, the execution time of the interactive task, and the interaction results;

[0068] The robot retrieves historical interaction information based on the IDs of each user within its service scope. If a target user has a failed interaction, it will lock the target user and generate a corresponding reply based on the historical conversation content and the current conversation content.

[0069] And based on the target user's conversation status, the corresponding active conversation strategy is generated.

[0070] Based on the generated active dialogue strategy and the corresponding reply text, perform the interaction task with the target user:

[0071] When the active dialogue receives a response from the target user, the failed interaction task in the history record is advanced until it is completed.

[0072] When the active dialogue does not receive a response from the target user, the dialogue status of the target user is updated and the interactive task is retried.

[0073] In order to better demonstrate the effect of the technical solution provided by this embodiment, several active dialogue strategies for specific scenarios are provided.

[0074] like Figure 3 As shown, the scenario is that there are unfinished multiple rounds of dialogue in the dialogue queue, such as speech recognition failure and missing parameters.

[0075] When a multi-turn conversation between a user and a robot requires multiple parameters, the robot can record the parameters required for the task and the parameters collected. If the task fails for any reason, the robot will record the current user X, the existing parameters for the task, and any missing parameters. The robot will then proactively initiate a multi-turn conversation at an appropriate time to continue executing the task with user X.

[0076] The history record module proactively restarts conversations based on unfinished tasks in the conversation tracking module (for example, a flight booking task that lacks a departure time parameter). The conversation management module doesn't rely on fixed templates; instead, it generates questions based on missing parameters. For example, if flight booking parameters are missing, the conversation system might ask the user, "Hello, you mentioned you wanted to book a flight from Hangzhou to Shenzhen on May 10th, but you haven't told me when you'd like to depart?" This facilitates the completion of the conversation.

[0077] Traditional voice robots (which lack embodied intelligence) are unable to identify the optimal time and accurately locate the user to restart a multi-turn conversation after multiple rounds of failed conversations. Therefore, their architecture doesn't store failed conversations or initiate recovery through proactive conversation strategy modules. They are forced to re-run the multi-turn dialogue template from the beginning, re-asking questions that have already been asked and answered correctly, resulting in a very poor user experience.

[0078] like Figure 4 As shown, there are unfinished single-round conversations in the conversation queue, such as speech recognition failure, intent understanding failure, etc.

[0079] A conversation might start as a single turn, but if it fails or stalls, it naturally needs to be resolved through multiple rounds to progress. For example, if voice pickup fails, the user's mouth may be moving while the robot is still face-to-face. In situations where the robot cannot understand the user's instructions or cannot hear clearly, a traditional conversational robot will not respond. However, an embodied intelligent robot will retry the failed turn based on its previous conversations with the user. The conversation generation module can also be used to fine-tune the conversational text using a large language generation model to make the interaction more natural.

[0080] like Figure 5 As shown, there are unfinished tasks in the task queue.

[0081] If the robot finds that the task mentioned in the conversation has not been executed, it will receive push notifications from the task management module in real time to align the task and conversation information.

[0082] If the conversation is successful, but a failed task execution push is received from the same user after the conversation, the robot's conversation and execution record module will discover this problem and promote the execution of the failed task by initiating multiple rounds of conversations.

[0083] For example, in a previous task, the user voice-commanded the robot to take them to the deep-sea fish exhibit. However, the user's attention was drawn to the Big Two-legged Fish exhibit and they went to see Big Two-legged Fish on their own. The user did not follow the robot to the deep-sea fish exhibit, and the robot stopped providing service after noticing the user was missing. This created an "uncompleted task record" in the user's conversation and task management history. Upon noticing the user's loss, the robot immediately turned its camera to locate the customer viewing Big Two-legged Fish not far away, moved over, and waited beside the user. After the user finished viewing Big Two-legged Fish and became available for service, the robot proactively said to the user, "Can I continue to take you to the deep-sea fish exhibit?" The customer replied, "Okay." The robot then continued to guide the user through the deep-sea fish exhibit.

[0084] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. An active multi-round dialogue system for an embodied intelligent robot, characterized in that: include: A data acquisition module is used to obtain environmental information within the robot's service range, including the relative position of the user and the robot, and the user's conversation status and content; A history recording module, which stores interaction information of multiple rounds of human-computer dialogues based on the user's identity ID, including dialogue content and interaction results; The dialogue generation module generates the corresponding reply text based on the input dialogue content and the dialogue content saved by the user in the history module; The embodied intelligence algorithm module is used to generate action instructions based on the acquired relative posture to control the robot to approach the user and enter the interactive state; An active dialogue strategy module generates a corresponding active dialogue strategy based on the user's dialogue state and the interaction results in the history module. The active dialogue strategy includes determining whether to actively initiate a dialogue based on the target user's dialogue state when there is a failed interaction result with the target user; The robot performs conversation tasks in the interaction state based on the generated reply text and active conversation strategy.

2. The active multi-round dialogue system of an embodied intelligent robot according to claim 1, characterized in that: The data acquisition module includes an auditory unit, a visual unit and an analysis unit; The auditory unit is used to collect sound information containing the user and / or other things in the environment; The visual unit is used to collect image information containing users and / or robots in the environment; The analysis unit is used to generate the relative posture of the user and the robot, and the user's dialogue status based on the sound information and image information.

3. The active multi-round dialogue system of an embodied intelligent robot according to claim 2, characterized in that: The analysis unit also includes the maintenance and update of the dialogue state, and the expressions for the maintenance and update are as follows: Where, represents the dialogue state at time t+1, represents the dialogue state at the previous moment t, represents the system row at the previous time t, Indicates the user action at time t+1.

4. The active multi-round dialogue system of an embodied intelligent robot according to claim 1 or 2, characterized in that: The relative posture includes the relative distance between the robot and the user, the horizontal angle, and the line of sight direction.

5. The active multi-round dialogue system of an embodied intelligent robot according to claim 1 or 2, characterized in that: The dialog state includes user actions and user action objects.

6. The active multi-round dialogue system of an embodied intelligent robot according to claim 1, characterized in that: The identity ID includes the user's face recognition information and / or voiceprint recognition information.

7. The active multi-turn dialogue system of an embodied intelligent robot according to claim 1, characterized in that: The dialogue generation module includes a pre-built large language model, which generates relevant reply text based on the input dialogue content.

8. The active multi-round dialogue system of an embodied intelligent robot according to claim 1, characterized in that: The action instructions include body turning instructions and body movement instructions of the robot; The body steering instruction is used to adjust the movement direction of the robot according to the direction of the user's interactive voice source; The body movement instruction is used to drive the robot to move to the user's location based on the relative distance between the robot and the user.

9. An active multi-round dialogue method, characterized in that: The active multi-round dialogue system based on the embodied intelligent robot according to any one of claims 1 to 8 is implemented, comprising the following steps: The robot generates a corresponding identity ID for the user with whom it has interacted within the service scope, and based on the identity ID, records the conversation content with the user, the execution time of the interactive task, and the interaction results; The robot retrieves historical interaction information based on the identity ID of each user within the service scope. If a target user has a failed interaction result, the robot will lock the target user and generate a corresponding reply text based on the historical conversation content and the current conversation content; And generate corresponding active dialogue strategies based on the target user's dialogue status; Based on the generated active dialogue strategy and the corresponding reply text, perform the interaction task with the target user: When the active conversation receives a response from the target user, the failed interaction task in the history record is advanced until it is completed; When the active dialogue does not receive a response from the target user, the dialogue status of the target user is updated and the interactive task is retried.

Images