Terminal device and voice control method

Abstract

The embodiment of the invention provides terminal equipment and a voice control method. A controller in the terminal equipment is configured to determine a target intention corresponding to a voice instruction and a first subtask corresponding to the target intention based on a ReAct tool; processing the first subtask based on a large language model, and determining a first task queue corresponding to the first subtask; if the multiple first tools comprise the ReAct tool, based on the first execution sequence, the first subtasks are sequentially processed through all the first tools, and second subtasks are obtained; processing the second sub-task based on a large language model, and determining a second task queue corresponding to the second sub-task; and if the at least one second tool does not include the ReAct tool, processing the second subtask through the at least one second tool based on the second execution sequence to obtain a processing result of the target intention. The problem that the terminal equipment cannot understand or easily makes mistakes in a complex environment and instruction can be solved.

Description

Technical Field

[0001] The present application relates to the field of voice control technology, and in particular to a terminal device and a voice control method. Background Art

[0002] Usually, terminal devices (such as robots, smart home appliances, etc.) can receive user voice commands and perform corresponding operations based on the user's voice commands. For example, when a user sends a voice command to the robot, "What's the weather like today?", the robot can broadcast the corresponding reply information based on the voice command. However, when the voice command input by the user is more complex, such as the voice command input by the user is "turn on the air conditioner", the robot may not be able to execute the command or may make an execution error because it does not know which air conditioner to turn on, affecting the user experience. Summary of the invention

[0003] To solve the above problems, the embodiments of the present application provide a terminal device and a voice control method, which can realize multiple rounds of dialogue between the user and the terminal device, solve the problem that the terminal device cannot understand or is prone to making mistakes in complex environments and instructions, and improve the user experience.

[0004] In a first aspect, an embodiment of the present application provides a terminal device, including a communicator and a controller coupled to the communicator. The communicator is configured to receive a voice command input by a user; the controller is configured to: determine the target intent corresponding to the voice command and the first subtask corresponding to the target intent based on the reasoning and action ReAct tool; the terminal device includes a tool set, and the tool set includes the ReAct tool; the first subtask is processed based on a large language model to determine the first task queue corresponding to the first subtask; the first task queue includes multiple first tools in the tool set and the first execution order corresponding to the multiple first tools; if the multiple first tools include the ReAct tool, the first subtask is processed in sequence by each first tool based on the first execution order to obtain the second subtask; the second subtask is processed based on the large language model to determine the second task queue corresponding to the second subtask; the second task queue includes at least one second tool in the tool set and the second execution order of at least one second tool; if the at least one second tool does not include the ReAct tool, the second subtask is processed by at least one second tool based on the second execution order to obtain the processing result of the target intent.

[0005] Based on the above scheme, the terminal device provided in the embodiment of the present application understands the user's intention through the ReAct tool, splits the voice command input by the user into multiple subtasks, and then arranges each subtask in turn through the large language model to determine the tool to execute each subtask. The terminal device executes each subtask by calling the corresponding tool until the user's target intention is completed. Among them, when the tools determined by the large language model include the ReAct tool, the ReAct tool can determine the next subtask according to the current execution result, until the tools determined by the large language model do not include the ReAct tool, indicating that the terminal device has completed the user's target intention. In other words, the embodiment of the present application splits the complex voice commands input by the user through the mutual cooperation between the ReAct tool, the large language model and the tool set, and completes the user's target intention by processing each subtask in turn. The voice control method provided in the embodiment of the present application solves the problem that the terminal device cannot understand and is prone to making mistakes in complex environments or instructions, improves the intelligence level and practicality of the voice control interaction of the terminal device, and improves the user experience.

[0006] In some embodiments, the controller executes the first subtask based on the first execution order, and processes the first subtask in sequence through each first tool to obtain the second subtask. Specifically, it is configured as follows: if the ReAct tool is the last tool among multiple first tools, then based on the first execution order, other first tools except the ReAct tool are called in sequence to execute the first subtask to obtain the first execution result; the first execution result is processed by the ReAct tool to obtain the second subtask.

[0007] Based on the above scheme, when the large language model arranges tasks for the first subtask, if the first tool arranged includes the ReAct tool, the ReAct tool is the last tool. After the first tools before the ReAct tool are processed in sequence, the first execution result is obtained, and then the ReAct tool performs intent analysis based on the first execution result to determine the second subtask. The ReAct tool can split complex intents into simple intents by splitting tasks, further improving the intelligence level of voice control interaction of terminal devices.

[0008] In some embodiments, multiple first tools include a text-to-speech tool, a speech recognition tool and the ReAct tool; the controller executes based on the first execution order, and sequentially calls other first tools except the ReAct tool to perform the first subtask, and obtains a first execution result, which is specifically configured as: broadcasting the inquiry information to the user by calling the text-to-speech tool; receiving the reply voice input by the user by calling the speech recognition tool, and determining the reply text corresponding to the reply voice; wherein the first execution result includes the reply text.

[0009] Based on the above scheme, the terminal device can perform the first subtask by conducting a conversation with the user. The conversation process requires text-to-speech and speech recognition tools. The text-to-speech tool can play the inquiry information of the terminal device to the user, and the speech recognition tool can convert the reply voice of the voice input into reply text, thereby realizing multiple rounds of conversations between the terminal device and the user, which is convenient for the subsequent ReAct tool to perform intent understanding processing.

[0010] In some embodiments, the controller executes processing of the first execution result through the ReAct tool to obtain the second subtask, and is specifically configured as follows: processing the reply text information through the ReAct tool to obtain a third subtask; processing the third subtask based on the large language model, and determining a third task queue corresponding to the third subtask; wherein the third task queue includes at least one third tool in the toolset, and a third execution order corresponding to at least one third tool; if at least one third tool includes the ReAct tool, then based on the third execution order, the third subtask is processed in sequence through at least one third tool to obtain the second subtask.

[0011] Based on the above scheme, after obtaining the first execution result (such as the reply text), the third subtask may be generated through the intent analysis of the ReAct tool. The large language model can perform task scheduling for the third subtask, and call each third tool to execute the third subtask based on the task scheduling result, and obtain the second subtask after the ReAct tool performs intent analysis. In other words, the first subtask (such as the first subtask) and the second subtask (such as the last subtask) may also include at least one third subtask. The embodiment of the present application can split the user's target intent into multiple subtasks for processing in sequence, solving the problem that the terminal device cannot understand and is prone to making mistakes in complex environments or instructions.

[0012] In some embodiments, the controller is further configured to: determine the first prompt information corresponding to the first subtask through the ReAct tool, and send the first subtask and the first prompt information to the large language model. The controller performs processing on the first subtask based on the large language model, determines the first task queue corresponding to the first subtask, and is specifically configured to: process the first subtask and the first prompt information based on the large language model, divide the first subtask into multiple execution instructions, and determine the first tools corresponding to each execution instruction in the multiple execution instructions, as well as the first execution order corresponding to each first tool.

[0013] Based on the above scheme, when the large language model performs task scheduling processing on the first subtask, the ReAct tool can, in addition to sending the first subtask, also send a first prompt message corresponding to the first subtask. The first prompt message can enable the large language model to complete the task scheduling processing of the first subtask more quickly and accurately, thereby improving the processing efficiency of the large language model and the efficiency of voice processing of the terminal device.

[0014] In some embodiments, before the communicator executes the voice command received from the user, it is also configured to: receive a wake-up command input by the user; in response to the wake-up command, determine a wake-up task corresponding to the wake-up command, and a preset tool corresponding to the wake-up task; wherein the preset tools include a text-to-speech tool, a speech recognition tool, and a ReAct tool; and broadcast response information corresponding to the wake-up command to the user by calling the text-to-speech tool.

[0015] Based on the above solution, before the terminal device processes the user's voice command, the user needs to input a wake-up command to wake up the terminal device. By receiving the voice command after waking up the terminal device, the terminal device can be prevented from being in a voice receiving state continuously, and the terminal device can be prevented from executing other received voices as the user's voice command.

[0016] In some embodiments, the controller is further configured to: receive a voice instruction by calling a voice recognition tool, and determine text information corresponding to the voice instruction. The controller performs a ReAct tool to determine the target intent corresponding to the voice instruction, and a first subtask corresponding to the target intent, and is specifically configured to: process the text information through the ReAct tool, determine the target intent, and the first subtask.

[0017] Based on the above scheme, the task scheduling of wake-up intention can be preset in the terminal device. After receiving the wake-up word, the wake-up intention can be processed by calling the preset tool. The speech recognition tool in the preset tool is used to obtain and recognize the voice commands input by the user. The preset tool also includes the ReAct tool. The ReAct tool is used to analyze the intent of the voice commands input by the user to determine the user's target intention and the first subtask after splitting.

[0018] In some embodiments, the ReAct tool includes an interaction module, and the controller executes processing of the first subtask based on the large language model to determine the first task queue corresponding to the first subtask, and is specifically configured as follows: sending the first subtask to the large language model through the interaction module to perform task scheduling processing on the first subtask through the large language model to obtain the first task queue; obtaining the first task queue through the interaction module.

[0019] Based on the above solution, the ReAct tool includes an interaction model, and the interaction module is used to realize the interaction between the ReAct tool and the large language model. The first subtask generated by the ReAct tool can be sent to the large language model for task scheduling, and the first task queue after task scheduling sent by the large language model can be received.

[0020] In some embodiments, the ReAct tool further includes a memory module, and the controller is further configured to: store the first subtask and the first task queue corresponding to the first subtask through the memory module.

[0021] Based on the above solution, the ReAct tool also includes a memory module, which is used to store the intent analysis results of the ReAct tool and the task scheduling results of the large language model, so that when the same question appears later, it can be processed directly, thereby improving the efficiency of voice processing of the terminal device.

[0022] The second aspect of the embodiment of the present application provides a voice control method, which is applied to a terminal device, and the method includes: first, obtaining a voice command input by a user, and determining a target intent corresponding to the voice command and a first subtask corresponding to the target intent based on the reasoning and action ReAct tool; wherein the terminal device includes a tool set, and the tool set includes the ReAct tool. Secondly, processing the first subtask based on the large language model to determine a first task queue corresponding to the first subtask; wherein the first task queue includes multiple first tools in the tool set, and a first execution order corresponding to the multiple first tools. Then, if the multiple first tools include the ReAct tool, then based on the first execution order, the first subtask is processed in sequence by each first tool to obtain a second subtask. Next, processing the second subtask based on the large language model to determine a second task queue corresponding to the second subtask; wherein the second task queue includes at least one second tool in the tool set, and a second execution order corresponding to the at least one second tool. Finally, if the at least one second tool does not include the ReAct tool, then based on the second execution order, the second subtask is processed by at least one second tool to obtain a processing result of the target intent.

[0023] Based on the above scheme, the voice control method provided in the embodiment of the present application understands the user's intention through the ReAct tool, splits the voice command input by the user into multiple subtasks, and then arranges each subtask in turn through the large language model to determine the tool to execute each subtask. The terminal device executes each subtask by calling the corresponding tool until the user's target intention is completed. Among them, when the tools determined by the large language model include the ReAct tool, the ReAct tool can determine the next subtask according to the current execution result, until the tools determined by the large language model do not include the ReAct tool, indicating that the terminal device has completed the user's target intention. In other words, the embodiment of the present application splits the complex voice commands input by the user through the mutual cooperation between the ReAct tool, the large language model and the tool set, and completes the user's target intention by processing each subtask in turn. The voice control method provided in the embodiment of the present application solves the problem that the terminal device cannot understand and is prone to making mistakes in complex environments or instructions, improves the intelligence level and practicality of the voice control interaction of the terminal device, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0026] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0027] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0028] Figure 4A A schematic diagram of a voice control system provided in an embodiment of the present application;

[0029] Figure 4B A schematic diagram of a voice control method provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of another voice control method provided in an embodiment of the present application;

[0031] Fig. 6A A schematic diagram of another voice control method provided in an embodiment of the present application;

[0032] Figure 6B A schematic diagram of another voice control method provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a speech processing process provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of another speech processing process provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0037] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0038] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0039] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0040] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0041] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0042] Figure 1 Schematic diagram of an operation scenario between a display device and a control device provided in an embodiment of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0043] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0044] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0045] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0046] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0047] Figure 2 Provided in the embodiments of this application Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0048] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0049] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0050] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0051] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0052] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0053] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0054] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0055] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0056] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0057] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0058] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0059] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0060] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0061] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0062] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0063] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0064] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0065] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0066] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0067] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0068] Usually, terminal devices (such as robots, smart home appliances, etc.) can interact with users through voice and perform corresponding operations according to the user's voice commands. For example, the user can send some simple voice commands to the robot, such as "What's the weather like today?" or "Turn on the TV in the living room." After receiving the command, the robot can send the voice command to the server, which will convert the voice command into a text command through the natural language processing model deployed on the server and send it to the robot. The robot will call the voice recognition tool according to the text command to answer the user's question, or call the appliance control tool to turn on the TV in the living room.

[0069] However, when the voice command input by the user is complex, the robot may fail to execute or execute incorrectly due to incomprehension. For example, when the user inputs the voice command "turn on the air conditioner", since there are multiple air conditioners in the environment, the robot may not know which air conditioner to turn on and cannot perform the corresponding operation, thus affecting the user experience.

[0070] In order to solve the above problems, the embodiments of the present application propose a terminal device and a voice control method, which integrates a Reasoning and Acting (ReAct) tool in the terminal device to implement the intent analysis of the voice commands input by the user through the ReAct tool, and split the target intent corresponding to the complex command into multiple subtasks, and then arrange the subtasks through a large language model, determine the tools for executing each subtask, and then the terminal device executes each subtask in turn by calling each tool, and finally completes the user's target intent.

[0071] Figure 4A A schematic diagram of a voice control system provided in an embodiment of the present application. Figure 4A As shown, the voice control system includes a large language model, an Agent and a tool set, wherein the Agent can be a terminal device.

[0072] In some examples, the Large Language Model (LLM) is the core component of the voice control system. The large language model in the embodiment of the present application can adopt the Transformer architecture, and has made improvements in parameter scale and training data diversity, respectively, to enhance the natural language communication capabilities in the fields of smart home appliances and robots. Agent is a bridge between the large language model and the toolset. Agent receives the user's voice command, converts it into text, understands the user's intent, and sends the intent understanding result to the large language model, which performs task scheduling. Among them, the user intent understanding can be executed by Agent Core, and Agent Core can be the ReAct tool. The ReAct tool is the core of the Agent, which can ensure continuous dialogue between the terminal device and the user, simplify complex tasks and promote the execution of intent.

[0073] like Figure 4A As shown in the figure, after obtaining the task scheduling structure of the large language model, Agent Core sends the task scheduling result to the intent tree, and the intent tree is handed over to the task execution module (Runner) in the Agent to issue the task. The tool set (Agent SDK) is the smallest unit of task execution in the terminal device. The tools in the tool set can be determined according to the application scenario of the terminal device. The tools are independent of each other, and multiple tools can complete the execution of a task through mutual collaboration.

[0074] In some examples, in order to achieve continuous dialogue between the terminal device and the user, the agent can also integrate a long short-term memory (LSTM) module. The LSTM module can capture the context information in the dialogue to ensure that the terminal device has a complete and accurate understanding of the user's needs. At the same time, the agent can also provide guidance on intent parsing for the large language model through the prompt module, thereby generating task scheduling results that the Runner can recognize and execute.

[0075] The voice control method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0076] Figure 4B A schematic diagram of a voice control method provided in an embodiment of the present application. Figure 4B As shown, the method includes steps 410 to 450 as shown below.

[0077] In some examples, the voice control method provided in the embodiments of the present application can be applied to a terminal device, which can be a robot or other terminal devices, such as the display device 200 and other smart home appliances in the above embodiments. The embodiments of the present application do not limit the specific type of the terminal device. The following embodiments can be exemplified by taking the terminal device as a robot.

[0078] Step 410, obtain the voice command input by the user, and determine the target intention corresponding to the voice command and the first subtask corresponding to the target intention based on the reasoning and action ReAct tool.

[0079] Exemplarily, the terminal device includes a tool set, which includes multiple tools. The tool set is a set of functional modules or services (such as tools) integrated on the software or hardware platform of the terminal device. The tools work together to support the terminal device to complete specific tasks or provide related services. The tool set can enhance the intelligence, flexibility and ease of use of the terminal device.

[0080] For example, the tool set of the terminal device may include multiple types of tools, such as perception tools, decision and planning tools, execution tools, interaction tools, management tools, etc. Among them, each type of tool may include at least one tool, such as the perception tool may include a visual recognition tool, a voice recognition tool, etc.

[0081] Exemplarily, the multiple tools in the toolset include the ReAct tool, which is similar to other tools and is used to perform corresponding tasks. ReAct is a technical framework that can give the Large Language Model (LLM) the ability to solve problems through reasoning and action, enabling artificial intelligence (AI) to analyze problems, make plans, perform operations, observe results, etc., and dynamically adjust its actions based on new information until the problem is satisfactorily solved. The embodiment of the present application designs the ReAct tool based on the ReAct framework concept, and integrates the ReAct tool into the robot's toolset to implement intent analysis of voice commands input by users.

[0082] In some examples, the terminal device may obtain a voice command input by a user, and the voice command may include an intelligent control command, an information query command, a reminder command, an interactive command, and a navigation command, etc. For example, a user may input an intelligent control command such as "turn on the air conditioner, play music", or may input an information query command such as "what is the weather like today, recommend a good movie", etc. The specific type of the voice command is not limited in the embodiment of the present application.

[0083] For example, the voice command input by the user can be a simple voice command or a complex voice command. When the user inputs a relatively simple voice command, such as "How is the weather today?", the robot can directly execute it and give the user the execution result, but when the user inputs a relatively complex voice command, the robot may not be able to execute the complex command.

[0084] In some examples, in order to ensure that the robot can more accurately obtain the voice commands input by the user, it is necessary to ensure that the robot is in an awake state before the user inputs the voice commands.

[0085] Figure 5 A schematic diagram of another voice control method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, before step 420, the method further includes steps 510 to 540 as shown below. Steps 510 to 520 may be performed before step 410, and step 410 may include steps 530 to 540.

[0086] Step 510: Obtain a wake-up instruction input by the user, and determine a preset tool corresponding to the wake-up task in response to the wake-up instruction.

[0087] In some examples, before the user inputs a voice command, the user needs to input a wake-up command to wake up the robot and put the robot into a wake-up state. When the robot is in a wake-up state, the terminal device can receive the voice command that the user can input. For example, the wake-up command can be set according to demand. For example, when the wake-up command is set to "Hi, xx", the robot can be woken up when the user inputs the voice of "Hi, xx". The specific form of the wake-up command is not limited in the embodiment of the present application.

[0088] Exemplarily, after receiving the wake-up instruction input by the user, the terminal device may respond to the wake-up instruction to execute the wake-up task. A preset tool for executing the wake-up task may be pre-set in the terminal device.

[0089] In some examples, when the robot receives the wake-up command, it can call various preset tools to perform the wake-up task. The preset tools may include a text-to-speech (TTS) tool, an automatic speech recognition (ASR) tool, and a ReAct tool.

[0090] Step 520: broadcast the response information corresponding to the wake-up instruction to the user by calling a text-to-sound tool.

[0091] In some examples, a text-to-speech tool (hereinafter also referred to as a TTS tool) can be used to convert text content into speech for output. For example, after receiving a wake-up instruction input by a user, the terminal device broadcasts response information to the user through the TTS tool.

[0092] For example, when the user enters the wake-up command "Hi, xx", the robot can call the TTS tool to broadcast the response message "I am here" to the user to inform the user that the robot has entered the wake-up state and can further receive relevant voice commands entered by the user.

[0093] Step 530, receiving a voice instruction by calling a voice recognition tool, and determining text information corresponding to the voice instruction.

[0094] In some examples, after obtaining the response information played by the terminal device, the user can enter a voice command. After the user enters the voice command, the terminal device can call a speech recognition tool (hereinafter also referred to as an ASR tool) to convert the obtained voice command into text information. Among them, the ASR tool can convert the voice form into a readable text form through operations such as voice input conversion and natural speech understanding, and perform natural speech processing on the converted text, so that the terminal device can perform the next step of processing and response.

[0095] For example, when a user inputs a voice command of “Please turn on the air conditioner”, the robot can convert the voice information of “Please turn on the air conditioner” into the text information of “Turn on the air conditioner” by calling the ASR tool.

[0096] Step 540, process the text information through the ReAct tool to determine the target intention and the first subtask.

[0097] In some examples, after converting the voice command into text information, the terminal device can continue to call the ReAct tool to further process the text information, such as intent understanding processing, to determine the target intent corresponding to the voice command, and the first subtask. The intent understanding processing of the ReAct tool includes determining the user's target intent based on the text information, and determining whether the target task corresponding to the target intent needs to be split into multiple simple subtasks (such as the first subtask) based on the target intent.

[0098] For example, when the user inputs the voice command "turn on the air conditioner", the ReAct tool determines that the user's target intention is to turn on the air conditioner (i.e., the target task). However, since multiple rooms in the user's space may have corresponding air conditioners, and the user did not instruct which air conditioner to turn on, in this case, the ReAct tool can first split the target task into the first subtask, that is, asking the user which air conditioner to turn on (i.e., the first subtask). In other words, the ReAct tool can generate the first subtask of "asking the user which air conditioner to turn on" based on the target intention "turn on the air conditioner".

[0099] It should be noted that the first subtask split by the ReAct tool is only an example, and the ReAct tool can generate the corresponding first subtask according to the actual situation. For example, the ReAct tool can also generate the first subtask of "asking the user whether to turn on the air conditioner in the living room" based on judging that the user is in the living room.

[0100] Step 420: Process the first subtask based on the large language model to determine a first task queue corresponding to the first subtask.

[0101] Exemplarily, the large language model is a pre-trained model, and the large language model is used to perform task scheduling processing on the tasks (such as the first subtask) sent by the ReAct tool. The task scheduling processing may include determining the tools required by the terminal device to perform each task, and the order in which each tool performs the task. For example, after the terminal device generates the first subtask by calling the ReAct tool, the ReAct tool sends the first subtask to the large language model, so that the large language model is used to perform task scheduling processing on the first subtask and determine the first task queue corresponding to the first subtask.

[0102] In some examples, the large language model can be deployed on a server or on a terminal device.

[0103] For example, when the large language model can be deployed on the server, after the terminal device determines the first subtask by calling the ReAct tool, the first subtask can be sent to the server. The server schedules the first subtask through the large language model and sends the task scheduling result (i.e., the first task queue) to the terminal device. When the large language model is deployed on the terminal device, after the first subtask is determined by the ReAct tool, the ReAct tool sends the first subtask to the large language model, and after processing by the large language model, the obtained first task queue is sent to the ReAct tool. The embodiments of the present application do not limit the specific deployment of the large language model.

[0104] Exemplarily, after the large language model performs task scheduling processing on the first subtask, a task scheduling result is obtained, and the task scheduling structure includes a first task queue corresponding to the first subtask. The first task queue includes multiple first tools in the tool set and a first execution order corresponding to the multiple first tools.

[0105] In some examples, the plurality of first tools are determined according to the first subtask, and the first tools are tools required for performing the first subtask. For example, the plurality of first tools may include a TTS tool, an ASR tool, and a ReAct tool.

[0106] In some embodiments, the above step 420 includes: determining the first prompt information corresponding to the first subtask through the ReAct tool, and sending the first subtask and the first prompt information to the large language model; processing the first subtask and the first prompt information based on the large language model, dividing the first subtask into multiple execution instructions, and determining the first tools corresponding to each execution instruction in the multiple execution instructions, as well as the first execution order corresponding to each first tool.

[0107] Exemplarily, when sending the first subtask to the large language model, the ReAct tool may also send the first prompt information corresponding to the first subtask to the large language model. The first prompt information is information that prompts the large language model to schedule the first subtask, and the first prompt information can improve the efficiency and accuracy of the large language model in scheduling tasks.

[0108] In some examples, the task scheduling process of the large language model includes performing natural language processing on the first subtask and the first prompt information, dividing the first subtask into multiple execution instructions, thereby determining a first tool for executing each execution instruction, and an execution order of each execution instruction, i.e., a first execution order.

[0109] It should be noted that the task scheduling process of the large language model can be a task refinement process. The large language model divides the first subtask into multiple execution instructions according to the first subtask and the first prompt information, determines the execution order of the multiple execution instructions, and each execution instruction corresponds to the first tool. After completing the task scheduling, the large language model will hand over the task scheduling results to the ReAct tool according to the sequence, and then pass it to the intent tree in the terminal device for task execution.

[0110] For example, the ReAct tool sends the first subtask of "asking the user which air conditioner to turn on" and the prompt information "turn on the air conditioner" corresponding to the first subtask to the large language model, and the large language model performs task arrangement and splits "asking the user which air conditioner to turn on" into multiple execution instructions such as "asking the user", "processing the user's answer", and "determining the next subtask based on the user's answer". Among them, the first tool for executing the execution instruction of "asking the user" can be a TTS tool, the first tool for executing the execution instruction of "processing the user's answer" can be an ASR tool, and the first tool for executing the execution instruction of "determining the next subtask based on the user's answer" is the ReAct tool, and the first execution order of each first tool is TTS tool, ASR tool, and ReAct tool.

[0111] In some examples, the ReAct tool is responsible for calling the large language model, and the intent information (such as the first subtask) and the prompt word information (such as the first prompt information) are submitted to the large language model through the ReAct tool, and the large language model completes the task arrangement based on the intent information and the prompt word information. In each round of task arrangement before the end of each subtask, the ReAct tool will be used as the last tool for arrangement, so as to ensure that the intent can continue to be arranged by the large language model after the current round of tasks is completed.

[0112] Exemplarily, the ReAct tool includes a tool module, which can ensure that the ReAct tool can be normally called by the task execution module.

[0113] In some embodiments, the ReAct tool further includes an interaction module, a processing module, and a memory module.

[0114] In some embodiments, step 420 includes: sending the first subtask to the large language model through the interaction module, so as to perform task scheduling processing on the first subtask through the large language model to obtain a first task queue; and obtaining the first task queue through the interaction module.

[0115] In some examples, the interaction module may send the first task queue to the processing module, and the processing module may send the first task queue to the task execution module of the terminal device.

[0116] Exemplarily, the interaction module in the ReAct tool is used to implement the interaction between the ReAct tool and the large language model. The interaction module sends the first subtask and the first prompt information generated by the ReAct tool to the large language model. The large language model performs task scheduling on the first subtask, obtains a first task queue, and then sends the first task queue to the interaction module. The interaction module sends the first task queue to the processing module, and the processing module sends the first task queue to the intent tree in the terminal device, and the intent tree executes the task through the task execution module (TaskRunner). In other words, the ReAct tool plays a connecting role in the voice interaction process and promotes the operation of the voice control system.

[0117] In some embodiments, the method further includes: storing the first subtask and the first task queue corresponding to the first subtask through a memory module.

[0118] In some examples, the memory module is used to record the execution results of each round of tasks during the intent processing process, so as to deliver comprehensive information to the large language model for reasonable task scheduling. For example, the memory model can record the intent understanding results of the ReAct tool, and can also record the task scheduling results of the large language model.

[0119] For example, prompt information for subsequent task scheduling by the large language model may be determined based on information recorded by the recording module.

[0120] Step 430: If the plurality of first tools include the ReAct tool, then based on the first execution order, the first subtask is processed in sequence by each of the first tools to obtain the second subtask.

[0121] In some examples, after determining the multiple first tools corresponding to the first subtask, the multiple first tools may be called in sequence according to the first execution order to execute the first subtask, wherein the last tool among the multiple first tools may be a ReAct tool or may not be a ReAct tool.

[0122] Combine the following Fig. 6A The case where the last tool in the first execution order is the ReAct tool is described.

[0123] Fig. 6A A schematic diagram of another voice control method provided in an embodiment of the present application. Fig. 6A As shown, the above step 430 includes steps 610 to 620 as shown below.

[0124] Step 610: If the ReAct tool is the last tool among the multiple first tools, based on the first execution order, other first tools except the ReAct tool are called in sequence to execute the first subtask to obtain a first execution result.

[0125] In some examples, when the last tool among the multiple first tools is the ReAct tool, it indicates that the execution process of the target intent has not ended, and therefore, it is necessary to further determine the next intent to be executed (i.e., the next subtask). For example, the first tools other than the ReAct tool can be called in sequence according to the first execution order to process the first subtask and obtain the first execution result.

[0126] In some embodiments, the plurality of first tools may include a text-to-speech tool, a speech recognition tool, and a ReAct tool.

[0127] In some embodiments, step 610 includes: broadcasting the inquiry information to the user by calling a text-to-speech tool; receiving the reply voice input by the user by calling a voice recognition tool, and determining the reply text corresponding to the reply voice.

[0128] In some examples, when the multiple first tools include a TTS tool, an ASR tool, and a ReAct tool, and the first execution order is the TTS tool, the ASR tool, and the ReAct tool, the terminal device (such as the task execution module) can play the inquiry information to the user by calling the TTS tool, and after receiving the user's reply information to the inquiry information, determine the reply text corresponding to the reply information by calling the ASR tool. The first execution result includes the reply text.

[0129] For example, when the first subtask is "asking the user which air conditioner to turn on", after calling the TTS tool to execute the first subtask, the terminal device can broadcast to the user "which air conditioner to turn on". If the user answers "bedroom", the ASR tool is called for voice recognition to obtain the reply text of "bedroom", that is, the first execution result corresponding to the first subtask is "bedroom".

[0130] Step 620: Process the first execution result by using the ReAct tool to obtain a second subtask.

[0131] In some examples, after the other first tools complete the execution of the first subtask and obtain the first execution result, the ReAct tool is called to process the first execution result to obtain the second subtask. In other words, the ReAct tool understands the intent of the first execution result and determines the second subtask.

[0132] In some examples, the ReAct tool can obtain the second subtask based on the first execution result corresponding to the first subtask, or can obtain at least one third subtask based on the first execution result corresponding to the first subtask, and determine the second subtask after executing the third subtask. In other words, if the first subtask is the first subtask corresponding to the target intent, and the second subtask is the last subtask corresponding to the target intent, then at least one third subtask can be included between the first subtask and the second subtask.

[0133] In some embodiments, step 620 includes: processing the reply text information through the ReAct tool to obtain a third subtask; processing the third subtask based on the large language model to determine a third task queue corresponding to the third subtask; if at least one third tool includes the ReAct tool, then based on the third execution order, processing the third subtask in sequence through at least one third tool to obtain the second subtask.

[0134] In some examples, the ReAct tool understands the intent of the first execution result (i.e., the reply text message), determines the third subtask, and sends the third subtask to the large language model, or sends the third subtask and the third prompt information corresponding to the third subtask to the large language model. The large language model schedules the third subtask and determines the third task queue corresponding to the third subtask. The third task queue includes at least one third tool for executing the third subtask, and a third execution order corresponding to executing a third tool. The terminal device executes the third subtask in accordance with the third execution order by calling each third tool to obtain the second subtask.

[0135] For example, when the first execution result is "bedroom", the ReAct tool can determine the user's first sub-intention as "turn on the bedroom air conditioner" based on "bedroom". The ReAct tool performs intent analysis on the first sub-intention. Since the terminal device is currently located in the living room, the terminal device needs to move from the living room to the bedroom before turning on the bedroom air conditioner. Therefore, after the ReAct tool performs intent analysis, it can generate the third sub-task of "navigating to the bedroom".

[0136] In some examples, the ReAct tool sends the third subtask (such as "navigate to the bedroom") to the large language model, and the large language model performs task scheduling on the third subtask. Since the terminal device still does not complete the target intention of "turning on the air conditioner" after executing "navigate to the bedroom", the at least one third tool determined by the large language model still includes the ReAct tool. For example, the at least one third tool may include a navigation tool and a ReAct tool.

[0137] For example, the terminal device controls the terminal device to move from its current location to the bedroom by calling the navigation tool. After that, the terminal device calls the ReAct tool again to perform intent analysis based on the execution result of "the robot has arrived in the bedroom" through the ReAct tool. Since the robot has arrived in the bedroom, it can execute the task of turning on the bedroom air conditioner. Therefore, the ReAct tool can generate a second subtask of "turning on the air conditioner remotely".

[0138] In some examples, the ReAct tool may send the second subtask to the large language model, or send the second subtask and second prompt information corresponding to the second subtask to the large language model.

[0139] Figure 6B A schematic diagram of another voice control method provided in an embodiment of the present application is shown as follows: Figure 6B As shown, step 431 may be further included after step 420. Step 431 includes: if the multiple first tools do not include the ReAct tool, based on the first execution order, the first subtask is processed by each first tool to obtain a processing result of the target intention.

[0140] For example, when the large language model schedules the first subtask, no ReAct tool is scheduled, that is, the multiple first tools do not include the ReAct tool, which indicates that the target intent can be processed by calling each first tool in the first task queue. Therefore, based on the first execution order, the first subtask can be processed by calling each first tool to obtain the processing result of the target intent.

[0141] It should be noted that the embodiment of the present application is illustratively described by taking the case where the multiple first tools include the ReAct tool as an example. When the first tool includes the ReAct tool, the following step 440 is continued to be executed.

[0142] Step 440: Process the second subtask based on the large language model and determine a second task queue corresponding to the second subtask.

[0143] In some examples, after receiving the second subtask, or the second subtask and the second prompt information, the large language model performs task scheduling processing on the second subtask and determines a second task queue corresponding to the second subtask. The second task queue includes at least one second tool in the tool set and a second execution order corresponding to the at least one second tool.

[0144] It should be noted that step 440 is similar to step 420 in the above embodiment and will not be described again here to avoid repetition.

[0145] Step 450: If the at least one second tool does not include the ReAct tool, based on the second execution order, the second subtask is processed by the at least one second tool to obtain a processing result of the target intent.

[0146] In some examples, when the large language model determines that the user's target intention can be achieved after the second subtask is completed, the ReAct tool may not be arranged when the task is arranged, that is, the at least one second tool does not include the ReAct tool. After the terminal device sequentially calls at least one second tool to execute the second subtask in accordance with the second execution order, the user's target intention can be achieved, that is, the processing result of the target intention is obtained.

[0147] For example, when the second subtask is "turn on the air conditioner by remote control", when the large language model performs task scheduling, the user's target intention of "turning on the air conditioner" can be achieved by executing the remote control to turn on the air conditioner. Therefore, at least one second tool may not include the ReAct tool, such as at least one second tool includes the infrared tool in the tool set. The robot calls the infrared tool to turn on the bedroom air conditioner to achieve the user's target intention. The processing result of the target intention is that the air conditioner has been turned on.

[0148] For another example, at least one second tool may also include a TTS tool. After calling the infrared tool to turn on the air conditioner, the robot may also call the TTS tool to broadcast a voice message of "the bedroom air conditioner is turned on" to the user, so that the user can obtain the robot's intention processing results in real time.

[0149] In some examples, at least one tool may also include a ReAct tool. When at least one second tool includes a ReAct tool, the target intent is not executed. For example, after determining that the bedroom air conditioner is turned on by infrared, the ReAct tool may also determine with the user whether there are other needs, such as whether the air conditioner in other places needs to be turned on. In this case, at least one second tool may also include a TTS tool, an ASR tool, and a ReAct tool.

[0150] For example, the robot calls the TTS tool to play the question "Do you have any other needs?" to the user. If the user answers "No", the robot calls the ASR tool for voice recognition and obtains the execution result of "No other needs". The ReAct tool determines that the user has no other needs based on the ASR execution result, indicating that the task has been completed and waiting for the next task to be executed.

[0151] like Figure 6BAs shown, step 451 may be further included after step 440. Step 451 includes: if the at least one second tool includes a ReAct tool, based on the second execution order, the second subtask is processed by the at least one second tool to obtain the next subtask to be processed.

[0152] For example, when at least one second tool includes the ReAct tool, it indicates that the target intention cannot be achieved after executing the second subtask. Therefore, the ReAct tool is needed to further determine the next subtask to be executed and schedule the next subtask to be executed based on the large language model until the user's target intention is achieved.

[0153] The voice control method provided in the embodiment of the present application uses the ReAct tool to understand the user's intention, splits the voice command input by the user into multiple subtasks, and then uses the large language model to arrange the tasks of each subtask in turn to determine the tool to execute each subtask. The terminal device executes each subtask by calling the corresponding tool until the user's target intention is completed. Among them, when the tools determined by the large language model include the ReAct tool, the ReAct tool can determine the next subtask according to the current execution result, until the tools determined by the large language model do not include the ReAct tool, indicating that the terminal device has completed the user's target intention. In other words, the embodiment of the present application splits the complex voice commands input by the user through the mutual cooperation between the ReAct tool, the large language model and the tool set, and completes the user's target intention by processing each subtask in turn. The voice control method provided in the embodiment of the present application solves the problem that the terminal device cannot understand and is prone to making mistakes in complex environments or instructions, improves the intelligence level and practicality of the voice control interaction of the terminal device, and improves the user experience.

[0154] Figure 7 A schematic diagram of a speech processing process provided in an embodiment of the present application. Figure 7 The above steps 410 to 450 are further described. It should be noted that: Figure 7 The terminal device is taken as an example of a robot.

[0155] like Figure 7 As shown in the figure, the wake-up command can trigger the robot to execute the wake-up task (such as process ①). The robot performs task scheduling by default and sends the task scheduling result to the intention tree (such as process ②). The intention tree sends the task scheduling result to the task execution module Runner (such as process ③) to enable the Runner to perform the first round of task execution. Figure 7In process ④-⑧), the Runner calls the TTS tool, ASR tool and ReAct tool in sequence for processing, and sends the processing results to the large language model. After the large language model performs task scheduling processing, the first task scheduling result is obtained, and the first task scheduling result is sent to the intent tree.

[0156] The intention tree sends the first task scheduling result to the Runner (such as process ⑨) so that the Runner can perform the second round of task execution. ), Runner calls tool 1 and ReAct tool in turn for processing, and sends the processing result to the large language model. After the large language model performs task scheduling processing, a second task scheduling result is obtained, and the second task scheduling result is sent to the intent tree.

[0157] The intent tree sends the second task scheduling result to the Runner (such as process ) to enable the Runner to perform the third round of task execution. ), Runner completes the processing of the target intent by calling tool 2, and the speech processing process ends.

[0158] That is to say, in the above-mentioned voice processing process, when the user wakes up the robot, the robot will arrange the task by default, arrange the text-to-speech tool, the voice-to-text tool (i.e., the ASR tool in the above-mentioned embodiment), and the ReAct tool into the intent tree and execute them sequentially. When the ReAct tool is executed, the user's intention question will be submitted to the big model. The big model will arrange the task according to the intention question and the prompt word, and submit it to the intent tree again; if the task arranges tool 1 at this time, and the task is not ended, then the last tool in the intent tree is ReAct, the intention is not interrupted, and it continues to respond to the user's needs; after tool 1 is executed, it will be executed again to the ReAct tool. At this time, the execution result of tool 1 and the memory content will be submitted to the big model at the same time, and the task will be arranged again. If the task arranges tool 2, and the big model determines that it is over after execution, then there is no need to call the ReAct tool in the end, and the execution intention ends.

[0159] Figure 8 A schematic diagram of another speech processing process provided in an embodiment of the present application. Figure 8 A specific speech processing process is described below. It should be noted that: Figure 7 The terminal device is taken as an example of a robot.

[0160] like Figure 8 As shown, Figure 8(1) to (3) in the figure are the processing procedures corresponding to the wake-up intention. After receiving the wake-up instruction "Hi, xx" input by the user, the robot can determine the wake-up tool corresponding to the wake-up task according to the preset task arrangement. The wake-up tool includes a broadcast tool (i.e., the TTS tool in the above embodiment), an ASR tool, and a ReAct tool. Among them, the broadcast tool first broadcasts the voice message "Are you there" to the user, and then the ASR tool obtains the voice instruction "Turn on the air conditioner" input by the user, and converts the voice instruction into the text message of "Turn on the air conditioner". Finally, the ReAct tool performs intent analysis and generates the first subtask of "Which air conditioner to turn on", thereby entering the intention processing process of turning on the air conditioner.

[0161] like Figure 8 As shown in (4)-(6) in the figure, the large language model determines the first tool corresponding to the first subtask based on the first subtask of "which air conditioner to turn on", and the first tool includes a broadcast tool, an ASR tool, and a ReAct tool. Among them, the broadcast tool first broadcasts the voice information of "which" to the user, and then the ASR tool obtains the voice command "bedroom" input by the user and converts the voice command into text information of "bedroom". Finally, the ReAct tool performs intent analysis to generate the second subtask of "how to go to the bedroom".

[0162] like Figure 8 As shown in (7), the large language model determines the second tool corresponding to the second subtask based on the second subtask of "how to get to the bedroom", and the second tool includes a finder tool and a ReAct tool. First, the finder tool finds the location of the bedroom air conditioner for the robot, that is, controls the robot to move to the bedroom.

[0163] like Figure 8 As shown in (8)-(10), when the robot moves to the bedroom, if the robot receives the wake-up command "Hi, xx" input by the user, it will enter the wake-up processing process corresponding to the wake-up intention again. According to the preset task arrangement, the wake-up tools are determined to include the announcement tool, the ASR tool and the ReAct tool. Among them, the announcement tool first announces the voice message "Are you there" to the user, and then the ASR tool obtains the voice command "What are you doing" input by the user, and converts the voice command into the text message of "What are you doing", and finally the ReAct tool performs intent analysis and generates a chat task of "Answering the question", thereby entering the intent processing process of the chat task.

[0164] like Figure 8 As shown in (11), the large language model determines the tool corresponding to the chat task based on the chat task of "answering questions", and the tool includes a broadcasting tool, which plays "go turn on the air conditioner" to the user, thereby completing the chat task.

[0165] In some examples, when the robot (such as the robot calling any tool) executes any subtask corresponding to the target intent (such as the second subtask), the user can also wake up the robot to input other voice commands. In this case, the terminal device can pause the execution of the current task (such as turning on the air conditioner) and execute the task corresponding to the new intent (such as the chat task). After completing the execution of the task of the new intent, continue to execute the task corresponding to the target intent.

[0166] like Figure 8 As shown in (12) to (13) in the figure, after completing the chat task, the robot continues to perform the second subtask, that is, continue to call the finder tool to find the bedroom air conditioner. After finding the bedroom air conditioner, the ReAct tool is called to perform intent analysis and generate the third subtask of "turn on the air conditioner".

[0167] like Figure 8 As shown in (14), the large language model determines the third tool corresponding to the third subtask of "turning on the air conditioner" based on the third subtask. After the third subtask is executed, the target intention of "turning on the air conditioner" can be completed. Therefore, the fourth tool includes the infrared tool and does not include the ReAct tool. The infrared tool is called to turn on the bedroom air conditioner, thereby completing the execution process of the target intention of "turning on the air conditioner".

[0168] Through the above-mentioned speech processing process, it can be known that the embodiment of the present application can decompose and execute complex tasks step by step through the design pattern of the ReAct tool and the intent tree, avoiding the problem that the intent tree is huge and requires more judgments to promote the execution of the intent after the task is decomposed once. After the execution of tool 1, the embodiment of the present application uses ReAct to analyze or ask the user, and then performs task scheduling for tool 2 or tool 3. Therefore, the embodiment of the present application effectively reduces the complexity and accuracy of task scheduling by decomposing in steps, and can ensure friendly and sustainable interaction with users.

[0169] Fig. 9 A schematic diagram of a terminal device provided in an embodiment of the present application. Fig. 9 As shown, the terminal device 900 includes a communicator 910 and a controller 920 coupled to the communicator 910. In particular:

[0170] The communicator 910 is configured to receive a voice command input by a user.

[0171] Controller 920 is configured to: determine the target intent corresponding to the voice instruction and the first subtask corresponding to the target intent based on the reasoning and action ReAct tool; wherein the terminal device includes a tool set, and the tool set includes the ReAct tool; process the first subtask based on the large language model to determine the first task queue corresponding to the first subtask; wherein the first task queue includes multiple first tools in the tool set and the first execution order corresponding to the multiple first tools; if the multiple first tools include the ReAct tool, then based on the first execution order, the first subtask is processed in sequence by each first tool to obtain the second subtask; process the second subtask based on the large language model to determine the second task queue corresponding to the second subtask; wherein the second task queue includes at least one second tool in the tool set and the second execution order of at least one second tool; if the at least one second tool does not include the ReAct tool, then based on the second execution order, the second subtask is processed by at least one second tool to obtain the processing result of the target intent.

[0172] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0173] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0175] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A terminal device, characterized in that: include: The communicator is configured to: receive a voice command input by a user; A controller is coupled to the communicator and is configured to: Based on the Reasoning and Action ReAct tool, determining a target intent corresponding to the voice instruction and a first subtask corresponding to the target intent; wherein the terminal device includes a tool set, and the tool set includes the ReAct tool; Processing the first subtask based on the large language model to determine a first task queue corresponding to the first subtask; wherein the first task queue includes a plurality of first tools in the tool set and a first execution order corresponding to the plurality of first tools; If the plurality of first tools include the ReAct tool, based on the first execution order, the first subtask is processed in sequence by each first tool to obtain a second subtask; Processing the second subtask based on the large language model to determine a second task queue corresponding to the second subtask; wherein the second task queue includes at least one second tool in the tool set and a second execution order of the at least one second tool; If the at least one second tool does not include the ReAct tool, based on the second execution order, the second subtask is processed by the at least one second tool to obtain a processing result of the target intention.

2. The terminal device according to claim 1, characterized in that: The controller executes the first subtask based on the first execution order, and processes the first subtask in sequence through each first tool to obtain the second subtask, which is specifically configured as follows: If the ReAct tool is the last tool among the multiple first tools, based on the first execution order, the other first tools except the ReAct tool are sequentially called to execute the first subtask to obtain a first execution result; The first execution result is processed by the ReAct tool to obtain the second subtask.

3. The terminal device according to claim 2, characterized in that: The plurality of first tools include a text-to-speech tool, a speech recognition tool, and the ReAct tool; The controller executes the first execution order based on which the other first tools other than the ReAct tool are sequentially called to execute the first subtask to obtain a first execution result, which is specifically configured as follows: Broadcasting the inquiry information to the user by calling the text-to-sound tool; The reply voice input by the user is received by calling the voice recognition tool, and the reply text corresponding to the reply voice is determined; wherein the first execution result includes the reply text.

4. The terminal device according to claim 3, characterized in that: The controller executes the processing of the first execution result by the ReAct tool to obtain the second subtask, which is specifically configured as follows: Processing the reply text information by the ReAct tool to obtain a third subtask; Processing the third subtask based on the large language model to determine a third task queue corresponding to the third subtask; wherein the third task queue includes at least one third tool in the tool set and a third execution order corresponding to the at least one third tool; If the at least one third tool includes the ReAct tool, based on the third execution order, the third subtask is processed in sequence by the at least one third tool to obtain the second subtask.

5. The terminal device according to any one of claims 1 to 4, characterized in that: The controller is also configured to: Determine, by the ReAct tool, first prompt information corresponding to the first subtask, and send the first subtask and the first prompt information to the large language model; The controller performs the processing of the first subtask based on the large language model and determines the first task queue corresponding to the first subtask, and is specifically configured as follows: The first subtask and the first prompt information are processed based on the large language model, the first subtask is divided into a plurality of execution instructions, and a first tool corresponding to each execution instruction in the plurality of execution instructions and the first execution order corresponding to each of the first tools are determined.

6. The terminal device according to any one of claims 1 to 4, characterized in that: Before the communicator executes the voice instruction received from the user, the communicator is further configured to: Receive a wake-up command input by a user; The controller is further configured to: in response to the wake-up instruction, determine a wake-up task corresponding to the wake-up instruction and a preset tool corresponding to the wake-up task; wherein the preset tool includes a text-to-sound tool, a speech recognition tool and the ReAct tool; The response information corresponding to the wake-up instruction is broadcast to the user by calling the text-to-sound tool.

7. The terminal device according to claim 6, characterized in that: The controller is also configured to: Determine the text information corresponding to the voice instruction by calling the voice recognition tool; The controller performs the ReAct tool to determine the target intent corresponding to the voice instruction and the first subtask corresponding to the target intent, and is specifically configured as follows: The text information is processed by the ReAct tool to determine the target intention and the first subtask.

8. The terminal device according to claim 7, characterized in that: The ReAct tool includes an interaction module, and the controller performs the processing of the first subtask based on the large language model and determines the first task queue corresponding to the first subtask, which is specifically configured as follows: Sending the first subtask to the large language model through the interaction module, so as to perform task scheduling processing on the first subtask through the large language model to obtain the first task queue; The first task queue is obtained through the interaction module.

9. The terminal device according to claim 8, characterized in that: The ReAct tool further includes a memory module, and the controller is further configured to: The first subtask and the first task queue corresponding to the first subtask are stored through the memory module.

10. A voice control method, characterized in that: Applied to a terminal device, the method comprises: Acquire a voice command input by a user, and determine a target intent corresponding to the voice command and a first subtask corresponding to the target intent based on a reasoning and action ReAct tool; wherein the terminal device includes a tool set, and the tool set includes the ReAct tool; Processing the first subtask based on the large language model to determine a first task queue corresponding to the first subtask; wherein the first task queue includes a plurality of first tools in the tool set and a first execution order corresponding to the plurality of first tools; If the plurality of first tools include the ReAct tool, based on the first execution order, the first subtask is processed in sequence by each first tool to obtain a second subtask; Processing the second subtask based on the large language model to determine a second task queue corresponding to the second subtask; wherein the second task queue includes at least one second tool in the tool set and a second execution order corresponding to the at least one second tool; If the at least one second tool does not include the ReAct tool, based on the second execution order, the second subtask is processed by the at least one second tool to obtain a processing result of the target intention.

Images