Server, display device and task processing method

Abstract

The invention provides a server, a display device and a task processing method. When receiving a task instruction sent by a display device, a server can obtain preposed semantic information based on the task instruction, and search corresponding memory information and tool information according to the preposed semantic information and the task instruction. And the server fuses the searched memory information and tool information corresponding to the task instruction and the preposed semantic information respectively to obtain to-be-input information for being input into the task processing model. And determining a target task and a target task tool corresponding to the task instruction based on the task processing model so as to complete the target task based on the target task tool and generate task completion information. The server can return the target data packet containing the task completion information to the display device, and the display device interacts with the user.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a server, a display device and a task processing method. Background Art

[0002] During the human-computer interaction process, the display device can use the intelligent agent deployed in the server to parse the task instructions issued by the user based on the large model to identify the user's true intentions, and plan and execute the target tasks according to the true intentions.

[0003] The agent can understand the task instructions through the large model deployed in the server to determine the target task, and then call the tools related to the target task to perform the target task. However, the recognition logic of the large model is affected by the model training method and model structure, so it is difficult to adapt to a variety of user scenarios, resulting in a decrease in the task processing ability of the agent.

[0004] In order to improve the task processing capability of the intelligent agent, a new data set can be used to train the large model. However, when the user usage scenarios are rich, the amount of data required to train the model is more. Therefore, the training cost will increase, and from a time perspective, retraining the model will be delayed. Improvements based on the operating framework of the intelligent agent are only applicable to the current usage scenarios and are not universal. Summary of the invention

[0005] Some embodiments of the present application provide a server, a display device and a task processing method to solve the problem of poor task processing capability of an intelligent agent.

[0006] In a first aspect, the present application provides a server, including a communicator and a processor. The communicator is configured to establish a communication connection with a display device. The processor is configured to:

[0007] In response to the received task instruction sent by the display device, the task instruction is parsed to obtain the pre-semantic information corresponding to the task instruction;

[0008] Generate extended information according to the task instruction and the pre-semantic information; the extended information includes memory information and tool information; the memory information includes historical sessions associated with the task instruction and the pre-semantic information; the tool information includes a system tool call record of the display device when the historical session is generated;

[0009] fusing the task instruction, the pre-semantic information and the extended information to obtain information to be input;

[0010] Inputting the information to be input into a task processing model to control the task processing model to determine a target task and a target task tool based on the information to be input;

[0011] The target data packet is transmitted back to the display device, so that the display device interacts with the user based on the target data packet.

[0012] In this way, the server can generate extended information based on the pre-semantic information and the task instruction, and further generate the information to be input based on the extended information, the pre-semantic information and the task instruction. Among them, the pre-semantic information and the extended information can provide additional prompt information for the task processing model to improve the task processing model's ability to understand the user's true intention, thereby improving the task processing model's task processing ability.

[0013] In some feasible embodiments, the processor executes the task instruction sent by the display device in response to the received task instruction, parses the task instruction, and obtains the pre-semantic information corresponding to the task instruction, which is specifically configured as follows:

[0014] Based on the semantic understanding structure, the task instruction is parsed to obtain the preceding semantic information; the semantic understanding structure of the preceding semantic information includes at least one of domain information, intent information, and slot information; the domain information is used to determine the conversation domain associated with the task instruction; the intent information is used to determine the task goal corresponding to the task instruction; and the slot information is used to represent replaceable words in the task instruction.

[0015] In this way, by decomposing the pre-semantic information based on the semantic understanding structure, the key words in the pre-semantic information can be fully extracted, and the memory information and tool information with a higher degree of association can be searched in the database according to the key words. This is conducive to improving the efficiency of the information to be input into the task processing model, thereby improving the task processing capability of the task processing model.

[0016] In some feasible embodiments, the processor is specifically configured to generate the extended information according to the task instruction and the pre-semantic information:

[0017] Based on the task instruction, searching for first memory information corresponding to the task instruction;

[0018] Based on the pre-semantic information, searching for second memory information corresponding to the pre-semantic information;

[0019] According to the semantic understanding structure of the preceding semantic information, the first memory information and the second memory information are fused to obtain memory information.

[0020] In this way, by searching the memory information separately through task instructions and prior semantic information, the user's true intention can be analyzed more fully in combination with the user's historical behavior records, thereby improving the task processing model's ability to perceive the user's true intention.

[0021] In some possible embodiments, the processor is further configured to:

[0022] Based on the task instruction, searching for first tool information corresponding to the task instruction;

[0023] Based on the preceding semantic information, searching for second tool information corresponding to the preceding semantic information;

[0024] The first tool information and the second tool information are combined to obtain tool information.

[0025] In this way, by searching for tool information through task instructions and pre-semantic information, the selectable tools can be fully called to process the target task. In addition, there is also an association between the target tool and the target task, which can also enable the task processing model to perceive the user's true intention based on the association between the target tool and the target task, thereby improving the task processing model's ability to perceive the user's true intention.

[0026] In some feasible embodiments, the processor executes inputting the information to be input into the task processing model to control the task processing model to determine the target task and the target task tool based on the information to be input, and is further configured to:

[0027] Acquire a task processing record of the task processing model; the task processing record includes a task processing result and a task iteration number of the task processing model;

[0028] If the task processing result indicates that the task processing model successfully executes the task instruction, generating a target data packet according to the target task and the target task tool determined by the task processing model;

[0029] If the task iteration number is greater than the iteration threshold, a failure prompt message is generated; the failure prompt message is used to prompt that the task execution corresponding to the current task instruction has failed.

[0030] In this way, it is possible to determine whether the task processing model has completed the target task by acquiring the task processing record of the task processing model. When the target task is completed, a target data packet is generated to be fed back to the display device. When the task fails to be completed, a failure prompt message is generated to prompt that the task has failed to be completed.

[0031] In some feasible embodiments, the processor executes inputting the information to be input into the task processing model to control the task processing model to determine the target task and the target task tool based on the information to be input, and is further configured to:

[0032] Controlling the task processing model to call the target task tool based on the target task; the target task tool is used to obtain associated information of executable target tasks; wherein, when the task processing model completes the target task based on the target task tool, the task processing model generates task completion information;

[0033] The task completion information is encapsulated to obtain a target data packet.

[0034] In this way, the task completion information generated by the task processing model can be encapsulated to obtain a target data packet, which is then fed back to the display device, and the display device interacts with the user based on the target data packet.

[0035] In some feasible embodiments, the target task includes a first sub-target task and a second sub-target task; the processor executes inputting the information to be input into the task processing model to control the task processing model to determine the target task and the target task tool based on the information to be input, and is further configured to:

[0036] Control the task processing model to call the first target task tool based on the first sub-target task, so that the task processing model executes the first sub-target task based on the first target task tool and obtains the first sub-completion information;

[0037] Control the task processing model to call the second target task tool based on the second sub-target task, so that the task processing model executes the second sub-target task based on the second target task tool and obtains the second sub-completion information;

[0038] The task processing model is controlled to fuse the first sub-completion information and the second sub-completion information based on the pre-semantic information included in the information to be input, so as to obtain task completion information.

[0039] In this way, the task processing model can be controlled to perform multiple rounds of thinking and decision-making based on the information to be input, and generate sub-completion information based on the sub-target task, and then obtain the task completion information by fusing the sub-completion information, and then encapsulate the task completion information into a target data packet.

[0040] In some feasible embodiments, the processor executes the fusion of the task instruction, the pre-semantic information and the extended information to obtain the information to be input, and is further configured to:

[0041] Acquire task prompt information; the task prompt information is used to prompt the task execution target and task execution process required by the task processing model;

[0042] The task instruction, the pre-semantic information, the expansion information and the task prompt information are integrated to obtain the information to be input.

[0043] In this way, by adding task prompt information to the information to be input, the task processing model can be prompted with the process and goal of executing the task based on the task prompt information, thereby improving the execution efficiency and accuracy of the task processing model, which is conducive to improving the task processing capability of the task processing model.

[0044] In a second aspect, the present application provides a display device, including a display, a communication device and a controller. The communication device is configured to establish a communication connection with a server. The controller is configured to:

[0045] In response to the received task instruction, the task instruction is sent to a server so that the server acquires pre-semantic information based on the task instruction; and the server acquires extended information based on the pre-semantic information and the task instruction; the task instruction, the pre-semantic information and the extended information are used to generate information to be input; the information to be input is used to input a task processing model; wherein, when the information to be input is input into the task processing model, the task processing model determines a target task and a target completion tool corresponding to the task instruction based on the information to be input;

[0046] Receive the target data packet sent back by the server to interact with the user based on the target task information and target completion tool information in the target data packet.

[0047] In this way, the display device can parse and process the task instructions with the help of the task processing model deployed in the server, and interact with the user by receiving the target data packet fed back by the server. By adding a step of obtaining pre-semantic information in the server, the task processing model's ability to perceive the user's true intention is enhanced, which can improve the task processing capability of the task processing model and further enhance the interaction capability of the display device with the user.

[0048] In a third aspect, the present application provides a task processing method, applied to the server described in the first aspect, the method comprising:

[0049] In response to the received task instruction sent by the display device, the task instruction is parsed to obtain the pre-semantic information corresponding to the task instruction;

[0050] Generate extended information according to the task instruction and the pre-semantic information; the extended information includes memory information and tool information; the memory information includes historical sessions associated with the task instruction and the pre-semantic information; the tool information includes a system tool call record of the display device when the historical session is generated;

[0051] fusing the task instruction, the pre-semantic information and the extended information to obtain information to be input;

[0052] Inputting the information to be input into a task processing model to control the task processing model to determine a target task and a target task tool based on the information to be input;

[0053] The target data packet is transmitted back to the display device, so that the display device interacts with the user based on the target data packet.

[0054] It can be seen from the above technical content that the present application provides a server, a display device and a task processing method. When the server receives the task instruction sent by the display device, it can obtain the pre-semantic information based on the task instruction, and search for the corresponding memory information and tool information according to the pre-semantic information and the task instruction respectively. The server fuses the searched memory information and tool information corresponding to the task instruction and the pre-semantic information respectively, and can obtain the information to be input into the task processing model. Then, based on the task processing model, the target task and the target task tool corresponding to the task instruction are determined to complete the target task based on the target task tool and generate task completion information. The server can transmit the target data packet containing the task completion information back to the display device, and the display device interacts with the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 A schematic diagram of a display device operation scenario provided in some embodiments of the present application;

[0057] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0058] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0059] Figure 4 A timing interaction diagram of a display device and a server when executing a user task instruction provided in some embodiments of the present application;

[0060] Figure 5 A flow chart of server analysis task instructions provided for some embodiments of the present application;

[0061] Figure 6 A schematic diagram of memory information search provided in some embodiments of the present application;

[0062] Figure 7A schematic diagram of tool search provided for some embodiments of the present application;

[0063] Figure 8 A schematic diagram of an intelligent agent completing a sub-goal task provided in some embodiments of the present application;

[0064] Fig. 9 A deployment architecture diagram of each module in a server provided in some embodiments of the present application;

[0065] Fig.10 A timing interaction diagram of various modules when a server parses task instructions provided in some embodiments of the present application. DETAILED DESCRIPTION

[0066] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0067] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0068] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0069] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0070] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0071] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0072] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0073] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0074] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0075] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0076] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0077] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0078] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0079] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0080] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0081] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0082] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0083] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0084] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0085] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0086] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0087] In some embodiments, the user input interface 280 may be used to receive instructions from a user input. The user input interface 280 may include at least one of a microphone, a touch panel, a sensor, a remote controller, etc. Then, the display device 200 may receive instructions input by the user based on the user input interface 280 to perform interactive functions with the user.

[0088] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0089] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0090] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0091] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0092] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0093] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0094] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0095] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0096] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0097] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0098] like Figure 4 As shown, when using the display device 200, the user can input task instructions in a variety of input methods such as pictures, voice, and text to drive the display device to perform the target task corresponding to the task instruction. When the display device 200 receives the task instruction, it can forward the task instruction to the server, and the server calls the intelligent agent to parse the task instruction, thereby assisting the display device to complete the target task corresponding to the task instruction.

[0099] It should be noted that the intelligent agent is different from the general natural language processing model. The intelligent agent can also continuously learn and adapt to the environment through interaction with the environment, and then determine the user's true intention in the process of semantic analysis based on the perception of the environment. In addition, the intelligent agent can directly call the tool to perform the target task based on the conclusions obtained from the semantic analysis, without receiving external instructions in the decision-making process.

[0100] However, in the process of semantic analysis, the intelligent agent usually needs to call a large model related to natural language processing to analyze the semantics, and then analyze the user's true intentions in combination with environmental factors. However, in this way, the language analysis logic of the large model itself has a great impact on the semantic analysis results. Therefore, in some specific scenarios, it is difficult to analyze the semantics in combination with the user's scenario habits, resulting in a decrease in semantic analysis capabilities.

[0101] To solve this problem, we can use a large amount of interaction data generated in specific scenarios to train and fine-tune intelligent agents and large models. However, the data cost is too high, and the process of training intelligent agents and large models is lagging, making it difficult to improve them in time and apply them to multiple scenarios.

[0102] In addition, the framework of the agent can be adjusted by introducing mechanisms such as the ReAct mechanism and the SOP standard process. Among them, the ReAct mechanism is an agent reasoning framework that can improve the dynamics and adaptability of the agent. Its core is to provide observation results to the language model so that the model can update the context window to re-evaluate the received information, thereby improving the model's reasoning ability.

[0103] The SOP standard process refers to an established process that can be provided to the agent for performing tasks. The established process contains pre-defined steps and rules for performing specific tasks.

[0104] However, these methods involve changes to the framework, which makes it difficult to balance the universality and professionalism of the intelligent agent. It is still difficult to quickly adapt to the usage needs in various scenarios.

[0105] In order to solve the above problems, Figure 5 As shown, the present application provides a server to improve the ability of an agent to recognize the true intention of a user. The server includes a communicator and a processor, wherein the server can establish a communication connection with a display device through the communicator to receive a task instruction sent by the display device. And the processor calls the agent to parse the task instruction, such as Figure 5 As shown, the processor is configured as:

[0106] S100: In response to a received task instruction sent by a display device, the task instruction is parsed to obtain pre-semantic information corresponding to the task instruction.

[0107] like Figure 4 As shown, in some embodiments, the user can input a task instruction to the display device 200 through the user input interface 280, such as "Which city is cooler recently, City A or City B?" The display device 200 can send the task instruction to the server through the communication device 220, and the server will parse it. In this way, the real intention of the user can be fully understood with the computing power of the server, thereby improving the interactive ability of the display device 200 and the user.

[0108] In some embodiments, a pre-semantic understanding module is deployed in the server, and when receiving a task instruction sent by the display device 200, the task instruction is first parsed by the pre-semantic understanding module to obtain pre-semantic information corresponding to the task instruction. In this way, the server can analyze the user's true intention based on the task instruction and the pre-semantic information, which is conducive to improving the perception of the user's true intention.

[0109] In some embodiments, when performing pre-semantic understanding on the task instruction, the pre-semantic understanding obtained by analysis can also be decomposed to facilitate determining the key semantics in the pre-semantic understanding. That is, the processor executes the task instruction sent by the display device in response to the received task instruction, parses the task instruction, and obtains the pre-semantic information corresponding to the task instruction, which is specifically configured as follows:

[0110] Based on the semantic understanding structure, the task instruction is parsed to obtain the preceding semantic information; the semantic understanding structure of the preceding semantic information includes at least one of domain information, intent information, and slot information; the domain information is used to determine the conversation domain associated with the task instruction; the intent information is used to determine the task goal corresponding to the task instruction; and the slot information is used to represent replaceable words in the task instruction.

[0111] In some embodiments, the process of pre-semantic understanding and decomposition is completed by the pre-semantic understanding module. Among them, the semantic understanding structure may include domain information, intent information and slot information. Taking the task instruction "Which city is cooler recently, City A or City B" as an example, the pre-semantic understanding module can determine that the field associated with the task instruction is weather during the parsing process, the intent of the task instruction is weather query, and the slot information contained in the task instruction is "City A-city", "City B-city", "cool-weather". Among them, the slot information can be used to characterize the replaceable words in the task instruction. When searching for associated memory information or tool information based on the pre-semantic information and the task instruction, memory information or tool information with different replaceable words can also be searched. Setting the slot information is conducive to expanding the search scope and increasing the search depth.

[0112] It should be noted that through pre-semantic understanding, more detailed pre-semantic information can be obtained on the basis of task instructions, which is conducive to obtaining richer and more accurate memory information or tool information through task instructions and pre-semantic information search.

[0113] S200: Generate extended information according to the task instruction and the preceding semantic information.

[0114] In some embodiments, the extended information includes memory information and tool information. The memory information includes historical conversations associated with the task instructions and the pre-semantic information. The memory information can be pre-stored in a storage space deployed in a server, and the memory information is used to characterize the interaction record between the user and the display device 200.

[0115] It is understandable that when the display device 200 parses the task instruction, it usually sends it to the server, which parses the user's task instruction and generates processing information corresponding to the task instruction and then transmits it back to the display device. In this way, the server can store the processing process of the task instruction as an interaction record to generate a user database through the user's daily use process, so as to analyze the user's behavior, which is conducive to reducing the cost of data collection.

[0116] In some embodiments, the pre-semantic understanding module can generate extended information based on the task instruction and the pre-semantic information to increase the prediction of user needs and behaviors based on the extended information, thereby increasing the understanding of the user's true intentions and improving the subsequent task processing capabilities. The extended information can be information that has a degree of association with the semantics represented by the task instruction and the pre-semantic information. For example, if the task instruction contains weather, the extended information can include weather temperature, which is conducive to increasing the search range and search depth of memory information, and can filter out the memory information with the highest degree of association with the task instruction and the pre-semantic information from the memory information database.

[0117] In some embodiments, the tool information may include system tools that the processor can currently call, such as a weather query tool and a time query tool. In this way, the processor can obtain tool information according to the task instruction and the pre-semantic information, and then call the system tool to execute the target task corresponding to the task instruction and the pre-semantic information.

[0118] It is understandable that the tool information obtained by filtering based on the task instructions and pre-semantics is not necessarily used to execute the target task. Instead, it can be used as part of the information input by the user into the task processing model, and the system tools called by the processor can also reflect the type and content of the target task. This can improve the task processing model's ability to understand the task instructions, thereby improving the task processing model's processing capabilities.

[0119] In other embodiments, the information to be input may also include task prompt information, which may be used to prompt the task execution target and task execution process required by the task processing model. For example, "You are a semantically enhanced intelligent agent, and you can refer to the semantic parsing results to understand the user's input and complete the task. In the process of completing the task, please select the appropriate tool and refer to the user's conversation history to predict and understand the user's possible needs."

[0120] It can be understood that by planning the task execution goals and task execution processes, the intelligent agent can be guided to better understand the information to be input, and execute the target task according to the information to be input, so as to improve the task processing ability of the intelligent agent.

[0121] S300: The task instruction, the pre-semantic information and the extended information are integrated to obtain information to be input.

[0122] It is understandable that the processor needs to input the information to be input into the task processing model so that the task processing model can analyze the input information to determine the target task and the target task tool, and further complete the target task according to the target task tool.

[0123] In some embodiments, the processor can obtain the information to be input by fusing the task instruction, the pre-semantic information and the extended information. In this way, both the pre-semantic information and the extended information can be input into the task processing model as prompt information of the task instruction, so that when the task processing model analyzes the task instruction, it can increase the perception of the user's intention in combination with the prompt information, which is conducive to improving the task processing capability of the task processing model.

[0124] In some embodiments, the task processing model may be an intelligent agent, which has the ability to perceive the environment, make decisions, and self-iterate, and the intelligent agent can analyze input information and complete the target task according to the analysis results.

[0125] S400: Inputting the information to be input into a task processing model to control the task processing model to determine a target task and a target task tool based on the information to be input.

[0126] In some embodiments, the processor may input the input information into the task processing model, and then control the task processing model to analyze the input information to be input, so as to determine the target task and target task tool corresponding to the task instruction based on the input information based on the task processing model. The task processing model may be an agent, and the subsequent embodiments are all described as an agent.

[0127] After determining the target task and the target task tools, the agent can make multiple rounds of decisions to complete the target task. In the decision-making process, the target task can be divided into multiple sub-target tasks, and then the target task can be completed by completing the sub-target tasks one by one.

[0128] It is understandable that the agent will refer to different memory information and tool information in the decision-making process, and then form decision iterations to determine the task completion method that best matches the user's true intention through decision iterations, and then generate task completion information. This can improve the agent's task processing capabilities.

[0129] S500: Transmitting a target data packet back to the display device, so that the display device interacts with the user based on the target data packet.

[0130] In some embodiments, the target data packet includes task completion information, so that the server can transmit the target data packet back to the display device 200, so that the display device 200 completes the interaction with the user according to the target data packet.

[0131] In some embodiments, a memory information database is deployed in the server, and the processor stores the task processing process of the agent in the memory information database to dynamically supplement the memory information database, so that the memory information is continuously enriched, which is conducive to improving the agent's perception of various scenarios. Then, when the processor searches for memory information according to task instructions and pre-semantic information, it can obtain memory information from the memory information database. The memory information database can include long-term dialogue memory information, short-term multi-round dialogue information, and time summary memory information, which can provide rich data for the search process.

[0132] like Figure 6 As shown, the processor generates the extended information according to the task instruction and the pre-semantic information, and is specifically configured as follows:

[0133] Based on the task instruction, first memory information corresponding to the task instruction is searched.

[0134] Based on the preceding semantic information, second memory information corresponding to the preceding semantic information is searched.

[0135] According to the semantic understanding structure of the preceding semantic information, the first memory information and the second memory information are fused to obtain memory information.

[0136] In some embodiments, the processor can search for memory information from the memory information database based on the task instruction and the pre-semantic information, respectively. For example, the processor searches for the first memory information from the memory information database based on the task instruction, and searches for the second memory information from the memory information database based on the pre-semantic information.

[0137] It is understandable that by searching the first memory information and the second memory information, the search scope and search depth of the memory information are expanded, which is conducive to providing the task processing model with richer user usage scenarios. The first memory information and the second memory information do not need to be exactly the same, but only need to be associated with the task instruction or the previous semantic information. In this way, when the first memory information and the second memory information are input into the intelligent agent, they can be used alone as key information for the intelligent agent to understand the user's true intentions, and can also be verified with each other as prompt information for the task processing model to understand the user's true intentions.

[0138] In some embodiments, after searching and obtaining the first memory information and the second memory information, the processor may fuse the first memory information and the second memory information to obtain the memory information, and then generate the information to be input based on the memory information.

[0139] In this way, the information to be input includes rich memory information. When the intelligent agent analyzes the user's true intention based on the information to be input, it can combine the user usage scenarios provided by the memory information database to fully perceive the user's true intention, which is conducive to improving the intelligent agent's ability to perceive the user's true intention. Improve the task processing ability of the intelligent agent.

[0140] In some embodiments, a system tool set is further deployed in the server, and the processor can further search for associated tool information from the system tool set according to the task instruction and the pre-semantic information, that is, the processor is further configured to:

[0141] Based on the task instruction, first tool information corresponding to the task instruction is searched.

[0142] Based on the preceding semantic information, second tool information corresponding to the preceding semantic information is searched.

[0143] The first tool information and the second tool information are combined to obtain tool information.

[0144] like Figure 7 As shown, the processor can search for the first tool information from the system tool set according to the task instruction, and search for the second tool information from the system tool set according to the pre-semantic information.

[0145] In some embodiments, the system tools represented by the first tool information and the second tool information may be different, and only need to be associated with the task instruction or the pre-semantic information. Different system tools can be used to perform different target tasks. Therefore, by obtaining the tool information associated with the task instruction or the pre-semantic information, the target task corresponding to the task instruction or the pre-semantic information can also be mapped based on the tool information. This is conducive to the intelligent agent to perceive the user's true intention based on the tool information.

[0146] In some embodiments, after obtaining the first tool information and the second tool information, the processor may fuse the first tool information and the second tool information to obtain the tool information. The information to be input may be generated based on the tool information, so that the information to be input includes rich tool information. The tool information and the memory information may be used as key information for the intelligent agent to perceive the user's true intention, and may also be used to confirm each other as prompt information for the intelligent agent to perceive the user's true intention, which is conducive to improving the intelligent agent's ability to perceive the user's true intention.

[0147] It can be understood that memory information and tool information, as components of the information to be input, can both be used as prompt information for task instructions so that the intelligent agent can fully perceive the user's true intention.

[0148] In some embodiments, the processor may record the task completion status of the agent to provide feedback to the display device according to the task completion status. That is, the processor executes inputting the information to be input into the agent to control the agent to determine the target task and the target task tool based on the information to be input, and is further configured to:

[0149] Acquire a task processing record of the task processing model; the task processing record includes a task processing result and a task iteration number of the task processing model.

[0150] If the task processing result indicates that the task processing model successfully executes the task instruction, a target data packet is generated according to the target task and the target task tool determined by the task processing model.

[0151] If the task iteration number is greater than the iteration threshold, a failure prompt message is generated; the failure prompt message is used to prompt that the task execution corresponding to the current task instruction has failed.

[0152] It is understandable that the agent can generate a task processing record during the execution of the task. The task processing record may include the task processing result and the number of task iterations. In this way, the processor can determine the task completion status of the agent by obtaining the task processing record.

[0153] In some embodiments, the agent determines the target task and the target task tool based on the information to be input, and after completing the target task based on the target task tool, it can stop iteration and use the final thinking result as the task completion information. For example, when the task instruction is "Which is cooler, City A or City B", the task completion information finally output by the agent can be "Comparing the temperature information of City A and City B, the temperature of City B will be more comfortable and make people feel cooler". At this time, the processor can generate a target data packet based on the task completion information, and then transmit the target data packet back to the display device 200, so that the display device 200 interacts with the user.

[0154] In other embodiments, when the agent fails to complete a task, a failure prompt message may be generated. The processor may also generate a target data packet based on the failure prompt message and transmit it back to the display device 200, so that the display device 200 interacts with the user and prompts the user to change the task instruction. This can avoid the user waiting time and causing a decrease in the user experience.

[0155] In addition, in order to prevent the agent from continuously iterating and causing the task processing time to be delayed, the iteration number of the agent can be preset, and the iteration number can be monitored during the iteration of the agent. When the task iteration number of the agent is greater than the iteration threshold, the agent is controlled to generate a failure prompt message, and the display device 200 interacts with the user by generating a target data packet and transmitting it back to the display device 200, prompting the user to change the task instruction. In this way, the interaction efficiency between the display device 200 and the user can be guaranteed to alleviate the problem of the user waiting time being too long.

[0156] In some embodiments, when the agent performs the target task, it can call the associated target task tool according to the target task to perform the target task, that is, the processor executes inputting the information to be input into the task processing model to control the task processing model to determine the target task and the target task tool based on the information to be input, and is further configured to:

[0157] The task processing model is controlled to call the target task tool based on the target task.

[0158] The task completion information is encapsulated to obtain a target data packet.

[0159] It is understandable that the target task and the target task tool have an association relationship. For example, when the target task is to query the weather, the agent will call the weather query tool; when the target task is to query the time, the agent will call the time query tool. Then, when the agent determines the target task according to the information to be input, it can call the corresponding system tool as the target task tool according to the target task to complete the target task through the target task tool.

[0160] In some embodiments, after completing the target task, the agent may generate task completion information, and the processor may obtain the task completion information, encapsulate the task completion information according to the communication protocol between the display device 200 and the server to obtain a target data packet, and then transmit the target data packet back to the display device 200.

[0161] It is understandable that the data format of the target data packet can also be set in combination with the interaction mode between the display device 200 and the user. For example, if the display device 200 interacts with the user through voice, the target data packet can be encapsulated as audio format data. This can reduce the workload of the display device 200 and improve the interaction efficiency between the display device 200 and the user with the help of the high computing power of the server.

[0162] like Figure 8As shown in the figure, in the process of executing the target task, the agent can extract multiple sub-target tasks from the input information, that is, the target task is composed of multiple sub-target tasks. And these sub-target tasks also have certain correlations. The agent can generate the final task completion information based on the completion status of each sub-target task. The process includes:

[0163] The task processing model is controlled to call the first target task tool based on the first sub-target task, so that the task processing model executes the first sub-target task based on the first target task tool and obtains the first sub-completion information.

[0164] The task processing model is controlled to call a second target task tool based on the second sub-target task, so that the task processing model executes the second sub-target task based on the second target task tool and obtains second sub-completion information.

[0165] The task processing model is controlled to fuse the first sub-completion information and the second sub-completion information based on the pre-semantic information included in the information to be input, so as to obtain task completion information.

[0166] In some embodiments, the agent can decompose multiple sub-goal tasks according to the information to be input. For example, if the task instruction is "Which is cooler today, City A or City B", the agent can determine that the sub-goal tasks are "What is today's date", "The temperature of District A in City A today", and "The temperature of District B in City B today". Then the agent can call the system tools one by one to process the sub-goal tasks. That is, call the date query tool to query the date corresponding to today, and then call the weather query tool to obtain the temperature of District A in City A and the temperature of District B in City B respectively, and generate sub-completion information respectively.

[0167] It is understandable that when the intelligent agent generates the sub-completion information corresponding to each sub-target task, it can fuse the sub-completion information to obtain the final task completion information. For example, after executing the above-mentioned sub-target task, the intelligent agent can generate the task completion information "Comparing the temperature information of District a in City A and District b in City B, the temperature in District b in City B will be more comfortable and make people feel cool. If you have any other needs, please feel free to tell me." In this way, the processor can obtain the task completion information finally generated by the intelligent agent and encapsulate the target data packet to transmit the target data packet back to the display device 200, and the display device 200 interacts with the user. Among them, when encapsulating the target data packet, additional custom interaction information can be added, such as "If you have any other needs, please feel free to tell me", so as to enhance the affinity of the interaction process between the display device 200 and the user through custom interaction information, which is conducive to improving the user experience.

[0168] like Fig. 9 and Fig.10As shown, when the server provided by the embodiment of the present application receives the task instruction sent by the display device 200, the pre-semantic understanding module can parse the task instruction to obtain the pre-semantic information, and based on the pre-semantic information, the associated memory information can be obtained from the memory information database, and the associated tool information can be obtained from the tool set. Then, the information to be input can be jointly generated with a variety of information, so that the information to be input includes a rich user usage scenario, thereby improving the intelligent agent's perception of the user's true intentions, so as to improve the intelligent agent's task processing ability. Among them, the executor can be a processor in the server, or it can generally refer to the controller 250 in the display device 200. In addition, the task processing process of the intelligent agent will be recorded in the memory information database, so as to dynamically update the memory information database, enrich the user usage scenario, and thus form a data closed loop, which can not only alleviate the pressure caused by data demand, but also improve the intelligent agent's ability to adapt to the environment and perceive the user's true intentions.

[0169] Some embodiments of the present application also provide a display device 200, including a display 260, a communication device 220 and a controller 250. The communication device 220 is configured to establish a communication connection with a server. The controller 250 is configured to:

[0170] In response to a received task instruction, the task instruction is sent to a server so that the server obtains pre-semantic information based on the task instruction; and the server obtains extended information based on the pre-semantic information and the task instruction; the task instruction, the pre-semantic information and the extended information are used to generate information to be input; the information to be input is used to input a task processing model; wherein, when the information to be input is input into the task processing model, the task processing model determines the target task and target completion tool corresponding to the task instruction based on the information to be input.

[0171] Receive the target data packet sent back by the server to interact with the user based on the target task information and target completion tool information in the target data packet.

[0172] like Figure 4As shown, when the display device receives a task instruction input by the user through the user input interface 280, the task instruction can be forwarded to the server, and the server calls the task processing model deployed in the server for processing. In addition, in the process of parsing the task instruction, the server can search for memory information and tool information through the task instruction and the pre-semantic understanding corresponding to the task instruction, and input the information obtained from the search into the task processing model only after all the information obtained is integrated. In this way, additional reference information for solving the problem can be provided to the intelligent agent based on the information associated with the pre-semantic understanding, so that the intelligent agent can more fully feel the user's true intention, which is conducive to improving the task processing ability of the intelligent agent. In turn, the interactive ability of the display device and the user can be improved, and the display device's processing ability for the task instructions issued by the user can be enhanced.

[0173] Some embodiments of the present application also provide a task processing method, which can be applied to a server, and the method includes:

[0174] In response to the received task instruction sent by the display device, the task instruction is parsed to obtain the pre-semantic information corresponding to the task instruction;

[0175] Generate extended information according to the task instruction and the pre-semantic information; the extended information includes memory information and tool information; the memory information includes historical sessions associated with the task instruction and the pre-semantic information; the tool information includes a system tool call record of the display device when the historical session is generated;

[0176] fusing the task instruction, the pre-semantic information and the extended information to obtain information to be input;

[0177] Inputting the information to be input into a task processing model to control the task processing model to determine a target task and a target task tool based on the information to be input;

[0178] The target data packet is transmitted back to the display device, so that the display device interacts with the user based on the target data packet.

[0179] It can be seen from the above technical content that the present application provides a server, a display device and a task processing method. When the server receives the task instruction sent by the display device, it can obtain the pre-semantic information based on the task instruction, and search for the corresponding memory information and tool information according to the pre-semantic information and the task instruction respectively. The server fuses the searched memory information and tool information corresponding to the task instruction and the pre-semantic information respectively, and can obtain the information to be input into the task processing model. Then, based on the task processing model, the target task and the target task tool corresponding to the task instruction are determined to complete the target task based on the target task tool and generate task completion information. The server can transmit the target data packet containing the task completion information back to the display device, and the display device interacts with the user.

[0180] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A server, characterized in that: include: a communicator configured to establish a communication connection with a display device; The processor is configured as: In response to the received task instruction sent by the display device, the task instruction is parsed to obtain the pre-semantic information corresponding to the task instruction; Generate extended information according to the task instruction and the pre-semantic information; the extended information includes memory information and tool information; the memory information includes historical sessions associated with the task instruction and the pre-semantic information; the tool information includes a system tool call record of the display device when the historical session is generated; fusing the task instruction, the pre-semantic information and the extended information to obtain information to be input; Inputting the information to be input into a task processing model to control the task processing model to determine a target task and a target task tool based on the information to be input; The target data packet is transmitted back to the display device, so that the display device interacts with the user based on the target data packet.

2. The server according to claim 1, characterized in that: The processor executes the task instruction sent by the display device in response to the received task instruction, parses the task instruction, and obtains the pre-semantic information corresponding to the task instruction, and is specifically configured as follows: Based on the semantic understanding structure, the task instruction is parsed to obtain the pre-semantic information; the semantic understanding structure of the pre-semantic information includes at least one of domain information, intent information, and slot information; The domain information is used to determine the conversation domain associated with the task instruction; the intention information is used to determine the task target corresponding to the task instruction; The slot information is used to represent replaceable words in the task instruction.

3. The server according to claim 2, characterized in that: The processor generates the extended information according to the task instruction and the pre-semantic information, and is specifically configured to: Based on the task instruction, searching for first memory information corresponding to the task instruction; Based on the pre-semantic information, searching for second memory information corresponding to the pre-semantic information; According to the semantic understanding structure of the preceding semantic information, the first memory information and the second memory information are fused to obtain memory information.

4. The server according to claim 2, characterized in that: The processor is further configured to: Based on the task instruction, searching for first tool information corresponding to the task instruction; Based on the preceding semantic information, searching for second tool information corresponding to the preceding semantic information; The first tool information and the second tool information are combined to obtain tool information.

5. The server according to claim 1, characterized in that: The processor executes inputting the information to be input into the task processing model to control the task processing model to determine the target task and the target task tool based on the information to be input, and is further configured to: Acquire a task processing record of the task processing model; the task processing record includes a task processing result and a task iteration number of the task processing model; If the task processing result indicates that the task processing model successfully executes the task instruction, generating a target data packet according to the target task and the target task tool determined by the task processing model; If the task iteration number is greater than the iteration threshold, a failure prompt message is generated; The failure prompt information is used to prompt that the task execution corresponding to the current task instruction has failed.

6. The server according to claim 5, characterized in that: The processor executes inputting the information to be input into the task processing model to control the task processing model to determine the target task and the target task tool based on the information to be input, and is further configured to: Controlling the task processing model to call the target task tool based on the target task; the target task tool is used to obtain associated information of executable target tasks; wherein, when the task processing model completes the target task based on the target task tool, the task processing model generates task completion information; The task completion information is encapsulated to obtain a target data packet.

7. The server according to claim 6, characterized in that: The target task includes a first sub-target task and a second sub-target task; the processor executes inputting the information to be input into the task processing model to control the task processing model to determine the target task and the target task tool based on the information to be input, and is further configured to: Control the task processing model to call the first target task tool based on the first sub-target task, so that the task processing model executes the first sub-target task based on the first target task tool and obtains the first sub-completion information; Control the task processing model to call the second target task tool based on the second sub-target task, so that the task processing model executes the second sub-target task based on the second target task tool and obtains the second sub-completion information; The task processing model is controlled to fuse the first sub-completion information and the second sub-completion information based on the pre-semantic information included in the information to be input, so as to obtain task completion information.

8. The server according to claim 1, characterized in that: The processor executes and fuses the task instruction, the pre-semantic information and the extended information to obtain information to be input, and is further configured to: Get task prompt information; The task prompt information is used to prompt the task execution target and task execution process required by the task processing model; The task instruction, the pre-semantic information, the expansion information and the task prompt information are integrated to obtain the information to be input.

9. A display device, characterized in that: include: monitor; A communication device configured to establish a communication connection with a server; The controller is configured as: In response to the received task instruction, sending the task instruction to a server so that the server acquires pre-semantic information based on the task instruction; and enabling the server to acquire extended information based on the pre-semantic information and the task instruction; The task instruction, the pre-semantic information and the extended information are used to generate information to be input; The information to be input is used to input into a task processing model; wherein, when the information to be input is input into the task processing model, the task processing model determines the target task and the target completion tool corresponding to the task instruction based on the information to be input; Receive the target data packet sent back by the server to interact with the user based on the target task information and target completion tool information in the target data packet.

10. A task processing method, characterized in that: The server according to any one of claims 1 to 8, characterized in that it comprises: In response to the received task instruction sent by the display device, the task instruction is parsed to obtain the pre-semantic information corresponding to the task instruction; Generate extended information according to the task instruction and the pre-semantic information; the extended information includes memory information and tool information; the memory information includes historical sessions associated with the task instruction and the pre-semantic information; the tool information includes a system tool call record of the display device when the historical session is generated; fusing the task instruction, the pre-semantic information and the extended information to obtain information to be input; Inputting the information to be input into a task processing model to control the task processing model to determine a target task and a target task tool based on the information to be input; The target data packet is transmitted back to the display device, so that the display device interacts with the user based on the target data packet.

Images