Server, display device and task processing method

Abstract

The application provides a server, a display device and a task processing method. When receiving a task instruction sent by the display device, the server can obtain preposition semantic information based on the task instruction, and search corresponding memory information and tool information according to the preposition semantic information and the task instruction respectively. The server fuses the memory information and the tool information searched respectively corresponding to the task instruction and the preposition semantic information, and can obtain to-be-input information used for inputting into a task processing model. Then, the target task and the target task tool corresponding to the task instruction are determined based on the task processing model, so as to complete the target task based on the target task tool and generate task completion information. The server can return a target data packet containing the task completion information to the display device, and the display device performs interaction with the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a server, a display device and a task processing method. BACKGROUND

[0002] In the process of human-computer interaction, the display device can analyze the task instruction issued by the user based on the large model deployed in the server to identify the real intention of the user, and plan and execute the target task according to the real intention.

[0003] The intelligent agent can determine the target task by understanding the task instruction through the large model deployed in the server, and then call the tool related to the target task to execute the target task. However, the identification logic of the large model is affected by the training method and the model structure of the model, so it is difficult to adapt to various user scenarios, resulting in a decline in the task processing capability of the intelligent agent.

[0004] In order to improve the task processing capability of the intelligent agent, a new data set can be used to train the large model, but in the case of rich user scenarios, the amount of data required for training the model is larger. Therefore, it will increase the training cost, and from the time point of view, retraining the model has a lag. And improving the running framework based on the intelligent agent is only suitable for the current use scenario, and does not have universality. SUMMARY

[0005] Some embodiments of the present application provide a server, a display device and a task processing method to solve the problem of poor task processing capability of the intelligent agent.

[0006] In a first aspect, the present application provides a server, comprising a communicator and a processor. The communicator is configured to establish a communication connection with a display device. The processor is configured to:

[0007] In response to the received task instruction sent by the display device, parse the task instruction to obtain the preposition semantic information corresponding to the task instruction;

[0008] Generate augmented information according to the task instruction and the preposition semantic information; the augmented information includes memory information and tool information; the memory information includes historical conversations associated with the task instruction and the preposition semantic information; the tool information includes system tool call records of the display device when generating the historical conversations;

[0009] Fuse the task instruction, the preposition semantic information and the augmented information to obtain input information;

[0010] Input the input information into a task processing model to control the task processing model to determine a target task and a target task tool based on the input information.

[0011] The target data packet is fed back to the display device, so that the display device performs interaction with the user based on the target data packet.

[0012] In this way, the server can generate the extended information based on the pre-semantic information and the task instruction, and further generate the to-be-input information according to the extended information, the pre-semantic information and the task instruction. The pre-semantic information and the extended information can both provide additional prompt information for the task processing model, so as to improve the understanding ability of the task processing model to the real intention of the user, and further improve the task processing ability of the task processing model.

[0013] In some possible embodiments, the processor is configured to, in response to the received task instruction sent by the display device, parse the task instruction to obtain pre-semantic information corresponding to the task instruction, and is specifically configured to:

[0014] parse the task instruction based on the semantic understanding structure to obtain the pre-semantic information; the semantic understanding structure of the pre-semantic information at least includes one of domain information, intention information and slot information; the domain information is used to determine a conversation domain associated with the task instruction; the intention information is used to determine a task target corresponding to the task instruction; and the slot information is used to represent replaceable words in the task instruction.

[0015] In this way, the pre-semantic information is decomposed based on the semantic understanding structure, so that the key words in the pre-semantic information can be fully extracted, and the memory information and the tool information with higher association degree in the database can be searched according to the key words. This is beneficial to improve the efficiency of the to-be-input information input to the task processing model, and further improve the task processing ability of the task processing model.

[0016] In some possible embodiments, the processor is configured to generate extended information according to the task instruction and the pre-semantic information, and is specifically configured to:

[0017] search first memory information corresponding to the task instruction based on the task instruction;

[0018] search second memory information corresponding to the pre-semantic information based on the pre-semantic information;

[0019] fuse the first memory information and the second memory information to obtain memory information according to the semantic understanding structure of the pre-semantic information.

[0020] In this way, the memory information is searched respectively through the task instruction and the pre-semantic information, so that the real intention of the user can be analyzed more fully by combining the historical behavior record of the user, and the perception ability of the task processing model to the real intention of the user is improved.

[0021] In some possible embodiments, the processor is further configured to:

[0022] search first tool information corresponding to the task instruction based on the task instruction;

[0023] search second tool information corresponding to the preposition semantic information based on the preposition semantic information;

[0024] merge the first tool information and the second tool information to obtain tool information.

[0025] In this way, the tool information is searched based on the task instruction and the preposition semantic information respectively, so that the selectable tool can be fully called to process the target task. Moreover, the target tool and the target task have an association relationship, so that the task processing model can also perceive the real intention of the user in combination with the association relationship between the target tool and the target task, and the perception ability of the task processing model for the real intention of the user is improved.

[0026] In some possible embodiments, the processor performs inputting the to-be-input information to the task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information, and is further configured to:

[0027] obtain a task processing record of the task processing model; the task processing record includes a task processing result and a task iteration number of the task processing model;

[0028] if the task processing result represents that the task processing model successfully executes the task instruction, generating a target data packet according to the target task and the target task tool determined by the task processing model;

[0029] if the task iteration number is greater than an iteration threshold, generating a failure prompt information; the failure prompt information is used to prompt that the task corresponding to the current task instruction fails.

[0030] In this way, whether the target task is completed by the task processing model can be determined by obtaining the task processing record of the task processing model. Moreover, the target data packet is generated when the target task is completed to be fed back to the display device. When the target task fails, the failure prompt information is generated to prompt that the target task fails.

[0031] In some possible embodiments, the processor performs inputting the to-be-input information to the task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information, and is further configured to:

[0032] The task processing model is controlled to call the target task tool based on the target task; the target task tool is used to obtain associated information of an executable target task; and when the task processing model completes the target task based on the target task tool, the task processing model generates task completion information.

[0033] The task completion information is encapsulated to obtain a target data packet.

[0034] In this way, the target data packet can be obtained by encapsulating the task completion information generated by the task processing model, and then the target data packet can be fed back to the display device, and the display device can perform interaction with the user according to the target data packet.

[0035] In some possible embodiments, the target task includes a first sub-target task and a second sub-target task; the processor is configured to input the to-be-input information into the task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information, and is further configured to:

[0036] The task processing model is controlled to call a first target task tool based on the first sub-target task, so that the task processing model performs the first sub-target task based on the first target task tool and obtains first sub-completion information;

[0037] The task processing model is controlled to call a second target task tool based on the second sub-target task, so that the task processing model performs the second sub-target task based on the second target task tool and obtains second sub-completion information;

[0038] The task processing model is controlled to fuse the first sub-completion information and the second sub-completion information based on preposition semantic information contained in the to-be-input information to obtain task completion information.

[0039] In this way, the task processing model can be controlled to perform multi-round thinking and decision making based on the to-be-input information, and sub-completion information can be generated based on sub-target tasks, and then the task completion information can be obtained by fusing the sub-completion information, and the task completion information can be encapsulated into a target data packet.

[0040] In some possible embodiments, the processor is configured to fuse the task instruction, the preposition semantic information, and the extended information to obtain the to-be-input information, and is further configured to:

[0041] Obtain task prompt information; the task prompt information is used to prompt a task execution target and a task execution process required by the task processing model;

[0042] The task instruction, the preposition semantic information, the extended information, and the task prompt information are fused to obtain the to-be-input information.

[0043] In this way, the task prompt information can be added in the to-be-input information, so as to prompt the task processing model to perform the flow and target of the task based on the task prompt information. In this way, the execution efficiency and accuracy of the task processing model are improved, and the task processing capability of the task processing model is improved.

[0044] In a second aspect, the present application provides a display device, comprising a display, a communication device and a controller. The communication device is configured to establish a communication connection with a server. The controller is configured to:

[0045] In response to the received task instruction, the task instruction is sent to the server to enable the server to obtain pre-semantic information based on the task instruction; and the server obtains expansion information based on the pre-semantic information and the task instruction; the task instruction, the pre-semantic information and the expansion information are used to generate to-be-input information; the to-be-input information is used to input a task processing model; wherein, when the to-be-input information is input into the task processing model, the task processing model determines a target task and a target completion tool corresponding to the task instruction based on the to-be-input information;

[0046] The target data packet returned by the server is received to interact with the user based on the target task information and the target completion tool information in the target data packet.

[0047] In this way, the display device can analyze and process the task instruction by means of the task processing model deployed in the server, and interact with the user by receiving the target data packet fed back by the server. By adding the step of obtaining pre-semantic information in the server, the perception ability of the task processing model to the real intention of the user is enhanced, the task processing capability of the task processing model is improved, and the interaction ability of the display device with the user is enhanced.

[0048] In a third aspect, the present application provides a task processing method applied to the server of the first aspect, and the method comprises:

[0049] In response to the received task instruction sent by the display device, the task instruction is analyzed to obtain pre-semantic information corresponding to the task instruction;

[0050] Expansion information is generated according to the task instruction and the pre-semantic information; the expansion information comprises memory information and tool information; the memory information comprises a historical conversation associated with the task instruction and the pre-semantic information; and the tool information comprises a system tool call record of the display device when the historical conversation is generated;

[0051] The task instruction, the pre-semantic information and the expansion information are fused to obtain to-be-input information;

[0052] inputting the to-be-input information into the task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information;

[0053] feeding back a target data packet to the display device to enable the display device to perform interaction with a user based on the target data packet.

[0054] From the above technical content, the present application provides a server, a display device and a task processing method. When the server receives a task instruction sent by the display device, the server can obtain preposition semantic information based on the task instruction, and search corresponding memory information and tool information according to the preposition semantic information and the task instruction respectively. The server fuses the memory information and the tool information searched respectively corresponding to the task instruction and the preposition semantic information, and can obtain to-be-input information for inputting into a task processing model. Then, the target task and the target task tool corresponding to the task instruction are determined based on the task processing model, so as to complete the target task based on the target task tool and generate task completion information. The server can feed back a target data packet containing the task completion information to the display device, and the display device performs interaction with the user. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0056] Figure 1 a schematic diagram of a display device operation scene provided by some embodiments of the present application;

[0057] Figure 2 a hardware configuration schematic diagram of a display device provided by some embodiments of the present application;

[0058] Figure 3 a software configuration schematic diagram of a display device provided by some embodiments of the present application;

[0059] Figure 4 a timing interaction diagram of a display device and a server when executing a user task instruction provided by some embodiments of the present application;

[0060] Figure 5 a flowchart of a server analyzing a task instruction provided by some embodiments of the present application;

[0061] Figure 6 a memory information search schematic diagram provided by some embodiments of the present application;

[0062] Figure 7A tool search diagram provided for some embodiments of the present application;

[0063] Figure 8 A diagram of an agent completing a sub-goal task provided for some embodiments of the present application;

[0064] Figure 9 A deployment architecture diagram of modules in a server provided for some embodiments of the present application;

[0065] Figure 10 A timing interaction diagram of modules when a server parses a task instruction provided for some embodiments of the present application. DETAILED DESCRIPTION

[0066] The embodiments will be described in detail with reference to the drawings, wherein like reference numerals refer to like elements throughout. The following detailed description is not intended to restrict the embodiments to particular embodiments described herein, but rather the description is intended to provide a description of some embodiments falling within the scope of the claims.

[0067] The embodiments will be described in detail with reference to the drawings, wherein like reference numerals refer to like elements throughout. The following detailed description is not intended to restrict the embodiments to particular embodiments described herein, but rather the description is intended to provide a description of some embodiments falling within the scope of the claims.

[0068] It should be noted that the brief description of terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0069] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0070] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive of inclusion, for example, a product or device including a series of components does not necessarily limit to all components clearly listed, but can include other components not clearly listed or inherent to such products or devices.

[0071] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software codes that can perform the function related to the element.

[0072] In the embodiments of the present application, the display device 200 generally refers to a device having the ability of picture display and data processing. For example, the display device 200 includes, but is not limited to, a smart television, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0073] Figure 1 The schematic diagram of the operation scenario between the display device and the control device is provided for some embodiments of the present application. As shown in Figure 1 The user can operate the display device 200 through the touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a touch pen, a handle, etc.

[0074] The mobile terminal 300 can be used as a kind of control device for performing the human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a kind of communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install a software application with the display device 200, realize the connection communication through a network communication protocol, and achieve the purpose of one-to-one control operation and data communication. The mobile terminal 300 can also transmit the audio and video content displayed on the mobile terminal 300 to the display device 200, and realize the synchronous display function.

[0075] As shown in Figure 1 It is also shown in that the display device 200 also communicates data with the server 400 through various communication modes. The display device 200 can be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0076] The display device 200 can provide a broadcast receiving television function, and can also additionally provide a smart network television function supporting a computer function, including but not limited to a network television, a smart television, an Internet protocol television (IPTV), etc.

[0077] Figure 2 The schematic diagram of the operation scenario between the display device and the control device is provided for some embodiments of the present application. As shown in Figure 1 The hardware configuration block diagram of the display device 200 is shown in

[0078] In some embodiments, the display device 200 can include at least one of a tuning demodulator 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, a user input interface.

[0079] In some embodiments, the detector 230 is configured to collect signals of the external environment or the external interaction. For example, the detector 230 includes a light receiver configured to collect ambient light intensity; or the detector 230 includes an image collector, such as a camera, configured to collect an external environment scene, a user attribute, or a user interaction gesture; or the detector 230 includes a sound collector, such as a microphone, configured to receive external sound.

[0080] In some embodiments, the display 260 includes a display functional component configured to present a picture, and a driving component configured to drive the image display. The display 260 is configured to receive an image signal output from the controller 250 for display. For example, the display 260 can be configured to display video content, image content, and components of a menu control interface, and a user control UI interface.

[0081] In some embodiments, the communication device 220 is a component configured to communicate with the external device or the server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 according to different supported communication manners. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0082] The communication device 220 can be configured to connect the display device 200 to the external device or the server 400 in a wireless or wired manner. The wired connection can be achieved by connecting the display device 200 to the external device through a data line, an interface, or the like. The wireless connection can be achieved by connecting the display device 200 to the external device through a wireless signal or a wireless network. The display device 200 can be directly connected to the external device, or can be indirectly connected to the external device through a gateway, a router, a connection device, or the like.

[0083] In some embodiments, the controller 250 can include at least one of a central processor, a video processor, an audio processor, a graphics processor, a power supply processor, a first interface to an n-th interface for input / output, and the controller 250 controls the operation of the display device and responds to the user's operation by controlling various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0084] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, i.e., the tuner demodulator 210 can also be located in an external device of the main body device where the controller 250 is located, such as an external set-top box or the like.

[0085] In some embodiments, the user can input user commands through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the graphical user interface (GUI).

[0086] In some embodiments, the audio output device 270 can be a native loudspeaker of the display device 200, or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 can further be provided with an external audio output terminal, and the audio output device can be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0087] In some embodiments, the user input interface 280 can be used to receive instructions from user input. The user input interface 280 can include at least one of a microphone, a touchpad, a sensor, a remote control, etc. In turn, the display device 200 can execute interactive functions with the user based on the instructions received by the user input interface 280 from the user input.

[0088] In order to perform user interaction, in some embodiments, the display device 200 can run an operating system. The operating system is a computer program used to manage and control hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0089] It should be noted that the operating system can be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0090] The operating system can be divided into different modules or levels according to the functions implemented, for example, as shown in FIG. 2, in some embodiments, the system is divided into four layers, from top to bottom, the application layer (referred to as "application layer" for short), the application framework layer (referred to as "framework layer" for short), the system library layer and the kernel layer. Figure 3

[0091] ​In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0092] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0093] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0094] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0095] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions implemented by the framework layer.

[0096] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, memory management, etc. For example, as shown in Figure 3 The kernel layer can be configured with hardware drivers. The drivers contained in the kernel layer can be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power supply driver, etc.

[0097] It should be noted that the above examples are only a simple division of the functions of the operating system, and do not constitute a limitation on the specific operating system form of the display device 200 in the embodiments of the present application. According to the function of the display device, the type of operating system, and other factors, the number and specific type of layers contained in the operating system can be in other forms.

[0098] As shown in Figure 4 During use of the display device 200, the user can input task instructions in various input modes such as pictures, voice, and text to drive the display device to perform target tasks corresponding to the task instructions. When the display device 200 receives the task instructions, it can forward the task instructions to the server, and the server can call the agent to analyze the task instructions, thereby assisting the display device to complete the target tasks corresponding to the task instructions.

[0099] It should be noted that the agent is different from the general natural language processing large model. The agent can also learn and adapt to the environment through interaction with the environment, and then determine the real intention of the user in the process of performing semantic analysis by combining the perception ability of the environment. Moreover, the agent can directly call tools to perform target tasks according to the conclusions obtained by semantic analysis, and does not need to accept external instructions in the decision-making process.

[0100] However, the agent usually needs to call a natural language processing related large model to analyze the semantics in the process of semantic analysis, and then analyze the real intention of the user in combination with environmental factors. However, in this way, the language analysis logic of the large model itself has a great influence on the semantic analysis result, so it is difficult to analyze the semantics in combination with the user's scene habits in some specific scenarios, resulting in a decline in semantic analysis capability.

[0101] To solve such problems, the agent and the large model can be trained and fine-tuned by using a large amount of interaction data generated in specific scenarios. However, the data cost is too high, and the process of training the agent and the large model is lagging, which is difficult to improve and applicable to multiple scenarios in a timely manner.

[0102] In addition, the framework of the agent can be adjusted by introducing the ReAct mechanism and the SOP standard process. The ReAct mechanism is an agent reasoning framework that can improve the dynamics and adaptability of the agent. The core is to provide the observation results to the language large model, so that the model can update the context window to re-evaluate the received information, thereby improving the reasoning ability of the model.

[0103] The SOP standard process refers to a predetermined process for the agent to perform a task, which includes predefined steps and rules for performing a specific task.

[0104] However, these methods involve changes to the framework, and it is difficult to balance the universality and professionalism of the agent. It is still difficult to quickly adapt to the use requirements in multiple scenarios.

[0105] To solve the above problems, as shown in Figure 5 The server is provided to improve the recognition ability of the agent to the real intention of the user. The server includes a communicator and a processor, wherein the server can establish a communication connection with the display device through the communicator to receive the task instruction sent by the display device. And through the processor, the agent is called to analyze the task instruction, as shown in Figure 5 The processor is configured to:

[0106] S100: In response to the received task instruction sent by the display device, the task instruction is analyzed to obtain the pre-semantic information corresponding to the task instruction.

[0107] As shown in Figure 4 In some embodiments, the user can input a task instruction to the display device 200 through a user input interface 280, such as "where is it cooler recently in A city and B city". The display device 200 can send the task instruction to the server through the communication device 220 for analysis by the server. In this way, the real intention of the user can be fully understood with the help of the computing power of the server, thereby improving the interaction ability of the display device 200 with the user.

[0108] In some embodiments, a pre-semantic understanding module is deployed in the server, and when the task instruction sent by the display device 200 is received, the task instruction is first analyzed by the pre-semantic understanding module to obtain the pre-semantic information corresponding to the task instruction. In this way, the server can analyze the real intention of the user based on the task instruction and the pre-semantic information, which is conducive to improving the perception ability of the real intention of the user.

[0109] In some embodiments, when the task instruction is subjected to the pre-semantic understanding, the obtained pre-semantic understanding can also be decomposed to facilitate the determination of the key semantics in the pre-semantic understanding. That is, the processor is configured to, in response to receiving the task instruction sent by the display device, parse the task instruction to obtain pre-semantic information corresponding to the task instruction, and specifically configured to:

[0110] based on the semantic understanding structure, parse the task instruction to obtain the pre-semantic information; the semantic understanding structure of the pre-semantic information at least includes one of domain information, intent information, and slot information; the domain information is used to determine the conversation domain associated with the task instruction; the intent information is used to determine the task target corresponding to the task instruction; and the slot information is used to represent the replaceable word in the task instruction.

[0111] In some embodiments, the process of pre-semantic understanding and decomposition is completed by a pre-semantic understanding module. The semantic understanding structure can include domain information, intent information, and slot information. Taking the task instruction "Where is it cooler recently in A city and B city" as an example, the pre-semantic understanding module can determine in the parsing process that the domain associated with the task instruction is weather, the intent of the task instruction is weather query, and the slot information contained in the task instruction is "A city - city", "B city - city", and "cool - weather". The slot information can be used to represent the replaceable word in the task instruction, and different memory information or tool information can also be searched according to the pre-semantic information and the task instruction. By setting the slot information, it is beneficial to expand the search range and increase the search depth.

[0112] It should be noted that through the pre-semantic understanding, more detailed pre-semantic information can be obtained based on the task instruction, which is beneficial to search for more abundant and accurate memory information or tool information through the task instruction and the pre-semantic information.

[0113] S200: generating expansion information according to the task instruction and the pre-semantic information.

[0114] In some embodiments, the expansion information includes memory information and tool information. The memory information includes historical conversations associated with the task instruction and the pre-semantic information. The memory information can be pre-stored in a storage space deployed in a server, and the memory information is used to represent the interaction record of the user and the display device 200.

[0115] It can be understood that the display device 200 usually sends the task instruction to the server for analysis by the server when parsing the task instruction, and generates processing information corresponding to the task instruction and returns it to the display device. In this way, the server can store the processing process of the task instruction as an interaction record to generate a user database through the user's daily use process, so as to analyze the user's behavior and reduce the cost of data collection.

[0116] In some embodiments, the pre-semantic understanding module can generate extended information according to the task instruction and the pre-semantic information, so as to increase the prediction of user demand and behavior based on the extended information, thereby increasing the understanding of the user's true intention and improving the subsequent task processing capability. The extended information can be information that has a correlation degree with the semantics represented by the task instruction and the pre-semantic information. For example, if the task instruction contains weather, the extended information can include the temperature, which is beneficial to increase the search range and search depth of the memory information, and can filter out the memory information with the highest correlation degree from the memory information database.

[0117] In some embodiments, the tool information can include system tools that the processor can currently call, such as weather query tools and time query tools. In this way, the processor can obtain tool information according to the task instruction and the pre-semantic information, and then call the system tools to execute the target task corresponding to the task instruction and the pre-semantic information.

[0118] It can be understood that the tool information selected according to the task instruction and the pre-semantic information is not necessarily used to execute the target task. It can also be used as part of the information input to the task processing model, and the system tools called by the processor can also reflect the type and content of the target task. This can improve the understanding ability of the task processing model for the task instruction, and thereby improve the processing capability of the task processing model.

[0119] In other embodiments, the to-be-input information can also include task prompt information, which can be used to prompt the task execution target and the task execution process required by the task processing model. For example, "You are a semantic enhancement intelligent agent, you can refer to the semantic analysis result to understand the user's input and complete the task. In the process of completing the task, select the appropriate tool and refer to the user's conversation history to predict and understand the user's possible demand."

[0120] It can be understood that by planning the task execution target and the task execution process, the intelligent agent can better understand the to-be-input information and execute the target task according to the to-be-input information, thereby improving the task processing capability of the intelligent agent.

[0121] S300: fuse the task instruction, the pre-semantic information, and the extended information to obtain input information.

[0122] It can be understood that the processor needs to input the input information to the task processing model to analyze the input information by the task processing model, and then determine the target task and the target task tool, and further complete the target task according to the target task tool.

[0123] In some embodiments, the processor can obtain the input information by fusing the task instruction, the pre-semantic information, and the extended information. In this way, the pre-semantic information and the extended information can be input to the task processing model as prompt information of the task instruction, so that the task processing model can increase the perception ability of the user's intention when analyzing the task instruction, which is beneficial to improve the task processing ability of the task processing model.

[0124] In some embodiments, the task processing model can be an intelligent agent. The intelligent agent has the ability of perceiving the environment, providing decision, and self-iteration, so that the intelligent agent can analyze the input information and complete the target task according to the analysis result.

[0125] S400: input the input information to the task processing model to control the task processing model to determine the target task and the target task tool based on the input information.

[0126] In some embodiments, the processor can input the input information to the task processing model, and then control the task processing model to analyze the input information, so as to determine the target task and the target task tool corresponding to the task instruction based on the input information by the task processing model. The task processing model can be an intelligent agent, which is described in the subsequent embodiments.

[0127] After the intelligent agent determines the target task and the target task tool, it can make multiple rounds of decisions to complete the target task. In the process of decision-making, the target task can be divided into multiple sub-target tasks, and then the target task can be completed by completing the sub-target tasks one by one.

[0128] It can be understood that the intelligent agent will refer to different memory information and tool information in the process of decision-making, and then form a decision iteration to determine the task completion mode that best fits the user's real intention, and then generate task completion information. In this way, the task processing ability of the intelligent agent can be improved.

[0129] S500: return the target data packet to the display device to make the display device perform interaction with the user based on the target data packet.

[0130] In some embodiments, the target data packet includes task completion information, so that the server can return the target data packet to the display device 200, so that the display device 200 can complete the interaction with the user according to the target data packet.

[0131] In some embodiments, the server is deployed with a memory information database, and the processor stores the task processing process of the intelligent agent in the memory information database to dynamically supplement the memory information database, so that the memory information is constantly enriched, which is beneficial to improve the perception ability of the intelligent agent to various scenes. Then, the processor can obtain the memory information from the memory information database when searching the memory information according to the task instruction and the pre-semantic information. The memory information database can include long-term dialogue memory information, short-term multi-round dialogue information, and time summary memory information, which can provide rich data for the search process.

[0132] As shown in Figure 6 the processor is specifically configured to:

[0133] search first memory information corresponding to the task instruction based on the task instruction.

[0134] search second memory information corresponding to the pre-semantic information based on the pre-semantic information.

[0135] fuse the first memory information and the second memory information according to a semantic understanding structure of the pre-semantic information to obtain memory information.

[0136] In some embodiments, the processor can search the memory information from the memory information database based on the task instruction and the pre-semantic information, respectively. For example, the processor searches the first memory information from the memory information database based on the task instruction, and searches the second memory information from the memory information database based on the pre-semantic information.

[0137] It can be understood that by searching the first memory information and the second memory information, the search range and the search depth of the memory information are expanded, which is beneficial to provide more rich user usage scenarios to the task processing model. The first memory information and the second memory information do not need to be completely the same, but only need to have a correlation degree with the task instruction or the pre-semantic information. In this way, the first memory information and the second memory information can be used as key information for the intelligent agent to understand the real intention of the user when input to the intelligent agent, and can also be mutually verified as prompt information for the task processing model to understand the real intention of the user.

[0138] In some embodiments, after searching the first memory information and the second memory information, the processor can fuse the first memory information and the second memory information to obtain the memory information, and then generate the to-be-input information based on the memory information.

[0139] In this way, the to-be-input information includes rich memory information, and when the agent analyzes the real intention of the user according to the to-be-input information, the agent can combine the use scenarios of the user provided by the memory information database to fully perceive the real intention of the user, which is beneficial to improving the perception ability of the agent to the real intention of the user and improving the task processing ability of the agent.

[0140] In some embodiments, a set of system tools is also deployed in the server, and then the processor can also search associated tool information from the set of system tools according to the task instruction and the preposition semantic information, that is, the processor is also configured to:

[0141] search first tool information corresponding to the task instruction based on the task instruction.

[0142] search second tool information corresponding to the preposition semantic information based on the preposition semantic information.

[0143] merge the first tool information and the second tool information to obtain tool information.

[0144] As shown in Figure 7 the processor can search first tool information from the set of system tools according to the task instruction, and search second tool information from the set of system tools according to the preposition semantic information.

[0145] In some embodiments, the system tools represented by the first tool information and the second tool information can be different, and only have an association with the task instruction or the preposition semantic information. Different system tools can be used to perform different target tasks, so by obtaining tool information associated with the task instruction or the preposition semantic information, the target task corresponding to the task instruction or the preposition semantic information can also be mapped based on the tool information. In this way, it is beneficial for the agent to perceive the real intention of the user based on the tool information.

[0146] In some embodiments, after obtaining the first tool information and the second tool information, the processor can fuse the first tool information and the second tool information to obtain tool information. And the to-be-input information can be generated based on the tool information, so that the to-be-input information includes rich tool information. The tool information and the memory information can be used as key information for the agent to perceive the real intention of the user, or can be used as prompt information for the agent to perceive the real intention of the user, which is beneficial to improving the perception ability of the agent to the real intention of the user.

[0147] It can be understood that the memory information and the tool information as components of the to-be-input information can be used as prompt information of the task instruction, so as to facilitate the agent to fully perceive the real intention of the user.

[0148] In some embodiments, the processor can record the task completion of the agent to feedback to the display device according to the task completion. That is, the processor performs inputting the to-be-input information to the agent to control the agent to determine the target task and the target task tool based on the to-be-input information, and is further configured to:

[0149] obtain a task processing record of the task processing model; the task processing record includes a task processing result and a task iteration number of the task processing model.

[0150] if the task processing result represents that the task processing model successfully executes the task instruction, generating a target data packet according to the target task and the target task tool determined by the task processing model.

[0151] if the task iteration number is greater than an iteration threshold, generating a failure prompt information; the failure prompt information is used to prompt that the task corresponding to the current task instruction fails.

[0152] It can be understood that the agent can generate a task processing record in the process of executing the task. The task processing record can include a task processing result and a task iteration number. In this way, the processor can determine the task completion of the agent by obtaining the task processing record.

[0153] In some embodiments, the agent determines the target task and the target task tool according to the to-be-input information, and after completing the target task based on the target task tool, can stop iteration and take the final thinking result as the task completion information. For example, when the task instruction is “which city is cooler, city A or city B”, the final output of the agent can be “compare the temperature information of city A and city B, the temperature of city B will be more comfortable and make people feel cooler”. At this time, the processor can generate a target data packet according to the task completion information, and then return the target data packet to the display device 200 for subsequent interaction with the user by the display device 200.

[0154] In other embodiments, the agent can generate a failure prompt information when the task fails to complete. The processor can also generate a target data packet according to the failure prompt information and return it to the display device 200 for interaction with the user by the display device 200 to prompt the user to replace the task instruction. In this way, the waiting time process of the user can be avoided, and the use experience can be improved.

[0155] In addition, in order to prevent the task processing time process caused by the continuous iteration of the intelligent agent, the number of iterations of the intelligent agent can also be set in advance, and the number of iterations is monitored during the iteration of the intelligent agent. When the number of iterations of the intelligent agent is greater than the iteration threshold, the intelligent agent generates a failure prompt information, and the display device 200 interacts with the user by generating a target data packet and returning to the display device 200, so as to prompt the user to replace the task instruction. In this way, the interaction efficiency between the display device 200 and the user can be ensured, so as to alleviate the problem of too long waiting time of the user.

[0156] In some embodiments, when the intelligent agent executes the target task, the intelligent agent can call the associated target task tool according to the target task to execute the target task, that is, the processor inputs the to-be-input information into the task processing model to control the task processing model to determine the target task and the target task tool based on the to-be-input information, and the task processing model is further configured to:

[0157] Control the task processing model to call the target task tool based on the target task.

[0158] Package the task completion information to obtain a target data packet.

[0159] It can be understood that the target task and the target task tool have an association relationship. For example, when the target task is to query the weather, the intelligent agent will call the weather query tool; when the target task is to query the time, the intelligent agent will call the time query tool. Then, when the intelligent agent determines the target task according to the to-be-input information, the corresponding system tool can be called as the target task tool according to the target task, so as to complete the target task through the target task tool.

[0160] In some embodiments, after the intelligent agent completes the target task, the intelligent agent can generate task completion information. Then, the processor can obtain the task completion information, and package the task completion information according to the communication protocol between the display device 200 and the server to obtain a target data packet, and then return the target data packet to the display device 200.

[0161] It can be understood that the data format of the target data packet can also be set in combination with the interaction mode between the display device 200 and the user. For example, if the display device 200 interacts with the user through voice, the target data packet can be packaged as audio format data. In this way, the workload of the display device 200 can be reduced, and the interaction efficiency between the display device 200 and the user can be improved by means of the high computing power of the server.

[0162] As Figure 8As shown, the agent can extract multiple sub-target tasks from the to-be-input information in the process of executing the target task, i.e., the target task is composed of multiple sub-target tasks. The sub-target tasks also have certain correlations. The agent can generate the final task completion information in combination with the completion of each sub-target task. The process includes:

[0163] The control module controls the task processing model to call a first target task tool based on the first sub-target task, so that the task processing model executes the first sub-target task based on the first target task tool and obtains first sub-completion information.

[0164] The control module controls the task processing model to call a second target task tool based on the second sub-target task, so that the task processing model executes the second sub-target task based on the second target task tool and obtains second sub-completion information.

[0165] The control module controls the task processing model to fuse the first sub-completion information and the second sub-completion information based on the preposition semantic information contained in the to-be-input information, to obtain task completion information.

[0166] In some embodiments, the agent can obtain multiple sub-target tasks according to the to-be-input information. For example, the task instruction is "where is cooler today in A city and B city", the agent can determine that the sub-target tasks are "what is the date today", "temperature in a district of A city today", and "temperature in b district of B city today". Then the agent can call system tools one by one to process the sub-target tasks. That is, a date query tool is called to query the date corresponding to today, and weather query tools are called to obtain the temperature in a district of A city and the temperature in b district of B city, respectively, and sub-completion information is generated respectively.

[0167] It can be understood that the agent can fuse the sub-completion information when generating the sub-completion information corresponding to each sub-target task to obtain the final task completion information. For example, the agent can generate the task completion information "compared to the temperature information of a district of A city and b district of B city, the temperature of b district of B city will be more comfortable and make people feel cooler. If you have any other needs, please tell me anytime" after executing the above sub-target tasks. In this way, the processor can encapsulate the target data packet with the task completion information generated by the agent finally, so as to return the target data packet to the display device 200 for interaction with the user by the display device 200. In addition, custom interaction information such as "if you have any other needs, please tell me anytime" can be additionally added when encapsulating the target data packet, so as to improve the affinity of the display device 200 and the user in the interaction process, and improve the user experience.

[0168] As Figure 9 and Figure 10As shown, the server provided by the embodiments of the present application, when receiving the task instruction sent by the display device 200, the pre-semantic understanding module can parse the task instruction to obtain pre-semantic information, and based on the pre-semantic information, the associated memory information can be obtained from the memory information database, and the associated tool information can be obtained from the tool set. Further, the to-be-input information can be generated by a plurality of information, so that the to-be-input information includes rich user usage scenarios, thereby improving the perception ability of the agent to the real intention of the user, so as to improve the task processing ability of the agent. Wherein, the executor can be a processor in the server, or can be a controller 250 in the display device 200. In addition, the task processing process of the agent is recorded in the memory information database, so as to dynamically update the memory information database, enrich the user usage scenarios, and further form a data closed loop, which not only relieves the pressure brought by data demand, but also improves the adaptability of the agent to the environment and the ability to perceive the real intention of the user.

[0169] The embodiments of the present application also provide a display device 200, which comprises a display 260, a communication device 220 and a controller 250. Wherein, the communication device 220 is configured to establish a communication connection with the server. The controller 250 is configured to:

[0170] In response to the received task instruction, the task instruction is sent to the server, so that the server obtains pre-semantic information based on the task instruction; and the server obtains expansion information based on the pre-semantic information and the task instruction; the task instruction, the pre-semantic information and the expansion information are used to generate to-be-input information; the to-be-input information is used to input a task processing model; wherein, when the to-be-input information is input to the task processing model, the task processing model determines a target task and a target completion tool corresponding to the task instruction based on the to-be-input information.

[0171] Receiving the target data packet returned by the server, so as to interact with the user based on the target task information and the target completion tool information in the target data packet.

[0172] As Figure 4As shown, when the display device receives the task instruction input by the user through the user input interface 280, the display device can forward the task instruction to the server, and the server invokes the task processing model deployed in the server to process. Moreover, in the process of analyzing the task instruction, the server can search for memory information and tool information through the task instruction and the preposition semantic understanding corresponding to the task instruction, respectively, and input the information searched to the task processing model after fusing all the information. In this way, the information associated based on the preposition semantic understanding can provide additional reference information for the agent to solve the problem, so that the agent can fully perceive the real intention of the user, which is conducive to improving the task processing capability of the agent. Further, the interaction capability of the display device with the user can be improved, and the processing capability of the display device for the task instruction issued by the user can be enhanced.

[0173] Part of the embodiments of the present application also provide a task processing method, which can be applied to a server, and the method comprises:

[0174] In response to the received task instruction sent by the display device, the task instruction is analyzed to obtain preposition semantic information corresponding to the task instruction;

[0175] According to the task instruction and the preposition semantic information, expansion information is generated; the expansion information comprises memory information and tool information; the memory information comprises a historical conversation associated with the task instruction and the preposition semantic information; and the tool information comprises a system tool call record of the display device when the historical conversation is generated;

[0176] The task instruction, the preposition semantic information and the expansion information are fused to obtain to-be-input information;

[0177] The to-be-input information is input to a task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information;

[0178] The target data packet is fed back to the display device, so that the display device performs interaction with the user based on the target data packet.

[0179] According to the technical content, the application provides a server, a display device and a task processing method. When receiving a task instruction sent by the display device, the server can obtain preposition semantic information based on the task instruction, and search corresponding memory information and tool information according to the preposition semantic information and the task instruction respectively. The server fuses the memory information and the tool information searched respectively corresponding to the task instruction and the preposition semantic information, and can obtain to-be-input information used for inputting into a task processing model. Then, the target task and the target task tool corresponding to the task instruction are determined based on the task processing model, so as to complete the target task based on the target task tool and generate task completion information. The server can return a target data packet containing the task completion information to the display device, and the display device performs interaction with the user.

[0180] The similar parts among the embodiments provided in the application can be referred to each other, the specific embodiments provided above are only several examples under the general concept of the application, and do not limit the protection scope of the application. Any other embodiments extended according to the application scheme without creative labor belong to the protection scope of the application for those skilled in the art.

Claims

1. A server, characterized by The method comprises the following steps: a communicator is configured to establish a communication connection with a display device; a processor is configured to: in response to a received task instruction sent by the display device, parse the task instruction to obtain pre-semantic information corresponding to the task instruction; wherein the pre-semantic information is obtained based on semantic understanding structure parsing of the task instruction; the semantic understanding structure of the pre-semantic information at least includes one of domain information, intent information and slot information; the domain information is used to determine the conversation domain associated with the task instruction; the intent information is used to determine the task target corresponding to the task instruction; the slot information is used to represent replaceable words in the task instruction; generate extended information according to the task instruction and the pre-semantic information; the extended information includes memory information and tool information; the memory information includes historical conversations associated with the task instruction and the pre-semantic information; the tool information includes system tool call records of the display device when generating the historical conversations; fuse the task instruction, the pre-semantic information and the extended information to obtain to-be-input information; input the to-be-input information into a task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information; return a target data packet to the display device to enable the display device to perform interaction with a user based on the target data packet; the target data packet is obtained by encapsulating task completion information generated by the task processing model when the target task tool completes the target task.

2. The server of claim 1, wherein, The processor performs generating extended information according to the task instruction and the pre-semantic information, which is specifically configured to: search first memory information corresponding to the task instruction based on the task instruction; search second memory information corresponding to the pre-semantic information based on the pre-semantic information; fuse the first memory information and the second memory information according to the semantic understanding structure of the pre-semantic information to obtain memory information.

3. The server of claim 1, wherein, The processor is further configured to: search first tool information corresponding to the task instruction based on the task instruction; search second tool information corresponding to the pre-semantic information based on the pre-semantic information; merge the first tool information and the second tool information to obtain tool information.

4. The server of claim 1, wherein, The processor performs inputting the to-be-input information into a task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information, which is further configured to: obtain a task processing record of the task processing model; the task processing record includes a task processing result and a task iteration number of the task processing model; if the task processing result indicates that the task processing model successfully executes the task instruction, generate a target data packet according to the target task and the target task tool determined by the task processing model; if the task iteration number is greater than an iteration threshold, generate a failure prompt information; the failure prompt information is used to prompt that the task corresponding to the current task instruction fails.

5. The server of claim 4, wherein, The processor executes inputting the to-be-input information to a task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information, and is further configured to: control the task processing model to call the target task tool based on the target task; the target task tool is used to obtain associated information of an executable target task; and when the task processing model completes the target task based on the target task tool, the task processing model generates task completion information; encapsulate the task completion information to obtain a target data packet.

6. The server of claim 5, wherein, The target task includes a first sub-target task and a second sub-target task; the processor executes inputting the to-be-input information to a task processing model to control the task processing model to determine a target task and a target task tool based on the to-be-input information, and is further configured to: control the task processing model to call a first target task tool based on the first sub-target task, so that the task processing model executes a first sub-target task based on the first target task tool and obtains first sub-completion information; control the task processing model to call a second target task tool based on the second sub-target task, so that the task processing model executes a second sub-target task based on the second target task tool and obtains second sub-completion information; control the task processing model to fuse the first sub-completion information and the second sub-completion information based on preposition semantic information contained in the to-be-input information to obtain task completion information.

7. The server of claim 1, wherein, The processor executes fusion of the task instruction, the preposition semantic information and the extended information to obtain to-be-input information, and is further configured to: obtain task prompt information; the task prompt information is used to prompt a task execution target and a task execution process required by the task processing model; fuse the task instruction, the preposition semantic information, the extended information and the task prompt information to obtain to-be-input information.

8. A display device, characterized by comprise: a display; a communication device configured to establish a communication connection with a server; a controller configured to: in response to the received task instruction, send the task instruction to the server to enable the server to obtain preposition semantic information based on the task instruction; and enable the server to obtain extended information based on the preposition semantic information and the task instruction; the task instruction, the preposition semantic information and the extended information are used to generate to-be-input information; the to-be-input information is used to input a task processing model; when the to-be-input information is input to the task processing model, the task processing model determines a target task and a target completion tool corresponding to the task instruction based on the to-be-input information; the preposition semantic information is obtained based on semantic understanding structure analysis of the task instruction; the semantic understanding structure of the preposition semantic information at least includes one of domain information, intent information and slot information; the domain information is used to determine a conversation domain associated with the task instruction; the intent information is used to determine a task target corresponding to the task instruction; and the slot information is used to represent replaceable words in the task instruction; Receiving the target data packet returned by the server, so as to interact with the user based on the target task information and the target completion tool information in the target data packet.

9. A task processing method characterized by, The server of any one of claims 1-7, comprising: In response to the received task instruction sent by the display device, the task instruction is parsed to obtain the pre-semantic information corresponding to the task instruction; wherein the pre-semantic information is obtained by parsing the task instruction based on semantic understanding structure; the semantic understanding structure of the pre-semantic information at least includes one of domain information, intent information and slot information; the domain information is used to determine the conversation domain associated with the task instruction; the intent information is used to determine the task target corresponding to the task instruction; the slot information is used to represent the replaceable word in the task instruction; According to the task instruction and the pre-semantic information, generate the expansion information; the expansion information includes memory information and tool information; the memory information includes the historical conversation associated with the task instruction and the pre-semantic information; the tool information includes the system tool call record of the display device when generating the historical conversation; Fusion of the task instruction, the pre-semantic information and the expansion information, to obtain the input information; The input information is input into the task processing model to control the task processing model to determine the target task and the target task tool based on the input information; Return the target data packet to the display device to make the display device interact with the user based on the target data packet; the target data packet is obtained by encapsulating the task completion information, and the task completion information is generated by the task processing model when the target task tool completes the target task.

Images