Display device and display method

Abstract

The invention relates to the technical field of display equipment, and provides display equipment and a display method.The display equipment comprises a displayer, a player and a first controller, and the first controller is configured to send a user intention to a second controller when the user intention meets a preset condition; and after the control instruction sent by the second controller is received, the interaction content is obtained according to the control instruction sent by the second controller, and the interaction content is pushed to the user according to the control instruction sent by the second controller. According to the technical scheme, after the user intention is obtained through the first controller, the second controller generates the control instruction according to the user intention when the user intention meets the preset condition. Therefore, the second controller completes the decision-making task with relatively high computing power resource consumption, so that the first controller can accurately and quickly respond to the intention of the user, and meanwhile, the execution effect of other control tasks of the first controller is prevented from being influenced.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a display method. Background Art

[0002] With the development of human-computer interaction technology, intelligent interaction technology has gradually become a standard configuration of terminal devices (such as mobile phones, tablets or smart home products such as smart appliances). In the intelligent interaction scenario, users can control smart home appliances through the dialogue understanding system. Taking the TV as an example, users can control the content displayed on the TV through voice or text to achieve a series of interactive tasks such as watching videos, listening to music or checking the weather.

[0003] The current dialogue understanding system still uses the traditional natural language processing solution, which relies on word segmentation and semantic disambiguation. When dealing with open-ended questions or demands raised by users, it cannot accurately respond to users' questions or demands. In response to users' open-ended questions or demands, there are technical solutions in the prior art that apply large model technology to dialogue understanding systems. Dialogue understanding systems that apply large model technology can effectively respond to open-ended questions or demands raised by users, and can improve the response accuracy of dialogue understanding systems.

[0004] However, when a dialogue understanding system that uses large model technology makes decisions on open-ended questions or demands, the decision-making task requires a large amount of computing resources and the response time is long, which causes the terminal device to affect the response effect of other control tasks when processing the decision-making task. Summary of the invention

[0005] The present application provides a display method for a display device, thereby ensuring accurate and rapid response to user intentions while avoiding affecting the execution effects of other control tasks on the display device.

[0006] The present application provides a display device, comprising:

[0007] a display configured to display a user interface;

[0008] a player coupled to the display and configured to play audio;

[0009] A first controller is coupled to the display and the player and is configured to:

[0010] Acquire the user intention, and send the user intention to the second controller when the user intention meets the preset condition;

[0011] After receiving the control instruction sent by the second controller, the interactive content is obtained according to the task instruction in the control instruction, and the control instruction is generated by the second controller according to the user's intention;

[0012] According to the interactive instructions in the control instructions, the display is driven to display the interactive content on the user interface, and / or the player is driven to play the interactive content.

[0013] In this way, after obtaining the user's intention through the first controller, when the user's intention meets the preset conditions, the user's intention is sent to the second controller, and the second controller generates a control instruction according to the user's intention. In this way, the second controller completes the decision-making task that consumes a lot of computing resources, and the first controller responds to the user's intention according to the control instruction, thereby ensuring that the first controller can accurately and quickly respond to the user's intention while avoiding affecting the execution effect of other control tasks of the first controller.

[0014] In a possible implementation, the preset condition is that the user intent includes at least two interrelated intent requests; wherein the two intent requests are interrelated means that the two intent requests point to the same interactive content. Thus, when the user intent meets the preset condition, before sending the user intent to the second controller, the first controller is further configured to:

[0015] Parse multiple intent requests contained in user intent;

[0016] It is determined whether at least two intent requests among the multiple intent requests point to the same interactive task.

[0017] In this way, when the user intention meets the preset conditions, it is determined that the complexity of the user intention is high. At this time, the user intention is sent to the second controller, and the second controller generates a control instruction based on the user intention. When the user intention does not meet the preset conditions, it is determined that the complexity of the user intention meets the requirements, and the first controller can directly respond to the user intention.

[0018] In a possible implementation, in the step of obtaining the user's intention, the controller is configured to:

[0019] In response to a first instruction input by a user on a user interface, the first instruction is parsed to obtain a user intention corresponding to the first instruction.

[0020] The first instruction may be a voice instruction or a text instruction, so that the user can choose to input a voice instruction or a text instruction according to usage habits, thereby improving the user experience.

[0021] In a possible implementation, in the step of parsing the first instruction to obtain the user intention corresponding to the first instruction, the controller is specifically configured as follows:

[0022] In the case where it is parsed that the first instruction includes the first text information and the image information, extracting the indication information pointing to the image information in the first text information;

[0023] extracting second text information corresponding to the indication information from the image information according to the indication information;

[0024] According to the second text information, adjusting the first text information to obtain third text information;

[0025] The third text information is parsed to obtain the user intention.

[0026] In this way, the use of graphic and text understanding can provide users with more ways to input user intentions. In particular, when the user's intention involves an image, but the user does not understand the content displayed by the image information, the user can extract the knowledge information in the image information by inputting a first instruction including first text information and image information, thereby enabling the display device to accurately respond to the user's intention request.

[0027] In a possible implementation, before parsing the first instruction in response to the first instruction input by the user on the user interface, the controller is further configured to:

[0028] A historical first instruction input by a user on a user interface is obtained, and the first instruction is adjusted according to the historical first instruction.

[0029] In this way, the hidden information of the first instruction can be accurately obtained through the content of the user's continuous conversation, ensuring that the first controller can accurately respond to the user's intended request, thereby improving the user experience.

[0030] In a possible implementation, the second controller is disposed in the display device, and after the first controller sends the user intention to the second controller, the second controller is configured to:

[0031] Query the agent library for multiple task instructions and interaction instructions corresponding to the user's intention;

[0032] Query the association relationship between multiple task instructions and interaction instructions in the agent library, and determine the execution order corresponding to the multiple task instructions and interaction instructions;

[0033] Generate control instructions based on interaction instructions, multiple task instructions and execution order.

[0034] In this way, the generated control instructions include the execution order of multiple task instructions and interaction instructions, which can avoid execution conflicts of multiple task instructions and interaction instructions, and ensure that the execution order of multiple task instructions and interaction instructions is more reasonable, thereby improving the response efficiency of the user's first instruction and enhancing the user experience.

[0035] In a possible implementation, in the step of acquiring the interactive content according to the task instruction in the control instruction, the first controller is specifically configured as follows:

[0036] According to the execution order, multiple task instructions are executed in sequence to obtain interactive content corresponding to multiple control instructions, and, according to the interactive instructions, the display is driven to display the task content on the user interface, and / or the player is driven to play the task content.

[0037] In this way, by responding to interactive instructions and multiple task instructions in a reasonable execution order, on the one hand, the user's first instruction can be responded to quickly, and on the other hand, when executing interactive instructions, a reasonable execution order can be adopted to improve the user experience.

[0038] In a possible implementation, in the step of executing the multiple task instructions in sequence according to the execution order of the multiple task instructions to obtain the interactive contents corresponding to the multiple control instructions, the second controller is specifically configured as follows:

[0039] Execute the first task instruction among the multiple task instructions according to the execution order of the multiple task instructions to obtain the task content corresponding to the first task instruction;

[0040] Execute the current task instruction according to the task content corresponding to one or more previous task instructions to obtain the task content corresponding to the current task instruction;

[0041] The task content corresponding to the last task instruction is used as the interaction content.

[0042] In this way, when there are multiple interrelated task instructions, a reasonable execution order is adopted to respond to the task instructions, and the task content obtained by the previously executed task instructions is used to serve the subsequent task instructions, thereby ensuring the rapid and accurate execution of multiple task instructions.

[0043] In a possible implementation, after generating the control instruction, the second controller is further configured to:

[0044] The control instructions are stored in the memory corresponding to the second controller, and the control instructions are sent to the first controller. There is data isolation between the memory corresponding to the second controller and the memory corresponding to the first controller. In this way, the data exposure range of the memory corresponding to the first controller can be limited, and the risk of data in the memory corresponding to the first controller being transmitted in the public network can be reduced, thereby better protecting user privacy. In addition, when there is data isolation between the memory corresponding to the second controller and the memory corresponding to the first controller, only necessary data changes in the data in the memory corresponding to the first controller need to be synchronized to the memory corresponding to the second controller. In this way, the frequency and amount of data synchronization can be reduced, thereby speeding up the response speed and reducing the network load, and reducing the time-consuming data interaction between the memory corresponding to the second controller and the memory corresponding to the first controller.

[0045] The present application also provides a display method, which is applied to a display device, the display device including a display capable of displaying a user interface, and a player capable of playing a video, and the display method includes:

[0046] Acquire the user intention, and when the user intention meets the preset condition, drive the first controller to send the user intention to the second controller;

[0047] Driving the first controller to receive the control instruction sent by the second controller, wherein the task instruction control instruction is generated by the second controller according to the user's intention;

[0048] The first controller is driven to obtain the interactive content according to the interactive instruction in the control instruction, and the display is driven to display the task content on the user interface, and / or the player is driven to play the task content.

[0049] By applying the technical solution provided in the embodiment of the present application, after obtaining the user's intention through the first controller, when the user's intention meets the preset conditions, the user's intention is sent to the second controller, and the second controller generates a control instruction according to the user's intention. In this way, the second controller completes the decision-making task that consumes a large amount of computing resources, and the first controller responds to the user's intention according to the control instruction, thereby ensuring that the first controller can accurately and quickly respond to the user's intention while avoiding affecting the execution effect of other control tasks of the first controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0052] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0053] Figure 3 Schematic diagram of software configuration of display device provided in some embodiments of the present application

[0054] Figure 4 Schematic diagram of the display method provided in some embodiments of the present application Figure 1 ;

[0055] Figure 5 User interface diagram provided for some embodiments of the present application Figure 1 ;

[0056] Figure 6 User interface diagram provided for some embodiments of the present application Figure 2 ;

[0057] Figure 7 Schematic diagram of the display method provided in some embodiments of the present application Figure 2 ;

[0058] Figure 8 User interface diagram provided for some embodiments of the present application Figure 3 ;

[0059] Fig. 9 Schematic diagram of the display method provided in some embodiments of the present application Figure 3 ;

[0060] Fig.10 User interface diagram provided for some embodiments of the present application Figure 4 ;

[0061] Fig.11 Schematic diagram of the display method provided in some embodiments of the present application Figure 4 ;

[0062] Fig.12 Schematic diagram of the deployment position of the second controller provided in some embodiments of the present application Figure 1 ;

[0063] Fig.13 Schematic diagram of the deployment position of the second controller provided in some embodiments of the present application Figure 2 ;

[0064] Fig.14 A schematic diagram of the timing control of the display method provided in some embodiments of the present application. DETAILED DESCRIPTION

[0065] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0066] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0067] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0068] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0069] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0070] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0071] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0072] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0073] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0074] like Figure 1As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0075] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0076] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0077] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0078] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0079] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0080] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0081] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0082] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0083] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0084] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0085] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0086] In some embodiments, the user input interface 280 may be used to receive instructions input by a user, wherein the instructions input by the user may be commands, for example, inputting a voice instruction through a voice input control set on a remote controller of a smart TV. For another example, triggering a voice input control on a user interface through a remote controller is a voice instruction.

[0087] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface. For example, the operating system can directly control the display device to provide a user interface, or provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0088] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0089] The operating system can be divided into different modules or layers according to the functions implemented, for example Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0090] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0091] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0092] like Figure 3As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0093] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0094] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0095] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver (Universal Serial Bus: USB), HD MI driver (High Definition Multimedia Interface: HDMI), sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0096] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0097] The embodiment of the present application provides a display device, including a display, a player, and a first controller. The player is coupled to the display and configured to play audio. The first controller is coupled to the display and the player, and is configured to play audio. Figure 4 As shown, the first controller is configured to execute S410 to S430. It should be noted that, in actual application, in addition to executing the contents of S410 to S430, the first controller also needs to execute other control tasks of the display device.

[0098] S410, obtaining a user intention, and sending the user intention to a second controller when the user intention meets a preset condition.

[0099] In an embodiment of the present application, the user intent satisfies a preset condition to indicate that the user intent has a high complexity, for example, the preset condition is that the user intent includes at least two interrelated intent requests. For another example, the preset condition is that the number of intent requests included in the user intent is greater than a preset value.

[0100] Exemplarily, the preset condition is that the user intent includes at least two interrelated intent requests, and two intent requests are interrelated in that: the two intent requests point to the same interactive content. For example, the user intent is: to query the weather in the city where singer A's recent concert is held, and the user intent includes two intent requests. The first intent request is: to query the city where the concert is held, and the second intent request is: to query the weather in the city. The interactive task pointed to by these two intent requests is to push weather information to the user, that is, querying the city where the concert is held and querying the weather in the city are two interrelated intent requests.

[0101] For another example, the user intent is to compare the differences between actor B's works and actor C's works. This user intent contains three intent requests. The first intent request is to query actor B's work 1, the second intent request is to query actor C's work 2, and the third intent request is to compare the differences between work 1 and work 2. These three intent requests ultimately point to comparing the differences between the works of two actors, that is, querying actor B's work 1, querying actor C's work 2, and comparing the differences between work 1 and work 2 are related to each other.

[0102] Exemplarily, the preset condition is that the number of intent requests included in the user intent is greater than a preset value. For example, the preset value is 3, for example, the user intent is: play actor D's movie, increase the device volume, and query the time and location of actor D's recent concert. The intent includes four intent requests, the first intent request is: play actor D's movie, the second intent request is: increase the device volume, the third intent request is: query the time of actor D's recent concert, and the fourth intent request is: query the location of actor D's recent concert. In this case, the user intent includes at least four intent requests, resulting in a higher complexity of the user intent. At this time, the user intent will be sent to the second controller.

[0103] S420, after receiving the control instruction sent by the second controller, obtaining the interactive content according to the task instruction in the control instruction, where the control instruction is generated by the second controller according to the user's intention.

[0104] For user intents with higher complexity, it is necessary to respond to multiple intent requests included in the user intent in a certain manner or order, so as to avoid unreasonable response methods or orders of intent requests, resulting in an inability to accurately respond to user intent.

[0105] Therefore, in the embodiment of the present application, a large amount of computing resources are required to obtain a control instruction that represents how to respond to the intent request according to the user's intention. In this case, if the task of generating the control instruction is performed by the first controller, the computing resources of the first controller will be occupied in large quantities, thereby affecting the first controller to complete other control tasks, thereby affecting the performance of the display device. For example, it causes the content being displayed by the display device to freeze, and for another example, it causes the response time of other control tasks on the display device to be too long.

[0106] In order to avoid the above problems, in the technical solution provided by the present application, the control instructions are generated by the second controller according to the user's intention, so that the first controller can avoid consuming a large amount of computing resources to generate the control instructions. After receiving the control instructions sent by the second controller, the first controller obtains the interactive content according to the task instructions in the control instructions. For example, the user's intention is to query the weather of the city where singer A's recent concert is located. The control instructions obtained by the second controller according to the user's intention are: 1. Drive the retrieval agent to query the city where singer A's recent concert is located; 2. On the basis of obtaining the city where the concert is located, drive the retrieval agent to query the weather information of the city; 3. Push the queried weather information to the user. After the first controller receives the control instruction, it queries the city where singer A's recent concert is located according to the query task instruction. If the query result is: City E, then query the weather information of City E according to the query task instruction, and use the weather information of City E as the interactive content pushed to the user.

[0107] S430: According to the interactive instruction in the control instruction, drive the display to display the interactive content on the user interface, and / or drive the player to play the interactive content.

[0108] Among them, the interactive content can be pushed to the user through the display. Alternatively, the interactive content can be pushed to the user through the player. Alternatively, the interactive content can be pushed to the user through the display and the player simultaneously. For example, the user intends to query the weather in the city where singer A's recent concert is held, and display the weather information of city E through the display, or play the weather information of city E through the player, or, when the weather information of city E is displayed on the display, play the weather information of city E through the player.

[0109] By applying the technical solution provided in the embodiment of the present application, after obtaining the user's intention through the first controller, when the user's intention meets the preset conditions, the user's intention is sent to the second controller, and the second controller generates a control instruction according to the user's intention. In this way, the second controller completes the decision-making task that consumes a large amount of computing resources, and the first controller responds to the user's intention according to the control instruction, thereby ensuring that the first controller can accurately and quickly respond to the user's intention while avoiding affecting the execution effect of other control tasks of the first controller.

[0110] In some implementations, the preset condition may be that the user intent includes at least two interrelated intent requests. Then, when the user intent meets the preset condition, before sending the user intent to the second controller, the first controller is further configured to: parse multiple intent requests included in the user intent; and determine whether there are at least two intent requests among the multiple intent requests pointing to the same interactive task.

[0111] In addition, the preset condition may also be that the number of intent requests included in the user intention is greater than a preset value. In this case, when the user intention meets the preset condition, before sending the user intention to the second controller, the first controller is also configured to: parse multiple intent requests included in the user intention; and determine whether the multiple intent requests are greater than the preset value.

[0112] In this way, when the user intention meets the preset conditions, it is determined that the complexity of the user intention is high. At this time, the user intention is sent to the second controller, and the second controller generates a control instruction based on the user intention. When the user intention does not meet the preset conditions, it is determined that the complexity of the user intention meets the requirements, and the first controller can directly respond to the user intention.

[0113] In some implementations, in the step of obtaining the user's intention, the controller is configured to: in response to a first instruction input by the user on the user interface, parse the first instruction to obtain the user's intention corresponding to the first instruction.

[0114] The user may input the first instruction by voice or text. Figure 5 As shown, a voice input control is provided on the user interface. After the user triggers the voice control, the first controller obtains the first instruction input by the user. And the voice recognition technology can be used to display the first instruction in text form on the user interface. For another example, Figure 6 As shown, a text input control is provided on the user interface. After the user triggers the text control, the first instruction can be input in a text editing manner, and the text content edited by the user can be displayed in real time on the user interface.

[0115] In some implementations, in the step of parsing the first instruction to obtain the user intention corresponding to the first instruction, as follows: Figure 7 As shown, the first controller is specifically configured to execute S710 to S740.

[0116] S710: When it is parsed that the first instruction includes first text information and image information, extract indication information pointing to the image information from the first text information.

[0117] For example, if the first text information is: Who is the director of the third movie in the second row on the page, the first controller extracts the corresponding image information such as prompt information, and the instruction information is: Identify the name of the third movie in the second row on the page. For another example, which TV series has the actor in blue clothes on the screen starred in? The first controller extracts the corresponding image information such as prompt information, and the instruction information is: Identify the identity of the actor in blue clothes on the screen.

[0118] The image information may be an existing image displayed on the user interface, an image involved in a historical conversation between the user and the dialogue understanding system, or an image input by the user through an image input control.

[0119] S720: Extract second text information corresponding to the indication information from the image information according to the indication information.

[0120] For example, the first text information is: Who is the director of the third movie in the second row on the page, and the instruction information is: Identify the name of the third movie in the second row on the page, then the corresponding second text information is: The third movie in the second row on the page is movie F.

[0121] For another example, the first text information is: which TV series has the actor in blue on the screen starred in, and the instruction information is: identify the identity of the actor in blue on the screen, then the corresponding second text information is: the actor in blue on the screen is actor G.

[0122] S730: Adjust the first text information according to the second text information to obtain third text information.

[0123] For example, the first text information is: Who is the director of the third movie in the second row on the page, and the second text information is: The third movie in the second row on the page is movie F, then the corresponding third text information is: Who is the director of movie F.

[0124] For another example, the first text information is: Which TV series has the actor in blue clothes on the screen starred in? The instruction information is: Identify the identity of the actor in blue clothes on the screen. The second text information is: The actor in blue clothes on the screen is actor G. The corresponding third text information is: Which TV series has actor G starred in?

[0125] S740: parse the third text information to obtain the user's intention.

[0126] Among them, the user intention is obtained by using the above-mentioned graphic and text understanding method, such as Figure 8 As shown, when the first text information input by the user is: Who is the director of the third movie in the second row on the page, the image information includes two rows and three columns of movie resources, and the third movie in the second row on the corresponding page is Movie 6, the display can be driven to display the third text information: Who is the director of Movie 6, so that the user can understand the user's intention after understanding the text and picture, thereby improving the user experience.

[0127] In this way, the use of graphic and text understanding can provide users with more ways to input user intentions. In particular, when the user's intention involves image information, but the user does not understand the content displayed by the image information, the user can extract the knowledge information in the image information by inputting a first instruction including first text information and image information, thereby enabling the display device to accurately respond to the user's intention request.

[0128] In some implementations, before parsing the first instruction in response to the first instruction input by the user on the user interface, the first controller is further configured to: obtain the historical first instructions input by the user on the user interface, and adjust the first instruction according to the historical first instructions.

[0129] For example, the user has a continuous conversation with the dialogue understanding system. The first instruction input by the user for the first time is: compare the difference between the works of actor B and the works of actor C. The first instruction input by the user for the second time is: the cooperation relationship between the two actors. At this time, if only based on the first instruction input by the user for the second time, the first controller cannot accurately respond to the user's intended request.

[0130] In view of the above problems, in the technical solution provided by the present application, the first controller obtains the historical first instruction input by the user on the user interface, which in this example is: compare the difference between the works of actor B and the works of actor C. Based on: compare the difference between the works of actor B and the works of actor V, the first controller adjusts the first instruction to obtain: the cooperative relationship between actor B and actor C. In this way, the hidden information of the first instruction can be accurately obtained through the content of the user's continuous dialogue, ensuring that the first controller can accurately respond to the user's intended request, thereby improving the user experience.

[0131] In some implementations, the second controller is disposed in the display device, and after the first controller sends the user intention to the second controller, such as Fig. 9 As shown, the second controller is configured to execute S910 to S930.

[0132] S910, querying the agent library for multiple task instructions and interaction instructions corresponding to the user's intention.

[0133] The number of interaction instructions may also be multiple. For example, if there are multiple interaction contents that need to be pushed to the user in multiple times, multiple interaction instructions need to be generated.

[0134] Exemplarily, the first instruction input by the user is: play actor D's movie, turn up the volume, and find out when and where his latest concert is held. Then the user's intention is: play actor D's movie, increase the device volume, find out when actor D's latest concert is held, and find out where actor D's latest concert is held.

[0135] The task instructions include: determining the movie of actor D that needs to be played in the local database or network database through the retrieval agent, querying the time of the latest concert of actor D A through the retrieval agent, and querying the venue of the latest concert of actor D through the retrieval agent. The interactive instructions include: driving the display to play the movie of actor D, driving the player to increase the volume of the device, and driving the display or player to push the time and venue of the latest concert of actor D to the user.

[0136] S920, query the association relationship between multiple task instructions and interaction instructions in the intelligent agent library, and determine the execution order corresponding to the multiple task instructions and interaction instructions.

[0137] In this example, the three tasks of determining the movies of actor D that need to be played, querying the time of actor D's latest concert, and querying the location of actor D's latest concert all require the use of retrieval agents. At this time, a reasonable execution order needs to be arranged to ensure that the retrieval agent completes the three retrieval tasks while taking into account the execution of interactive instructions.

[0138] Also, it is necessary to drive the display to play the movie of actor D, drive the player to increase the device volume, and drive the display or player to push the time and location of actor D's latest concert to the user.

[0139] For the above-mentioned task instructions and interaction instructions, it is necessary to determine a suitable execution order in order to execute multiple task instructions and interaction instructions. For example, the task order is: drive the retrieval agent to determine the movie of actor D that needs to be played, drive the display to play the movie of actor D, drive the player to increase the device volume, drive the retrieval agent to query the time of actor D's latest concert, drive the retrieval agent to query the venue of actor D's latest concert, and drive the display or player to push the time and venue of actor D's latest concert to the user.

[0140] In this way, the movie of actor D can be played for the user as soon as possible, reducing the user's waiting time. In the process of playing the movie of actor D, other task instructions can be completed, and other interactive content can be pushed to the user in an appropriate manner.

[0141] S930, generating a control instruction according to the interaction instruction, the plurality of task instructions and the execution order.

[0142] In this way, the generated control instructions include the execution order of multiple task instructions and interaction instructions, which can avoid execution conflicts of multiple task instructions and interaction instructions, and ensure that the execution order of multiple task instructions and interaction instructions is more reasonable, thereby improving the response efficiency of the user's first instruction and enhancing the user experience.

[0143] In some implementations, in the step of obtaining interactive content according to task instructions in the control instructions, the first controller is specifically configured to: execute multiple task instructions in sequence according to an execution order to obtain interactive content corresponding to multiple control instructions, and, according to the interactive instructions, drive the display to display the task content on the user interface, and / or drive the player to play the task content.

[0144] For example, the execution order of the control instructions is: driving the retrieval agent to determine the movie of actor D that needs to be played, driving the display to play the movie of actor D, driving the player to increase the device volume, driving the retrieval agent to query the time of actor D's latest concert, driving the retrieval agent to query the venue of actor D's latest concert, and driving the display or player to push the time and venue of actor D's latest concert to the user.

[0145] According to the execution order, the first controller drives the retrieval agent to determine the movie of actor D that needs to be played, and obtains the movie 1 of actor D. Then it drives the display to play the movie of actor D, and drives the player to increase the volume of the device. Then, the first controller drives the retrieval agent to query the time of actor D's recent concert, and drives the retrieval agent to query the venue of actor D's recent concert. After obtaining the time of actor D's recent concert, after obtaining the time and venue of actor D's recent concert, if Fig.10 As shown, when the movie 1 is displayed on the user interface of the display, the display is driven to push the time and location of the latest concert of actor D to the user. Alternatively, the player is driven to push the time and location of the latest concert of actor D to the user.

[0146] In this way, by responding to interactive instructions and multiple task instructions in a reasonable execution order, on the one hand, the user's first instruction can be responded to quickly, and on the other hand, when executing interactive instructions, a reasonable execution order can be adopted to improve the user experience.

[0147] In some implementations, in the step of sequentially executing the multiple task instructions according to the execution order of the multiple task instructions to obtain the interactive contents corresponding to the multiple control instructions, such as Fig.11 As shown, the second controller is specifically configured to execute S1110 to S1130.

[0148] S1110, executing a first task instruction among the multiple task instructions according to the execution order of the multiple task instructions to obtain task content corresponding to the first task instruction.

[0149] For example, the user's intention is to query the weather in the city where singer A's concert is held. The corresponding task instructions are: drive the retrieval agent to query the city where singer A's concert is held and drive the retrieval agent to query the weather in the city where the concert is held. The corresponding interaction instructions are: drive the display and / or player to push the weather of the city where singer A's concert is held to the user. Then the execution order is: drive the retrieval agent to query the city where singer A's concert is held, drive the retrieval agent to query the weather in the city where the concert is held, and drive the display and / or player to push the weather of the city where singer A's concert is held to the user.

[0150] In this way, the first controller first drives the retrieval agent to query the city where singer A's concert is located, and the task content is: singer A's concert is in city E.

[0151] S1120, executing the current task instruction according to the task content corresponding to one or more previous task instructions to obtain the task content corresponding to the current task instruction.

[0152] In this example, the task content corresponding to the previous task instruction is that singer A's concert is in city E. Based on the fact that singer A's concert is in city E, the driven retrieval agent is executed to query the weather of the city where the concert is located, that is, to retrieve the weather of city E, thereby obtaining the corresponding task content: the weather of the city where singer A's recent concert is located.

[0153] S1130, taking the task content corresponding to the last task instruction as the interactive content.

[0154] In this example, the user intention includes two interrelated task instructions, so the task content of the second task instruction is interactive content. In actual application, if the number of interrelated task instructions is greater, the implementation process is the same as the above implementation process.

[0155] In this way, when there are multiple interrelated task instructions, a reasonable execution order is adopted to respond to the task instructions, and the task content obtained by the previously executed task instructions is used to serve the subsequent task instructions, thereby ensuring the rapid and accurate execution of multiple task instructions.

[0156] In some implementations, the second controller may be disposed in the display device, or the second controller may be independently disposed outside the display device. Fig.12 As shown, the second controller is set on an independent terminal device and establishes a communication connection with the display. Fig.13 As shown, the second controller is set in the cloud, and the display device establishes a communication connection with the cloud.

[0157] When the second controller is set in the display device, the first controller may serve multiple applications, and in this case, the second controller may also serve multiple applications. When the second controller is independently set outside the display device, the first controller may serve multiple applications, and in this case, the second controller may not only serve multiple applications, but also serve multiple display devices.

[0158] In response to the above situation, in an embodiment of the present application, after generating a control instruction, the second controller is further configured to: store the control instruction in a memory corresponding to the second controller, and send the control instruction to the first controller, and there is data isolation between the memory corresponding to the second controller and the memory corresponding to the first controller.

[0159] In this way, the first controller sends the user intention to the second controller, the control instruction is stored in the memory corresponding to the second controller, and the first controller obtains and executes the control instruction. However, the control instruction is not stored in the database corresponding to the first controller, and data isolation is formed between the memory corresponding to the second controller and the memory corresponding to the first controller. In this way, when the second controller is set on the display device and the first controller serves multiple applications, data interaction between multiple applications can be avoided, thereby protecting user privacy. When the second controller is independently set outside the display device, data isolation is formed between the memory corresponding to the second controller and the memory corresponding to the first controller, so that the data exposure range of the memory corresponding to the first controller can be limited, and the risk of data transmission in the memory corresponding to the first controller in the public network can be reduced, so as to better protect user privacy. And when there is data isolation between the memory corresponding to the second controller and the memory corresponding to the first controller, only the necessary data changes in the data in the memory corresponding to the first controller need to be synchronized to the memory corresponding to the second controller. In this way, the frequency and amount of data synchronization can be reduced, thereby accelerating the response speed and reducing the network load, and reducing the time-consuming data interaction between the memory corresponding to the second controller and the memory corresponding to the first controller.

[0160] In order to more clearly illustrate the technical solutions of the embodiments of the present application, some embodiments of the technical solutions of the present application are introduced below from the perspective of an intelligent agent.

[0161] For example, in the embodiments of the present application, Fig.14 As shown, the intelligent agents include intent recognition agent, image and text understanding agent, decision-making agent, retrieval agent, retrieval enhancement agent and terminal interaction agent.

[0162] Among them, the intent agent does not require strong creative capabilities, so the computing power resource requirements are relatively small. It is deployed on the user's end-side display device or edge server and is responsible for quickly parsing the user's intent and rewriting the user's input content. Because it is close to the data source of the user's input, the end-side agent can provide a low-latency response and improve the user experience.

[0163] Decision-making agents require strong task analysis and orchestration capabilities, and are deployed independently, such as in the cloud, to query the agent library and generate decision-making scheduling tasks. Cloud agents use powerful computing power and rich data resources to process complex decision-making tasks and send them to corresponding agents for execution.

[0164] Among them, the agent library is responsible for storing the capability descriptions and calling methods of various agents, and can dynamically register and uninstall agents as needed. When registering the agent library, pay attention to the need to deploy according to the computing power requirements of the agent. For example, the image and text understanding agent is responsible for understanding the content of the image, consumes less computing resources, and also needs to be close to the data source to reduce response time, so it can be deployed on the display device on the terminal side. The enhanced search agent is responsible for collecting and summarizing network-related content. It requires strong summarization capabilities and needs to be deployed independently. A more powerful base model can be used. The terminal interaction agent is the agent closest to the user experience, so it needs to be deployed on the terminal display device to provide a shorter response time. It is responsible for generating terminal control commands and querying terminal resource information.

[0165] by Fig.14 For example, after the first controller obtains the first instruction input by the user, it drives the intention agent to recognize the first instruction. In some schemes, the first instruction includes first text information and image information, so it is necessary to request image parsing support. The first controller drives the image-text understanding agent to parse the image information and return the image-text understanding content. Based on the image-text understanding content, the first controller obtains the user's intention and sends the user's intention to the second controller. The second controller drives the decision-making agent to query the agent library for multiple task instructions and interaction instructions corresponding to the user's intention according to the user's intention, that is, the multi-task scheduling result, and feeds the multi-task scheduling result back to the first controller. The first controller drives the terminal interaction agent to perform tasks according to the multi-task scheduling structure. For example, query tasks, retrieval tasks, or dialogue tasks.

[0166] In some embodiments, the user intention may also include a summary of the search content. At this time, since the summary task requires large computing resources, a search-added task can be sent to the second controller, and the second controller drives the search-added agent to summarize the search content, thereby obtaining the task content of the search summary, and feeding back the content of the search summary to the first controller. On this basis, the controller again drives the terminal interaction agent to execute the corresponding control task, or displays the task content of the search summary to the user.

[0167] The embodiment of the present application further provides a display method, which is applied to a display device, wherein the display device includes a display capable of displaying a user interface and a player capable of playing a video, and the display method includes:

[0168] The user intention is obtained, and when the user intention meets a preset condition, the first controller is driven to send the user intention to the second controller.

[0169] The first controller is driven to receive the control instruction sent by the second controller, wherein the task instruction control instruction is generated by the second controller according to the user's intention.

[0170] The first controller is driven to obtain the interactive content according to the interactive instruction in the control instruction, and the display is driven to display the task content on the user interface, and / or the player is driven to play the task content.

[0171] By applying the technical solution provided in the embodiment of the present application, after obtaining the user's intention through the first controller, when the user's intention meets the preset conditions, the user's intention is sent to the second controller, and the second controller generates a control instruction according to the user's intention. In this way, the second controller completes the decision-making task that consumes a large amount of computing resources, and the first controller responds to the user's intention according to the control instruction, thereby ensuring that the first controller can accurately and quickly respond to the user's intention while avoiding affecting the execution effect of other control tasks of the first controller.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; a player, coupled to the display and configured to play audio; A first controller is coupled to the display and the player and is configured to: Acquire a user intention, and send the user intention to a second controller when the user intention meets a preset condition; After receiving the control instruction sent by the second controller, acquiring the interactive content according to the task instruction in the control instruction, wherein the control instruction is generated by the second controller according to the user intention; According to the interactive instruction in the control instruction, the display is driven to display the interactive content on the user interface, and / or the player is driven to play the interactive content.

2. The display device according to claim 1, characterized in that The preset condition is that the user intention includes at least two interrelated intention requests; The two intention requests being associated with each other means that the two intention requests point to the same interactive content.

3. The display device according to claim 2, characterized in that When the user intention meets the preset condition, before sending the user intention to the second controller, the first controller is further configured to: Parsing multiple intent requests included in the user intent; It is determined whether at least two intent requests among the multiple intent requests point to the same interactive task.

4. The display device according to claim 1, characterized in that In the step of obtaining the user's intention, the controller is configured to: In response to a first instruction input by a user on the user interface, the first instruction is parsed to obtain a user intention corresponding to the first instruction.

5. The display device according to claim 4, characterized in that In the step of parsing the first instruction to obtain the user intention corresponding to the first instruction, the controller is specifically configured to: In the case where it is parsed that the first instruction includes first text information and image information, extracting indication information pointing to the image information from the first text information; extracting second text information corresponding to the indication information from the image information according to the indication information; According to the second text information, adjusting the first text information to obtain third text information; The third text information is parsed to obtain the user intention.

6. The display device according to claim 4, characterized in that In response to a first instruction input by a user on the user interface, before parsing the first instruction, the controller is further configured to: A historical first instruction input by a user on the user interface is obtained, and the first instruction is adjusted according to the historical first instruction.

7. The display device according to claim 1, characterized in that The second controller is disposed in the display device, and after the first controller sends the user intention to the second controller, the second controller is configured to: Querying the agent library for multiple task instructions and interaction instructions corresponding to the user intention; Querying the association relationship between the plurality of task instructions and the interaction instructions in the agent library, and determining the execution order corresponding to the plurality of task instructions and the interaction instructions; The control instruction is generated according to the interaction instruction, a plurality of the task instructions and the execution order.

8. The display device according to claim 7, characterized in that In the step of acquiring the interactive content according to the task instruction in the control instruction, the first controller is specifically configured to: According to the execution order, multiple task instructions are executed in sequence to obtain interactive content corresponding to multiple control instructions, and, according to the interactive instructions, the display is driven to display the task content on the user interface, and / or the player is driven to play the task content.

9. The display device according to claim 8, characterized in that In the step of executing the plurality of task instructions in sequence according to the execution order of the plurality of task instructions to obtain the interactive contents corresponding to the plurality of control instructions, the second controller is specifically configured as follows: Execute the first task instruction among the plurality of task instructions according to the execution order of the plurality of task instructions to obtain the task content corresponding to the first task instruction; Execute the current task instruction according to the task content corresponding to one or more of the previous task instructions to obtain the task content corresponding to the current task instruction; The task content corresponding to the last task instruction is used as the interaction content.

10. The display device according to claim 7, characterized in that After generating the control instruction, the second controller is further configured to: The control instruction is stored in a memory corresponding to the second controller, and the control instruction is sent to the first controller, and data isolation exists between the memory corresponding to the second controller and the memory corresponding to the first controller.

11. A display method, characterized in that: Applied to a display device, the display device includes a display capable of displaying a user interface, and a player capable of playing a video, the display method includes: Acquire a user intention, and when the user intention meets a preset condition, drive the first controller to send the user intention to the second controller; driving the first controller to receive a task instruction in a control instruction sent by the second controller, wherein the control instruction is generated by the second controller according to the user intention; The first controller is driven to obtain interactive content according to the interactive instruction in the control instruction, and the display is driven to display the task content on the user interface, and / or the player is driven to play the task content.

Images