Display device and display device control method

Abstract

The invention discloses a display device and a display device control method, and the method comprises the steps: receiving an interaction instruction inputted by a user, and obtaining an interaction text corresponding to the interaction instruction; aiming at a process of outputting an operation instruction corresponding to the interaction text for the first time, at least taking the interaction text and an operation specification prompt for guiding the large model to output the operation instruction as input of the large model, and at least outputting the first operation instruction for operating a demonstration function; for the process of not outputting the operation instruction corresponding to the interaction text for the first time, at least inputting the interaction text, the content output by the large model last time and the operation specification prompt into the large model, and at least outputting the current operation instruction; and under the condition that the operation instruction is output each time, executing the output operation instruction to complete the execution action sequence for operating the demonstration function so as to realize the corresponding operation function until the execution of the operation instruction output at the last time is completed. By adopting the method, the intelligent controllability of the display equipment can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a display device control method. Background Art

[0002] With the rapid development of display device functions, the functions that display devices can provide to users are becoming more and more abundant. At present, display devices include smart TVs, smart set-top boxes, smart boxes, and products with smart display screens. Taking smart TVs as an example, smart TVs can be intelligently controlled by receiving user voice commands.

[0003] However, the voice control currently supported on smart TVs can only realize some very simple operations, such as turning on and off, adjusting the volume, etc., and the functions supported by voice control are all preset or planned in advance, and cannot complete the user's complex commands and non-preset commands, resulting in insufficient intelligent controllability and difficulty in meeting user needs. Summary of the invention

[0004] The present application provides a display device and a display device control method to solve the problem that traditional display devices lack intelligent controllability and are difficult to meet user needs.

[0005] In a first aspect, some embodiments provide a display device, comprising: a display and at least one processor. The display is configured to display content from a broadcast system or a network and / or a user interface; and the at least one processor is configured to:

[0006] Receive the interactive command input by the user and obtain the interactive text corresponding to the interactive command;

[0007] For the process of outputting the operation instruction corresponding to the interactive text for the first time, at least the interactive text and the operation specification prompt for guiding the large model to output the operation instruction are used as inputs of the large model, and at least the first operation instruction for operating the demonstration function is output;

[0008] For the process of outputting the operation instructions corresponding to the interactive text which is not the first time, at least the interactive text, the content outputted by the big model last time, and the operation specification prompt are inputted into the big model, and at least the current operation instructions are outputted;

[0009] In the case of each output operation instruction, the output operation instruction is executed to complete the execution action sequence of operating the demonstration function to realize the corresponding operation function until the execution of the last output operation instruction is completed.

[0010] Technical effect: By receiving the interactive instructions input by the user, the interactive text corresponding to the interactive instructions and the operating specification prompts for guiding the large model to output the operating instructions as the input of the large model, the decision-making and planning capabilities of the large model are used to realize intelligent path planning. The intelligent path planning includes the operating instructions for operating the demonstration function output by the large model each time; for the process of outputting the operating instructions corresponding to the interactive text that is not the first time, the input of the large model also includes the content of the last output of the large model, to ensure that the large model can decide the current output operating instructions based on the completed content, thereby improving the accuracy and rationality of each output operating instruction; in the case of each output operating instruction, the output operating instruction is executed to complete the execution action sequence for operating the demonstration function to realize the corresponding operating function, until the execution of the last output operating instruction is completed, thereby realizing intelligent control matching the user's interactive instructions, which is conducive to meeting the user's intelligent control needs for the display device.

[0011] In some embodiments, the processor executes the output operation instruction to complete the execution action sequence of operating the demonstration function, and is configured to:

[0012] In the case where the operation instruction is to open a third-party application, if the currently displayed page is an application icon summary page, text recognition is performed on a page image corresponding to the currently displayed page to obtain the location of the application name of each application;

[0013] Matching the application name of the third-party application to be opened with the application name in the currently displayed page, and obtaining the target position of the application name of the third-party application to be opened in the currently displayed page;

[0014] Based on the target location, open the third-party app.

[0015] Technical effect: When the operation instruction is to open a third-party application, a method combining text recognition and image recognition is used to identify the location of the application name of each application. Compared with the traditional method of using only image recognition, the recognition speed is improved, the power consumption is reduced, and the accuracy of the target location is improved.

[0016] In some embodiments, the processor performs a process of obtaining the application name of the third-party application to be opened, and is configured to:

[0017] Through the large model, extract the application names of the third-party applications that need to be opened in the interactive text;

[0018] Convert the reference application name into the generic application name of the third-party application that needs to be opened;

[0019] Based on the correspondence between the general application name and the application name presented by the display device after the third-party application to be opened is installed, the application name of the third-party application to be opened is obtained.

[0020] Technical effect: The reference application name of the third-party application that needs to be opened in the interactive text is extracted through a large model, and then the universal application name corresponding to the reference application name is determined, and the application name corresponding to the universal application name and presented after installation in the display device is used as the application name of the third-party application that needs to be opened. Through multi-level conversion, the application name in the interactive instruction input by the user is converted into an application name that can be recognized and processed by the display device, thereby improving the intelligent controllability and control accuracy of the display device.

[0021] In some embodiments, the processor executes the output operation instruction to complete the execution action sequence of operating the demonstration function, and is configured to:

[0022] In the case where the operation instruction is to input text, if a virtual keyboard is displayed in the current display page, text recognition is performed on the page image corresponding to the current display page to obtain the position of each key in the virtual keyboard;

[0023] Obtain a character sequence corresponding to the content to be searched, match each character in the character sequence with the character displayed on the key, and obtain a target position of each character in the character sequence in the virtual keyboard;

[0024] Based on the target location, the corresponding buttons are triggered in sequence for content search.

[0025] Technical effect: When the operation instruction is to input text, the position of each key in the virtual keyboard is obtained by performing text recognition on the page image corresponding to the currently displayed page to ensure the accuracy of key triggering, and each character in the character sequence corresponding to the content to be searched is matched with the character displayed on the key to obtain the target position of each character in the character sequence in the virtual keyboard. In this way, it can be ensured that the content matching the content to be searched is searched, which is conducive to improving the accuracy of content search.

[0026] In some embodiments, the processor is further configured to:

[0027] After the currently output operation instruction is executed, the page image before the execution is completed is compared with the page image after the execution is completed;

[0028] In case of inconsistency between the two, the currently output operation instruction is executed successfully and continues to be executed;

[0029] If the two are inconsistent, the currently output operation instruction fails to execute. At least the interactive text, the content of the previous output of the large model, and the operation specification prompt are input into the large model, and the current operation instruction is re-output and continued.

[0030] Technical effect: After the currently output operation instruction is executed, the page images before and after the execution are compared. When the two are consistent, it indicates that the execution is successful and the subsequent steps can be continued. When the two are inconsistent, it indicates that the execution has failed and the output steps of the current operation instruction will be re-executed. Through this abnormal state processing method, it is guaranteed that when the execution fails, the exception can be automatically handled to ensure the smooth progress of the control method, which is conducive to improving the user experience.

[0031] In some embodiments, the input content of the current processing process of the large model also includes a page image corresponding to the currently displayed page; before the processor executes the output current operation instruction to complete the execution action sequence of operating the demonstration function, it is also configured to:

[0032] When it is recognized through the large model that the currently displayed page displays recommended content, the recommended content is skipped or the recommended content presentation is waited for to be completed.

[0033] Technical effect: Identifying the recommended content in the currently displayed page, skipping the recommended content or waiting for the recommended content presentation to be completed can avoid interference of the recommended content with the action execution, which is conducive to improving the success rate of action execution.

[0034] In some embodiments, the large model also outputs at least one of an action description text of the output operation instruction or a description text of the target of the executed action.

[0035] Technical effect: By using at least one of the action description text or the description text of the target of the executed action as the content output by the big model, the action description text can describe the action of the operation instruction, and each action description text also includes the relevant description content of the last operation instruction; the description text can describe the target of the operation instruction, and the description text of each executed action also includes the relevant target description content of the last executed work, which is conducive to the big model accurately grasping the action executed by each operation instruction or at least one of the targets of the executed action, thereby facilitating the big model's decision-making accuracy for the next output operation instruction.

[0036] In some embodiments, for a process that is not the first time outputting an operation instruction corresponding to the interactive text, the input content of the current processing process of the large model also includes a summary text of the completed action.

[0037] Technical effect: The summary text includes a summary of all actions completed before the current processing. In this way, the large model can output the current operation instructions based on the summary text of the completed actions, which is conducive to improving the decision-making accuracy of the large model.

[0038] In a second aspect, some embodiments further provide a display device control method, which is applied to the display device provided in the first aspect, the display device comprising: a display and at least one processor, and the method comprising:

[0039] Receive the interactive command input by the user and obtain the interactive text corresponding to the interactive command;

[0040] For the process of outputting the operation instruction corresponding to the interactive text for the first time, at least the interactive text and the operation specification prompt for guiding the large model to output the operation instruction are used as inputs of the large model, and at least the first operation instruction for operating the demonstration function is output;

[0041] For the process of outputting the operation instructions corresponding to the interactive text which is not the first time, at least the interactive text, the content outputted by the big model last time, and the operation specification prompt are inputted into the big model, and at least the current operation instructions are outputted;

[0042] In the case of each output operation instruction, the output operation instruction is executed to complete the execution action sequence of operating the demonstration function to realize the operation function corresponding to the operation instruction, until the execution of the last output operation instruction is completed.

[0043] Technical effect: By receiving the interactive instructions input by the user, the interactive text corresponding to the interactive instructions and the operating specification prompts for guiding the large model to output the operating instructions as the input of the large model, the decision-making and planning capabilities of the large model are used to realize intelligent path planning. The intelligent path planning includes the operating instructions for operating the demonstration function output by the large model each time; for the process of outputting the operating instructions corresponding to the interactive text that is not the first time, the input of the large model also includes the content of the last output of the large model, to ensure that the large model can decide the current output operating instructions based on the completed content, thereby improving the accuracy and rationality of each output operating instruction; in the case of each output operating instruction, the output operating instruction is executed to complete the execution action sequence for operating the demonstration function to realize the corresponding operating function, until the execution of the last output operating instruction is completed, thereby realizing intelligent control matching the user's interactive instructions, which is conducive to meeting the user's intelligent control needs for the display device.

[0044] In some embodiments, executing the output operation instruction to complete the execution action sequence of operating the demonstration function includes:

[0045] In the case where the operation instruction is to open a third-party application, if the currently displayed page is an application icon summary page, text recognition is performed on a page image corresponding to the currently displayed page to obtain the location of the application name of each application;

[0046] Matching the application name of the third-party application to be opened with the application name in the currently displayed page, and obtaining the target position of the application name of the third-party application to be opened in the currently displayed page;

[0047] Based on the target location, open the third-party app.

[0048] Technical effect: When the operation instruction is to open a third-party application, a method combining text recognition and image recognition is used to identify the location of the application name of each application. Compared with the traditional method of using only image recognition, the recognition speed is improved, the power consumption is reduced, and the accuracy of the target location is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0051] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0052] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0053] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0054] Figure 5 A schematic diagram of a flow chart of a display device control method provided in some embodiments of the present application;

[0055] Figure 6 A flowchart of a large model decision-making process provided for some embodiments of the present application;

[0056] Figure 7 A schematic diagram of an application icon summary page provided for some embodiments of the present application;

[0057] Figure 8A schematic diagram of the processing flow of the OCR service provided in some embodiments of the present application;

[0058] Fig. 9 Schematic diagram of display page provided for some embodiments of the present application Figure 1 ;

[0059] Fig.10 Schematic diagram of display page provided for some embodiments of the present application Figure 2 ;

[0060] Fig.11 A schematic diagram of a page image before execution is completed provided in some embodiments of the present application;

[0061] Fig.12 A schematic diagram of a page image after execution is completed provided in some embodiments of the present application;

[0062] Fig.13 A schematic diagram of a processing flow of a result feedback service provided in some embodiments of the present application;

[0063] Fig.14 A schematic diagram of recommended content processing provided in some embodiments of the present application;

[0064] Fig.15 A schematic diagram of a system framework of a display device provided in some embodiments of the present application;

[0065] Fig.16 A schematic diagram of the overall flow of a display control method provided in some embodiments of the present application;

[0066] Fig.17 An internal structural diagram of a computer device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0067] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0068] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0069] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0070] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0071] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0072] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0073] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0074] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0075] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0076] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0077] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0078] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0079] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0080] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0081] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0082] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0083] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0084] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0085] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0086] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0087] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0088] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0089] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0090] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0091] In some embodiments, Figure 1As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0092] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0093] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0094] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0095] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0096] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0097] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0098] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0099] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0100] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0101] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0102] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0103] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0104] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0105] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0106] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0107] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0108] With the rapid development of display device functions, the functions that display devices can provide to users are becoming more and more abundant. At present, display devices include smart TVs, smart set-top boxes, smart boxes, and products with smart display screens. Taking smart TVs as an example, smart TVs can be intelligently controlled by receiving user voice commands.

[0109] However, the voice control currently supported on smart TVs can only realize some very simple operations, such as turning on and off, adjusting the volume, etc., and the functions supported by voice control are all preset or planned in advance, and cannot complete the user's complex commands and non-preset commands. For example, voice control of third-party applications is limited to opening or closing, and voice control of the internal interface of third-party applications cannot be performed. Therefore, traditional display devices have insufficient intelligent controllability and are difficult to meet user needs.

[0110] In order to solve the problem that traditional display devices lack intelligent controllability and are difficult to meet user needs, an embodiment of the present application provides a display device control method, which receives interactive instructions input by the user, interactive text corresponding to the interactive instructions, and operating specification prompts for guiding the large model to output operating instructions as input of the large model, and uses the decision-making and planning capabilities of the large model to realize intelligent path planning. The intelligent path planning includes the operating instructions for operating the demonstration function output by the large model each time; for the process of outputting the operating instructions corresponding to the interactive text that is not the first time, the input of the large model also includes the content output by the large model last time, to ensure that the large model can decide the current output operating instructions based on the completed content, thereby improving the accuracy and rationality of each output operating instruction; in the case of each output operation instruction, the output operation instruction is executed to complete the execution action sequence for operating the demonstration function to realize the corresponding operation function, until the last output operation instruction is executed, thereby realizing intelligent control matching the user interaction instruction, which is conducive to meeting the user's intelligent control needs for the display device.

[0111] like Figure 5 As shown, a flow chart of a display device control method provided by some embodiments is exemplarily shown. The method is applied to the above-mentioned display device, the display device includes: a display and at least one processor, and the method includes:

[0112] S100: receiving an interaction instruction input by a user, and obtaining an interaction text corresponding to the interaction instruction.

[0113] Among them, the interactive instruction refers to the instruction input by the user to control the display device. The interactive instruction can be an instruction in the form of voice, text, etc. Exemplarily, the processor can obtain the interactive instruction input by the user through the control device, or obtain the interactive instruction input by the user through the built-in voice receiving module. According to the control object, the interactive instruction can include interactive instructions for controlling local applications, interactive instructions for controlling system settings, and interactive instructions for controlling third-party applications. For example, the interactive instruction for controlling the local application can be "Please search for TV series A", the interactive instruction for controlling the system settings can be "Please turn up the volume", and the interactive instruction for controlling the third-party application can be "Please open application B to search for TV series C", etc.

[0114] The interactive text refers to the text corresponding to the interactive instruction. If the interactive instruction is a voice instruction, the processor can use a voice recognition algorithm to recognize the interactive instruction to obtain the interactive text; if the interactive instruction is a text instruction, the processor can use the text in the text instruction or the text keywords in the text instruction as the interactive text.

[0115] S200, for the process of outputting the operation instructions corresponding to the interactive text for the first time, at least the interactive text and the operation specification prompts for guiding the large model to output the operation instructions are used as inputs of the large model, and at least the first operation instruction for operating the demonstration function is output.

[0116] The big model refers to a machine learning model with large-scale parameters and complex computing structure. The big model can take the given prompt words as given conditions and automatically generate the corresponding output results of the interactive text. Therefore, the big model has certain decision-making and planning capabilities.

[0117] Demonstration functions refer to functions demonstrated in a display device, including audio and video playback control, application switching, system settings, content search, etc. The interactive instructions input by the user are used to operate at least one demonstration function.

[0118] An operation instruction refers to an instruction for operating a demonstration function, and each operation instruction is used to complete at least one execution action for operating the demonstration function. For example, the operation instructions include application homepage, my applications, open application, click, slide, input, homepage, and stop. When the operation instruction is to open an application, the corresponding at least one execution action may include exiting the current application, obtaining the name of the application to be opened, and opening the application corresponding to the application name.

[0119] In the scenario of intelligent path planning based on interactive instructions, the large model can output operating instructions for the demonstration function based on the input interactive text and the operating specification prompt. Among them, the operating specification prompt refers to the prompt word preset in the processor to guide the large model to output operating instructions. For example, the operating specification prompt can be "In the user instruction, the homepage specifically refers to the homepage of the native application; if the user instruction requires operation of the system settings, please use the action'system settings'; if you need to return to the homepage of the native application, use the action'homepage'; if you need to return to the homepage of the non-native application, use the action'application homepage'". It can be understood that the operating specification prompt is not limited to the above content. The operating specification prompt can be flexibly set according to the actual operating specifications of the display device. Due to the complexity of the functions in the display device and the diversity of applications, there can be multiple operating instructions for operating the demonstration function. Therefore, the processor uses the operating specification prompt as the input of the large model, which is conducive to guiding the large model to output operating instructions that meet the operating specifications of the display device.

[0120] In some embodiments, the interactive instructions are relatively complex, and the large model needs to output at least one operation instruction to complete the interactive instructions. For example, the interactive instruction is "Please open application B to search for TV series C", and the operation instructions output by the large model should at least include opening the application, clicking, inputting, etc., and each operation instruction should have a sequence of execution to smoothly complete the user needs indicated by the interactive instruction. In the display device control method proposed in the embodiment of the present application, the large model can output a single operation instruction each time, and multiple operation instructions can be obtained through multiple outputs of the large model.

[0121] In some embodiments, the processor uses the interactive text and the operation specification prompt as input to the large model, and the operation instruction output by the large model is used as the first operation instruction. For example, if the interactive instruction is "please open application B to search for TV series C", the first operation instruction output by the large model may be to open the application.

[0122] S300, for the process of outputting the operation instructions corresponding to the interactive text which is not the first time, at least the interactive text, the content outputted last by the large model, and the operation specification prompt are inputted into the large model, and at least the current operation instructions are outputted.

[0123] The content of the last output of the large model includes the operation instruction of the last output. For example, the content of the last output of the large model may be: Step-1: Operation instruction: Open application (B).

[0124] For the process of outputting the corresponding operation instructions of the interactive text other than for the first time, the input of the big model includes the interactive text, the content of the last output of the big model, and the operation specification prompts. The introduction of the operation instructions of the last output is helpful for the big model to clarify the historical operation process, thereby ensuring that the order of the operation instructions output by the big model each time meets the actual operation timing requirements corresponding to the interactive instructions. For example, the interactive instruction is "Please open the B application to search for TV series C", the operation instruction output for the first time can be open the application (B), the operation instruction output for the second time can be click (search box), and the operation instruction output for the third time can be input (TV series C). Figure 6 The following is a flowchart of the big model decision process provided by some embodiments of the present application. The input of the big model includes at least interactive text, the content of the last output of the big model, and operation specification prompts, and the content of the big model decision output includes at least operation instructions.

[0125] S400. In each case of outputting an operation instruction, executing the output operation instruction to complete an execution action sequence for operating the demonstration function to realize a corresponding operation function, until the execution of the last output operation instruction is completed.

[0126] Among them, the execution action sequence refers to multiple execution actions with a sequence used to complete the operation of the demonstration function, and the execution action sequence can also have a single execution action. Since it is often necessary to complete at least one execution action when executing an operation instruction to operate the demonstration function, and at least one execution action has an execution sequence, therefore, at least one execution action can constitute an execution action sequence, and the execution of each operation instruction requires the completion of the execution action sequence for the corresponding demonstration function. For example, the operation instruction is: open application (B), and the execution action sequence that needs to be completed may include exiting the current application, searching for application B, clicking the application icon of application B, etc.

[0127] Each time the large model outputs an operation instruction, the processor will complete the execution sequence of the demonstration function according to the operation instruction output by the large model, thereby realizing each corresponding operation function until the last operation instruction output by the large model is completed. In this way, through the execution of multiple operation instructions, intelligent control of interactive instructions is realized.

[0128] The above-mentioned display device and display device control method receive the interactive instructions input by the user, the interactive text corresponding to the interactive instructions and the operating specification prompts for guiding the large model to output the operating instructions as the input of the large model, and use the decision-making and planning capabilities of the large model to realize intelligent path planning. The intelligent path planning includes the operating instructions for operating the demonstration function output by the large model each time; for the process of outputting the operating instructions corresponding to the interactive text that is not the first time, the input of the large model also includes the content output by the large model last time, to ensure that the large model can decide the current output operating instructions based on the completed content, thereby improving the accuracy and rationality of each output operating instruction; in the case of each output operation instruction, the output operation instruction is executed to complete the execution action sequence for operating the demonstration function to realize the corresponding operation function, until the execution of the last output operation instruction is completed, thereby realizing intelligent control matching the user's interactive instructions, which is conducive to meeting the user's intelligent control needs for the display device.

[0129] In some embodiments, the output operation instruction is executed to complete an execution action sequence for operating a demonstration function, including: in a case where the operation instruction is to open a third-party application, if the currently displayed page is an application icon summary page, text recognition is performed on the corresponding page image of the currently displayed page to obtain the location of the application name of each application; the application name of the third-party application to be opened is matched with the application name in the currently displayed page to obtain the target location of the application name of the third-party application to be opened in the currently displayed page; based on the target location, the third-party application is opened.

[0130] The applications installed in the display device include native applications and third-party applications, wherein the native applications refer to the applications that come with the display device, and the third-party applications refer to other applications installed in the display device other than the native applications.

[0131] In some embodiments, the display device currently demonstrates application A, and the user's interactive instruction instructs to open application B, and application B is a third-party application. The operation instruction generated by the large model can be to open application (application B), that is, to open a third-party application. Among them, application A can be a native application or a third-party application different from application B. Exemplarily, the operation specification prompt for determining that the operation instruction is a third-party application can be: first determine whether the application in the interactive text is consistent with the currently moving application. If not, further determine whether the application in the interactive text is a native application or a third-party application. If it is a third-party application, determine that the operation instruction is to open the third-party application, and the application opened is the application in the interactive text.

[0132] The application icon summary page refers to a page that summarizes the application icons and application names of all third-party applications. The application icon summary page can consist of a single page or multiple pages. Figure 7 The schematic diagram of the application icon summary page provided by some embodiments of the present application is shown, wherein some applications have application icons in the application icon summary page, and some applications have application icons and application names in the application icon summary page.

[0133] The page image corresponding to the currently displayed page may be a screenshot of the currently displayed page. The third-party application to be opened refers to the third-party application opened as instructed by the operation instruction. Since the operation instruction is obtained based on the interaction instruction, the third-party application to be opened is the third-party application opened as instructed by the interaction instruction.

[0134] The execution action sequence required to operate the demonstration function when executing the operation instruction may include at least one execution action. For example, at least one execution action includes, if the current display page is an application icon summary page, performing text recognition on the page image corresponding to the current display page to obtain the location of the application name of each application; matching the application name of the third-party application to be opened with the application name in the current display page to obtain the target location of the application name of the third-party application to be opened in the current display page; based on the target location, opening the third-party application. Since the processor needs to click the application icon of the corresponding application in the application icon summary page to open the third-party application, the processor needs to determine the exact location of each application icon. In some embodiments, the processor may use an image recognition method to perform image recognition on the page image corresponding to the current display page to obtain the location of the application icon of each application. However, since the application icon may contain content that changes with the recommended information, the location of the application icon obtained by the image recognition method may be inaccurate. Since there are applications with both application icons and application names in the application icon summary page, the embodiment of the present application proposes to perform text recognition on the page image corresponding to the currently displayed page to obtain the location of the application name of each application. For example, using an OCR text recognition algorithm to perform text recognition is beneficial to improving the accuracy and recognition speed of the location of the obtained application name.

[0135] For example, the position identified by using the OCR text recognition algorithm is as follows:

[0136] {word=Homepage,PosBeanLists=[PosBean{x=143,y=51},PosBean{x=183,y=51},

[0137] PosBean{x=183,y=70},PosBean{x=143,y=70}},

[0138] {word=TV series,PosBeanLists=[PosBean{x=202,y=52},PosBean{x=248,y=52},

[0139] PosBean{x=248,y=69},PosBean{x=202,y=69},]},

[0140] {word=movie,PosBeanLists=[PosBean{x=270,y=53},

[0141] PosBean{x=305,y=53},PosBean{x=305,y=69},PosBean{x=270,y=69},]},

[0142] {word=variety show,PosBeanLists=[PosBean{x=324,y=52},

[0143] PosBean{x=359,y=52},PosBean{x=359,y=69},PosBean{x=324,y=69},]}.

[0144] like Figure 8 The figure shows a processing flow diagram of the OCR service provided by some embodiments of the present application. The processor can identify the location of the application name of each application in the currently displayed page through the OCR service, match the application name of the third-party application to be opened with the application name in the currently displayed page, and use the location of the successfully matched application name as the target location. In some embodiments, image recognition is performed on the page image corresponding to the currently displayed page to obtain the location of the application icon of each application. In this way, for applications with application names, the text recognition method is used to ensure the accuracy of the target location and the speed of recognition. For applications without application names, the image recognition method is used to further improve the accuracy of the target location.

[0145] In some embodiments, after obtaining the position of the application icon of each application in the currently displayed page, the processor matches the application name of the third-party application to be opened with the application name corresponding to the application icon in the currently displayed page, and uses the position of the successfully matched application icon as the target position.

[0146] In some embodiments, the cloud server pre-stores the positions of multiple application icons in the application icon summary page. If there is an application icon position that matches the third-party application that needs to be opened among the multiple applications pre-stored in the cloud server, the position of the successfully matched application icon is used as the target position; if there is no application icon position that matches the third-party application that needs to be opened among the multiple applications pre-stored in the cloud server, the target position is determined by OCR text recognition and image recognition methods.

[0147] The processor opens the third-party application based on the target location, for example, the execution action is to click (target location), and after the execution action is completed, the third-party application is entered. Alternatively, if the target location is the location where the application name is located, the location of the corresponding application icon is determined based on the target location, the execution action is to click (the location of the application icon corresponding to the target location), and after the execution action is completed, the third-party application is entered.

[0148] In an embodiment of the present application, when the operation instruction is to open a third-party application, a method combining text recognition and image recognition is used to identify the location of the application name of each application. Compared with the traditional method of using only image recognition, the recognition speed is improved, the power consumption is reduced, and the accuracy of the target location is improved.

[0149] In some embodiments, the process of obtaining the application name of a third-party application that needs to be opened includes: extracting the reference application name of the third-party application that needs to be opened in the interactive text through a large model; converting the reference application name into a general application name of the third-party application that needs to be opened; and obtaining the application name of the third-party application that needs to be opened based on the correspondence between the general application name and the application name presented by the third-party application that needs to be opened after being installed on the display device.

[0150] Among them, since the application name in the display device may be inconsistent with the application name in the user interaction instruction, it is necessary to convert the application name in the interaction instruction so that the processor can open the correct application. For example, the application name of Tencent Video in the display device is Cloud Video Aurora.

[0151] The reference application name refers to the application name of the third-party application that needs to be opened in the interactive text obtained by using the big model. For example, the reference application name obtained by using the big model is Youku Video.

[0152] The generic application name refers to the name that the application usually uses in the display device. The reference application name may be inconsistent with the generic application name. For example, the reference application name is Youku Video, while the generic application name is KuMao TV. Exemplarily, the reference application name and the generic application name have a preset correspondence. By searching the correspondence, the generic application name corresponding to the reference application name can be determined.

[0153] refer to Figure 7 The application name of the third-party application that needs to be opened and displayed on the display device after installation can be the application name in the application icon summary page, or it can be the application name displayed in the application icon. The processor pre-stores the correspondence between the general application name and the application name of the third-party application that needs to be opened and displayed on the display device after installation. For example, the general application name is KuMiao TV, and the application name displayed on the display device after installation is CIBN KuMiao. The application name of the third-party application that needs to be opened is the application name that corresponds to the general application name and is displayed on the display device after installation.

[0154] In an embodiment of the present application, a large model is used to extract the reference application name of the third-party application that needs to be opened in the interactive text, and then the universal application name corresponding to the reference application name is determined, and the application name corresponding to the universal application name and presented after installation in the display device is used as the application name of the third-party application that needs to be opened. Through multi-level conversion, the application name in the interactive instruction input by the user is converted into an application name that can be recognized and processed by the display device, thereby improving the intelligent controllability and control accuracy of the display device.

[0155] In some embodiments, the output operation instruction is executed to complete an execution action sequence for operating a demonstration function, including: in the case where the operation instruction is input text, if a virtual keyboard is displayed in the currently displayed page, text recognition is performed on the page image corresponding to the currently displayed page to obtain the position of each key in the virtual keyboard; the character sequence corresponding to the content to be searched is obtained, each character in the character sequence is matched with the character displayed on the key, and the target position of each character in the character sequence in the virtual keyboard is obtained; based on the target position, the corresponding keys are triggered in sequence for content search.

[0156] Among them, the display device supports a search function, and the search function usually requires inputting text to obtain search results matching the text. The virtual keyboard refers to an area with multiple keys displayed in the display page. In some embodiments, the processor can determine whether the current display page has a virtual keyboard by performing image recognition on the page image corresponding to the current display page.

[0157] Each key in the virtual keyboard can be triggered, and a corresponding character is displayed on each key. The characters displayed on the keys can be in various forms such as letters, pinyin, Chinese characters, words, numbers, etc., or a combination of various forms. Fig. 9 The following is a schematic diagram of a display page provided in some embodiments of the present application. Figure 1 In the figure, virtual keys are displayed on the display page, and the characters displayed on each key include pinyin and numbers.

[0158] The content to be searched refers to the content to be searched indicated in the interactive instruction input by the user. For example, if the interactive instruction is "search for TV series C", TV series C is the content to be searched.

[0159] In order to ensure that the content to be searched is obtained, it is necessary to trigger the corresponding keys in the virtual keyboard. Therefore, it is necessary to determine the exact position of each key in the virtual keyboard. The embodiment of the present application proposes that when the operation instruction is input text, the output operation instruction is executed to complete the execution action sequence of the demonstration function, including if a virtual key is displayed in the current display page, then text recognition is performed on the page image corresponding to the current display page, the position of each key in the virtual keyboard is obtained, and the character sequence corresponding to the content to be searched is obtained, and each character in the character sequence is matched with the character displayed on the key to obtain the target position of each character in the character sequence in the virtual keyboard; based on the target position, the corresponding keys are triggered in turn for content search. Among them, text recognition can use an OCR text recognition algorithm. Reference Figure 8 , the processor can obtain the position of each key in the virtual keyboard through the OCR service.

[0160] The character sequence refers to a sequence of characters corresponding to the content to be searched, and the characters in the character sequence should be composed of characters displayed on the keys to ensure that the processor can trigger the corresponding keys according to the characters in the character sequence. For example, the character sequence corresponding to the content to be searched is the pinyin of the title of TV series C.

[0161] In some embodiments, the big model supports Chinese to Pinyin service. Using the big model, the content to be searched can be converted into corresponding multiple Pinyins, and the multiple Pinyins form a character sequence.

[0162] The processor matches each character in the character sequence with the character displayed on the key, obtains the target position of each character in the character sequence in the virtual keyboard, and triggers the keys at the corresponding target positions in sequence according to the order of the characters in the character sequence. After the key triggering is completed, the corresponding content is searched. Fig.10 The following is a schematic diagram of a display page provided in some embodiments of the present application. Figure 2 In the figure, the character sequence "LANGLABANG" is displayed in the search box, and the search results show the content corresponding to the character sequence.

[0163] In an embodiment of the present application, when the operation instruction is to input text, the position of each key in the virtual keyboard is obtained by performing text recognition on the page image corresponding to the currently displayed page to ensure the accuracy of the key triggering, and each character in the character sequence corresponding to the content to be searched is matched with the character displayed on the key to obtain the target position of each character in the character sequence in the virtual keyboard. In this way, it can be ensured that the content matching the content to be searched is searched out, which is conducive to improving the accuracy of content search.

[0164] In some embodiments, the display device control method also includes: after the currently output operation instruction is executed, the page image before the execution is completed is compared with the page image after the execution is completed; if the two are inconsistent, the currently output operation instruction is executed successfully and continues to be executed; if the two are inconsistent, the currently output operation instruction fails to execute, at least the interactive text, the content output by the large model last time, and the operation specification prompt are input into the large model, and the current operation instruction is re-output and continued.

[0165] Among them, after the operation instruction is executed, the page image will usually change. Fig.11 FIG. 1 is a schematic diagram of a page image before execution is completed provided in some embodiments of the present application, and the figure is the home page of application A. Fig.12 The following is a schematic diagram of a page image after execution is completed in some embodiments of the present application. The figure shows the home page of application B. For example, the operation instruction is to open application (application B), and the page image before execution is the home page of application A (refer to Fig.11 ), the page image after execution is the application icon summary page. For another example, the operation instruction is click (the target position corresponding to application B), and the page image before execution is the application icon summary page (reference Figure 7 ), the page image after execution is the home page of application B (reference Fig.12 ).

[0166] The processor compares the page images before and after the execution is completed, wherein the comparison method may adopt an image similarity algorithm. If the similarity result is less than a preset value, it is determined that the two are inconsistent; if the similarity result is not less than the preset value, it is determined that the two are consistent.

[0167] When the page images before and after execution are inconsistent, it indicates that a page switch is currently being performed. The processor determines that the currently output operation instruction is executed successfully, and continues to execute the step of outputting the next operation instruction and continues to execute, that is, at least the interactive text, the content output by the large model this time, and the operation specification prompt are input into the large model, and at least the step of outputting the current operation instruction is executed.

[0168] When the page images before and after execution are consistent, it indicates that no page switching is currently being performed. The processor determines that the currently output operation instruction has failed to execute. To ensure the smooth execution of the interactive instruction, the processor will re-execute the step of outputting the current operation instruction and continue to execute, that is, re-execute at least the interactive text, the content output by the large model last time, and the operation specification prompts into the large model, and re-output the current operation instruction.

[0169] like Fig.13The figure shows a processing flow diagram of the result feedback service provided by some embodiments of the present application. Among them, the result feedback service obtains the comparison results of the page images before and after the execution is completed, and feeds back the execution results corresponding to the comparison results to the processor. In addition, the processor can perform exception processing on the execution results.

[0170] In the embodiment of the present application, after the currently output operation instruction is executed, the page images before and after the execution are compared. When the two are consistent, it indicates that the execution is successful and the subsequent steps can continue to be executed. When the two are inconsistent, it indicates that the execution has failed and the output steps of the current operation instruction will be re-executed. Through this abnormal state processing method, it is guaranteed that when the execution fails, the exception processing can be automatically performed to ensure the smooth progress of the control method, which is conducive to improving the user experience.

[0171] In some embodiments, the input content of the current processing process of the big model also includes a page image corresponding to the currently displayed page; before executing the output current operation instruction to complete the execution action sequence of the demonstration function, the method also includes: when the big model identifies that the currently displayed page displays recommended content, skip the recommended content or wait for the recommended content demonstration to be completed.

[0172] Among them, before executing the current output operation instruction to complete the execution action sequence for operating the demonstration function, the currently displayed page may display recommended content, and the display of the recommended content may continue for a preset time. The display of recommended content may cause the execution of the execution action sequence to fail. For example, the last output operation instruction was to open the application (application B), and the current output operation instruction is click (search box). After opening the application (application B), the current page does not enter the homepage of application B, but displays recommended content. Therefore, executing the execution action sequence corresponding to click (search box) will cause the action to fail.

[0173] Based on this, the embodiment of the present application proposes that when the processor recognizes through the large model that the current display page displays recommended content, it skips the recommended content or waits for the recommended content to be presented. This can avoid the interference of the recommended content on the action execution, which is conducive to improving the success rate of the action execution. Fig.14 The figure shows a schematic diagram of recommended content processing provided by some embodiments of the present application, wherein the image-text understanding module recognizes the page image of the currently displayed page, determines whether the currently displayed page displays recommended content, and if so, skips the recommended content or waits for the recommended content presentation to be completed.

[0174] For the selection of skipping the recommended content or waiting for the recommended content presentation to be completed, the processor may determine the user's identity type, and if the user's identity type is a type that supports skipping the recommended content, determine to skip the recommended content. If the user's identity type is a type that does not support skipping the recommended content, determine to wait for the recommended content presentation to be completed.

[0175] In the embodiment of the present application, the recommended content in the currently displayed page is identified, and the recommended content is skipped or the presentation of the recommended content is waited for to be completed, so as to avoid the interference of the recommended content with the execution of the action, and is conducive to improving the success rate of the action execution.

[0176] In some embodiments, reference Figure 6 The large model also outputs at least one of an action description text of the output operation instruction or a description text of the target of the executed action.

[0177] The content output by the large model also includes at least one of the action description text of the output operation instruction or the description text of the target of the executed action.

[0178] The action description text refers to the text that describes the operation instruction. The action description text of the current operation instruction includes the action description content of the previous operation instruction. For example, the operation instruction is to open the application (application B), and the action description text can be: I will open application B. For another example, the operation instruction is to click (search box), and the action description text can be: I will click the search box in application B. For another example, the operation instruction is to enter (the title of TV series C), and the action description text can be: I will enter the title of TV series C in the search box of application B.

[0179] The description text refers to the text that describes the action target of the action being performed. The description text of the currently performed action includes the description content of the target of the last performed action. For example, the operation instruction is to open the application (application B), and the description text can be: in order to perform subsequent TV series search operations. For another example, the operation instruction is to click (search box), and the description text can be: in order to activate the keyboard, and then enter the title of TV series C and search. For another example, the operation instruction is to enter (the title of TV series C), and the description text can be: in order to search for TV series.

[0180] In an embodiment of the present application, by using at least one of the action description text or the description text of the target of the executed action as the content output by the big model, the action description text can describe the action of the operation instruction, and each action description text also includes the relevant description content of the last operation instruction; the description text can describe the target of the operation instruction, and the description text of each executed action also includes the relevant target description content of the last work performed, which is conducive to the big model accurately grasping the action performed by each operation instruction or at least one of the targets of the executed action, thereby facilitating the big model to improve the decision-making accuracy of the next output operation instruction.

[0181] In some embodiments, reference Figure 6 ,For the process of outputting the corresponding operation instructions of the interactive text for the first time, the input content of the current processing process of the large model also includes the summary text of the completed actions.

[0182] Among them, the summary text refers to the summary text of the completed actions. For example, after the first operation instruction is executed, the summary text may be: According to the requirements of the user instruction, the operation of opening application B has been executed to prepare for searching TV series C. For another example, after the second operation instruction is executed, the summary text may be: The operation of opening application B has been executed, and the virtual keyboard has been activated, ready to enter the title of TV series C. For another example, after the third operation instruction is executed, the summary text may be: The operation of opening application B has been executed, and the title of TV series C has been entered in the search box for search.

[0183] In some embodiments, the big model supports a history summary proxy service, which utilizes the history summary proxy service of the big model to generate a summary description text of the completed actions after each operation instruction is executed.

[0184] For the process of outputting the operation instructions corresponding to the interactive text which is not the first time, the input content of the current processing process of the large model also includes the summary text of the completed actions.

[0185] In an embodiment of the present application, the summary description text includes a summary description of all actions completed before the current processing. In this way, the large model can output the current operation instructions based on the summary description text of the completed actions, which is conducive to improving the decision-making accuracy of the large model.

[0186] To explain the display device and the display device control method and the effect in detail, a most detailed embodiment is described below:

[0187] In an embodiment of the present application, a display device is provided, comprising: a display and at least one processor, wherein the display is configured to display content from a broadcast system or a network and / or a user interface; and the at least one processor is configured to execute a display device control method.

[0188] like Fig.15 The system framework diagram of the display device provided by some embodiments of the present application is shown. Among them, the system includes an application framework, a model service and a basic service. The application framework includes a basic capability execution unit and multiple processing modules. The basic capability execution unit includes but is not limited to application homepage, my application, open application, click, slide (up, down, left, right), input, homepage and stop, etc. Multiple processing modules include but are not limited to decision-making agent module, history summary agent module, result feedback agent module, history storage module, name conversion module, keyboard judgment module, recommended content filtering module, keyboard key acquisition module, Chinese to Pinyin module, etc. Model services include but are not limited to large models, graphic understanding models, OCR services, and cloud query services, etc.

[0189] Among them, the decision-making agent module can generate current operation instructions from user interaction instructions through operation specification prompts using the understanding ability of the large model, and then parse the corresponding execution action sequence of the operation instructions, and execute the corresponding actions according to the execution action sequence to realize the corresponding operation function.

[0190] Based on the operation specification prompts, the operation instructions output by the large model include but are not limited to application homepage, my application, open application, click, slide, input, homepage and stop, etc. The execution of each operation instruction requires the completion of the execution work sequence for operating the demonstration function. When each execution action in the execution work sequence is completed, it can ensure that the corresponding operation function can be completed correctly.

[0191] In addition, when the next decision is needed, the content of the previous output of the big model, the summary text of the completed actions, and other information can be used as input for the decision, ensuring that the big model can output the current action based on the completed records.

[0192] The history summary agent module can summarize and summarize the completed actions and generate summary description text of the completed actions as an important input prompt for the decision-making agent module.

[0193] The result feedback agent module can give the execution result, including execution success and execution failure, based on the comparison of the page images before and after the execution is completed, to provide the decision premise for the operation instruction planned by the next decision agent module.

[0194] The large model training process includes: first, collecting a large amount of open source data, which can come from a variety of sources such as books, articles, websites, as well as business data, self-collected and self-annotated data, and self-constructed data. Use the collected data to pre-train the model so that it has basic language processing capabilities. Then, by adding an appropriate amount of instruction fine-tuning data containing input and output, fine-tune the instructions of the pre-trained model to make its content generation and output style professional and consistent. Finally, by constructing a professional evaluation set to verify the ability to follow instructions, further optimize the model for feedback issues, and finally obtain a large model that meets the requirements.

[0195] The name conversion module can use the large model to convert the application name referring to the third-party application that needs to be opened in the interactive text into a universal application name, and then obtain the application name of the third-party application that needs to be opened based on the correspondence between the universal application name and the application name presented after the third-party application that needs to be opened is installed on the display device.

[0196] The keyboard determination module can identify the page image of the currently displayed page through the image-text understanding model to determine whether there is a virtual keyboard in the page image.

[0197] The Chinese to Pinyin module can convert the content to be searched into the corresponding Pinyin.

[0198] The recommended content filtering module can skip the recommended content or wait for the recommended content presentation to be completed when the currently displayed page displays recommended content.

[0199] The keyboard key acquisition module can perform text recognition on the page image corresponding to the current display page when a virtual keyboard is displayed on the current display page, and obtain the position of each key in the virtual keyboard.

[0200] Among them, when the operation instruction is to open a third-party application, if the currently displayed page is an application icon summary page, text recognition is performed on the corresponding page image of the currently displayed page to obtain the position of the application name of each application; and image recognition is performed on the page image to obtain the position of the application icon of each application, and the application name of the third-party application to be opened is matched with the application name in the currently displayed page. For applications that contain application names, the position of the application name is used as the target position, and for applications that do not contain application names, the position of the application icon is used as the target position.

[0201] The following is an example of using a specific interactive instruction to control a display device. The interactive instruction input by the user is "open application B and search for TV series C", and the current application is application A. Fig.16The figure shows a general flow chart of the display control method provided in some embodiments of the present application.

[0202] For the process of outputting the operation instruction corresponding to the interactive text for the first time, the interactive text corresponding to the interactive instruction and the operation specification prompt for guiding the large model to output the operation instruction are used as the input of the large model. The content output by the large model for the first time includes the first operation instruction for operating the demonstration function, the action description text and the description text of the target of the executed action. Exemplarily, the content output by the large model for the first time can be: Step-1: [Operation: I will open application B to perform subsequent TV series search operations; Action: Open application (application B)].

[0203] For the process of outputting the operation instructions corresponding to the interactive text non-for the first time, the interactive text, the content output by the big model last time, the operation specification prompts, and the summary description text of the completed actions are used as the input of the big model. The content output by the big model non-for the first time includes each operation instruction, action description text and the description text of the target of the executed action.

[0204] Exemplarily, the second output of the large model may be: Step-1: [Operation: I will open application B to perform subsequent TV series search operations; Action: Open application (application B)]; Step-2: [Operation: I will click on the search box in application B; Action: Click (search box)].

[0205] The third output of the large model may be: Step-1: [Action: I will open application B to perform subsequent TV series search operations; Action: Open application (application B)]; Step-2: [Action: I will click on the search box in application B; Action: Click (search box)]; Step-3: [Action: I will enter the title of TV series C in the search box of application B; Action: Enter (title of TV series C)].

[0206] The content of the fourth output of the large model can be: Step-1: [Operation: I will open application B to perform subsequent TV series search operations; Action: Open application (application B)]; Step-2: [Operation: I will click on the search box in application B; Action: Click (search box)]; Step-3: [Operation: I will enter the title of TV series C in the search box of application B; Action: Enter (title of TV series C)]; Step-4: [Operation: I will click on the option corresponding to TV series C in the search results to complete the search operation in the user's instructions; Action: Click (option corresponding to TV series C)].

[0207] In the above embodiment, a display device and a display device control method receive the interactive instructions input by the user, the interactive text corresponding to the interactive instructions and the operation specification prompt for guiding the large model to output the operation instructions as the input of the large model, and use the decision-making and planning capabilities of the large model to realize intelligent path planning, and the intelligent path planning includes the operation instructions for operating the demonstration function output by the large model each time; for the process of outputting the operation instructions corresponding to the interactive text not for the first time, the input of the large model also includes the content output by the large model last time, so as to ensure that the large model can decide the current output operation instructions according to the completed content, and improve the accuracy and rationality of each output operation instruction; in the case of each output operation instruction, the output operation instruction is executed to complete the execution action sequence for operating the demonstration function to realize the corresponding operation function, until the execution of the last output operation instruction is completed, and the intelligent control matching the user's interactive instruction is realized, which is conducive to meeting the user's intelligent control needs for the display device. At the same time, by combining image recognition and text recognition, the clickable icon position can be quickly and accurately identified, and the operation of the user's interactive instruction is finally realized.

[0208] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0209] Based on the same inventive concept, the embodiment of the present application also provides a display device control apparatus for implementing the display device control method involved above. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more display device control apparatus embodiments provided below can refer to the limitations on the display device control method above, and will not be repeated here.

[0210] In an exemplary embodiment, a display device control apparatus is provided, comprising: an acquisition module, a first output module, a second output module and an execution module, wherein:

[0211] The acquisition module is used to receive the interaction instruction input by the user and obtain the interaction text corresponding to the interaction instruction.

[0212] The first output module is used for the process of outputting the operation instructions corresponding to the interactive text for the first time, at least taking the interactive text and the operation specification prompts for guiding the large model to output the operation instructions as the input of the large model, and at least outputting the first operation instruction for operating the demonstration function.

[0213] The second output module is used to input at least the interactive text, the content of the previous output of the large model, and the operation specification prompts into the large model for the process of outputting the operation instructions corresponding to the interactive text that is not the first time, and at least output the current operation instructions.

[0214] The execution module is used to execute the output operation instruction each time the operation instruction is output to complete the execution action sequence of operating the demonstration function to realize the corresponding operation function until the execution of the last output operation instruction is completed.

[0215] The above-mentioned display device control device receives the interactive instructions input by the user, the interactive text corresponding to the interactive instructions and the operating specification prompts for guiding the large model to output the operating instructions as the input of the large model, and uses the decision-making and planning capabilities of the large model to realize intelligent path planning. The intelligent path planning includes the operating instructions for operating the demonstration function output by the large model each time; for the process of outputting the operating instructions corresponding to the interactive text that is not the first time, the input of the large model also includes the content output by the large model last time, to ensure that the large model can decide the current output operating instructions based on the completed content, thereby improving the accuracy and rationality of each output operating instruction; in the case of each output operating instruction, the output operating instruction is executed to complete the execution action sequence for operating the demonstration function to realize the corresponding operating function, until the execution of the last output operating instruction is completed, thereby realizing intelligent control matching the user's interactive instructions, which is conducive to meeting the user's demand for intelligent control of the display device.

[0216] In some embodiments, the output operation instruction is executed to complete the execution action sequence for operating the demonstration function, and the execution module is also used for: when the operation instruction is to open a third-party application, if the currently displayed page is an application icon summary page, text recognition is performed on the corresponding page image of the currently displayed page to obtain the location of the application name of each application; the application name of the third-party application to be opened is matched with the application name in the currently displayed page to obtain the target location of the application name of the third-party application to be opened in the currently displayed page; based on the target location, the third-party application is opened.

[0217] In some embodiments, in the process of obtaining the application name of a third-party application that needs to be opened, the execution module is also used to: extract the reference application name of the third-party application that needs to be opened in the interactive text through a large model; convert the reference application name into a universal application name of the third-party application that needs to be opened; and obtain the application name of the third-party application that needs to be opened based on the correspondence between the universal application name and the application name presented by the third-party application that needs to be opened after it is installed on the display device.

[0218] In some embodiments, the output operation instruction is executed to complete the execution action sequence for operating the demonstration function, and the execution module is also used for: when the operation instruction is input text, if a virtual keyboard is displayed in the currently displayed page, text recognition is performed on the page image corresponding to the currently displayed page to obtain the position of each key in the virtual keyboard; the character sequence corresponding to the content to be searched is obtained, and each character in the character sequence is matched with the character displayed on the key to obtain the target position of each character in the character sequence in the virtual keyboard; based on the target position, the corresponding keys are triggered in sequence for content search.

[0219] In some embodiments, the display device control device also includes an exception handling module, which is used to: after the currently output operation instruction is executed to complete the execution action sequence for operating the demonstration function, compare the page image before the execution is completed with the page image after the execution is completed; if the two are inconsistent, the currently output operation instruction is executed successfully and continues to be executed; if the two are inconsistent, the currently output operation instruction fails to execute, at least the interactive text, the content output by the large model last time, and the operation specification prompt are input into the large model, and the current operation instruction is re-output and continued to be executed.

[0220] In some embodiments, the input content of the current processing process of the large model also includes a page image corresponding to the currently displayed page; before executing the output current operation instruction to complete the execution action sequence of the demonstration function, the display device control device also includes a recommended content processing module, and the recommended content processing module is used to: when the large model identifies that the current displayed page displays recommended content, skip the recommended content or wait for the recommended content demonstration to be completed.

[0221] In some embodiments, the large model also outputs at least one of an action description text of the output operation instruction or a description text of the target of the executed action.

[0222] In some embodiments, for a process that is not the first time outputting an operation instruction corresponding to the interactive text, the input content of the current processing process of the large model also includes a summary text of the completed action.

[0223] Each module in the above display device control device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0224] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.17 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a display device control method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0225] Those skilled in the art will understand that Fig.17 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0226] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0227] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0228] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0229] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0230] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0231] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0232] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A display device, characterized in that: include: a display configured to display content from a broadcast system or network and / or a user interface; and at least one processor configured to: Receiving an interactive instruction input by a user, and obtaining an interactive text corresponding to the interactive instruction; For the process of outputting the operation instruction corresponding to the interactive text for the first time, at least the interactive text and the operation specification prompt for guiding the large model to output the operation instruction are used as inputs of the large model, and at least the first operation instruction for operating the demonstration function is output; For a process of outputting the operation instruction corresponding to the interactive text which is not the first time, at least the interactive text, the content outputted last by the large model, and the operation specification prompt are inputted into the large model, and at least the current operation instruction is outputted; In the case of each output operation instruction, the output operation instruction is executed to complete the execution action sequence of operating the demonstration function to realize the corresponding operation function until the execution of the last output operation instruction is completed.

2. The display device according to claim 1, characterized in that The processor executes the output operation instruction to complete the execution action sequence of operating the demonstration function, and is configured as follows: In the case where the operation instruction is to open a third-party application, if the currently displayed page is an application icon summary page, text recognition is performed on a page image corresponding to the currently displayed page to obtain the location of the application name of each application; Matching the application name of the third-party application to be opened with the application name in the currently displayed page, and obtaining the target position of the application name of the third-party application to be opened in the currently displayed page; Based on the target location, a third-party application is opened.

3. The display device according to claim 2, characterized in that The processor executes a process of obtaining an application name of a third-party application to be opened, and is configured to: Extracting the application name of the third-party application to be opened in the interactive text through the large model; Convert the reference application name into a common application name of the third-party application to be opened; Based on the correspondence between the general application name and the application name of the third-party application that needs to be opened and presented on the display device after being installed, the application name of the third-party application that needs to be opened is obtained.

4. The display device according to claim 1, characterized in that The processor executes the output operation instruction to complete the execution action sequence of operating the demonstration function, and is configured as follows: In the case where the operation instruction is to input text, if a virtual keyboard is displayed in the currently displayed page, text recognition is performed on the page image corresponding to the currently displayed page to obtain the position of each key in the virtual keyboard; Acquire a character sequence corresponding to the content to be searched, match each character in the character sequence with the character displayed on the key, and obtain a target position of each character in the character sequence in the virtual keyboard; Based on the target location, corresponding buttons are triggered in sequence for content search.

5. The display device according to claim 1, characterized in that The processor is further configured to: After the currently output operation instruction is executed, the page image before the execution is completed is compared with the page image after the execution is completed; In case of inconsistency between the two, the currently output operation instruction is executed successfully and continues to be executed; If the two are inconsistent, the currently output operation instruction fails to execute, and at least the interactive text, the content last output by the large model, and the operation specification prompt are input into the large model, and the current operation instruction is re-output and continued.

6. The display device according to claim 1, characterized in that The input content of the current processing process of the large model also includes a page image corresponding to the current display page; before the processor executes the output current operation instruction to complete the execution action sequence of operating the demonstration function, it is also configured to: When it is identified through the large model that the currently displayed page displays recommended content, the recommended content is skipped or the presentation of the recommended content is waited for to be completed.

7. The display device according to claim 1, characterized in that The large model also outputs at least one of an action description text of the output operation instruction or a description text of the target of the executed action.

8. The display device according to claim 1, characterized in that For a process that is not the first time to output the operation instructions corresponding to the interactive text, the input content of the current processing process of the large model also includes a summary text of the completed actions.

9. A display device control method, characterized in that: Applied to the display device as claimed in any one of claims 1 to 8; the method comprises: Receiving an interactive instruction input by a user, and obtaining an interactive text corresponding to the interactive instruction; For the process of outputting the operation instruction corresponding to the interactive text for the first time, at least the interactive text and the operation specification prompt for guiding the large model to output the operation instruction are used as inputs of the large model, and at least the first operation instruction for operating the demonstration function is output; For a process of outputting the operation instruction corresponding to the interactive text which is not the first time, at least the interactive text, the content outputted last by the large model, and the operation specification prompt are inputted into the large model, and at least the current operation instruction is outputted; In the case of each output operation instruction, the output operation instruction is executed to complete the execution action sequence of operating the demonstration function to realize the corresponding operation function until the execution of the last output operation instruction is completed.

10. The method according to claim 9, characterized in that The execution output operation instruction is executed to complete the execution action sequence of operating the demonstration function, including: In the case where the operation instruction is to open a third-party application, if the currently displayed page is an application icon summary page, text recognition is performed on a page image corresponding to the currently displayed page to obtain the location of the application name of each application; Matching the application name of the third-party application to be opened with the application name in the currently displayed page, and obtaining the target position of the application name of the third-party application to be opened in the currently displayed page; Based on the target location, a third-party application is opened.

Images