Display device and control method for display device

Abstract

The present disclosure relates to the technical field of display devices, and relates to a display device and a control method for the display device. The display device may comprise: a display, a memory, at least one external device interface, and at least one processor, which is connected to the display, the memory, and the at least one external device interface and is configured to execute computer instructions such that the display device performs: in response to a control instruction for entering a game application, the display displaying a game application interface; and in response to a control instruction associated with a first button of a remote controller, when entering the game application, determining a first instruction associated with the first button, the first instruction being used for displaying at least one video thumbnail in a first display area of the display when the display displays the game application interface, the video thumbnail being a thumbnail generated from a video associated with the game application. The solution of the present embodiments has a simple trigger mode, and can meet personalized requirements of a user, thereby improving user experience.

Description

Display device and control method of display device

[0001] Cross-reference to Related Applications

[0002] The present disclosure claims priority to Chinese applications No. 202411321885.5, filed on September 20, 2024, and No. 202411388526.1, filed on September 30, 2024, the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of display devices, and in particular to a display device and a control method of the display device. BACKGROUND

[0004] With the development of display technology, the applications that can be realized through display devices such as smart TVs are increasing, for example, electronic games can be played through display devices.

[0005] When playing electronic games through display devices, there is usually a need to view content related to the game, such as viewing game-related game strategies. To meet this need, users usually need to use additional terminal devices, such as mobile terminals or computer devices, to query. Therefore, users need to additionally operate another terminal device to obtain content related to the game, which seriously affects the efficiency of obtaining information and user experience. SUMMARY

[0006] In a first aspect, a display device is provided, which can include: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol; at least one processor connected to the display, the at least one external device interface, and the memory, configured to execute the computer instructions to cause the display device to perform: in response to entering a game application, controlling the display to display a game application interface; in response to a control instruction associated with a first button of a remote controller, determining a first instruction associated with the first button in the case of entering the game application; the first instruction is used to display at least one video thumbnail in a first display area of the display when the game application interface is displayed on the display; the video thumbnail is a thumbnail generated from a video associated with the game application.

[0007] In a second aspect, a display device is provided, which can include: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol; and at least one processor connected with the display, the at least one external device interface, and the memory, configured to execute the computer instructions to cause the display device to perform: in response to a control instruction for entering a game application, controlling the display to display a game application interface; in response to a control instruction associated with a first button of a remote controller, determining, in a case of entering the game application, a first instruction associated with the first button, the first instruction being used to display, in a sixth display area of the display, text information and a preset number of video thumbnails when the game application interface is displayed on the display, the text information being text information associated with an interface element of the game application interface, and the video thumbnails being thumbnails generated from videos associated with the game application.

[0008] In a third aspect, a display device is provided, which can include: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol; and at least one processor connected with the display, the at least one external device interface, and the memory, configured to execute the computer instructions to cause the display device to perform: in response to a control instruction for entering a game application, controlling the display to display a game application interface; in a case of displaying the game application interface, in response to received voice input information, displaying, in a first display area of the display, at least one video thumbnail when the game application interface is displayed on the display, the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

[0009] In a fourth aspect, a display device is provided, which can include a display, a memory configured to store computer instructions and data associated with the display device, at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol, and at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform the following operations: in response to a control instruction for entering a game application, controlling the display to display a game application interface; in a case where the game application interface is displayed, in response to received voice input information, displaying, on the display, text information and a preset number of video thumbnails in a sixth display area of the display while the game application interface is displayed on the display, the text information being text information associated with the voice input information and an interface element of the game application interface, and the video thumbnails being thumbnails generated from a video associated with the voice input information and the game application.

[0010] In a fifth aspect, a display device is provided, which can include a display, a memory configured to store computer instructions and data associated with the display device, at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol, and at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform the following operations: in response to a game application start instruction, loading game application data, and controlling the display to display a game application interface; in a case where the game application data is loaded and the game application interface is displayed on the display, displaying at least one video thumbnail in a first display area of the display, the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

[0011] In a sixth aspect, a control method of a display device is provided, which can include: in response to a control instruction for entering a game application, displaying a game application interface; in response to a control instruction associated with a first key of a remote controller, determining a first instruction associated with the first key in a case where the game application is entered; and based on the first instruction, displaying at least one video thumbnail in a first display area of the display while the game application interface is displayed on the display, the video thumbnail being a thumbnail generated from a video associated with the game application.

[0012] In a seventh aspect, a control method of a display device is provided. The method can include: in response to a control instruction for entering a game application, displaying a game application interface; in response to a control instruction associated with a first key of a remote controller, determining a first instruction associated with the first key when the game application is entered; and based on the first instruction, displaying text information and a preset number of video thumbnails in a sixth display area of the display when the game application interface is displayed on the display, the text information being text information associated with an interface element of the game application interface, and the video thumbnails being thumbnails of videos associated with the game application.

[0013] In an eighth aspect, a control method of a display device is provided. The method can include: in response to a control instruction for entering a game application, displaying a game application interface; in response to received voice input information when the game application interface is displayed, displaying at least one video thumbnail in a first display area of the display when the game application interface is displayed on the display, the video thumbnail being a thumbnail of a video associated with the voice input information and the game application.

[0014] In a ninth aspect, a control method of a display device is provided. The method can include: in response to a control instruction for entering a game application, displaying a game application interface; in response to received voice input information when the game application interface is displayed, displaying text information and a preset number of video thumbnails in a sixth display area of the display when the game application interface is displayed on the display, the text information being text information associated with the voice input information and an interface element of the game application interface, and the video thumbnails being thumbnails of videos associated with the voice input information and the game application.

[0015] In a tenth aspect, a control method of a display device is provided. The method can include: in response to a game application start instruction, loading game application data and displaying a game application interface; and when the game application data is loaded and the game application interface is displayed on the display, displaying at least one video thumbnail in a first display area of the display, the video thumbnail being a thumbnail of a video associated with the voice input information and the game application. BRIEF DESCRIPTION OF DRAWINGS

[0016] FIG. 1 is a schematic diagram of an operating scenario between a display device and a control device according to some embodiments;

[0017] FIG. 2 is a schematic diagram of a hardware configuration of a display device according to some embodiments;

[0018] FIG. 3 is a schematic diagram of a hardware configuration of a control device according to some embodiments;

[0019] FIG. 4 is a schematic diagram of a software configuration of a display device according to some embodiments;

[0020] FIG. 5 is a schematic diagram of an interface displaying video thumbnails according to some embodiments;

[0021] FIG. 6 is a schematic diagram of an interface displaying a guide prompt according to some embodiments;

[0022] FIG. 7 is a schematic diagram of another interface displaying video thumbnails according to some embodiments;

[0023] FIG. 8 is a schematic diagram of an interface when playing game video content according to some embodiments;

[0024] FIG. 9 is a schematic diagram of another interface when playing game video content according to some embodiments;

[0025] FIG. 10 is a schematic diagram of a hidden interface after playing according to some embodiments;

[0026] FIG. 11 is a schematic diagram of a part of a game knowledge graph according to some embodiments;

[0027] FIG. 12 is a schematic diagram of a principle architecture of a display device obtaining text information and game video information according to some embodiments;

[0028] FIG. 13 is a schematic diagram of an interaction flow of a display device processing procedure according to some embodiments;

[0029] FIG. 14 is a schematic diagram of a flow of a control method of a display device according to some embodiments;

[0030] FIG. 15 is a schematic diagram of a video playing interface according to some embodiments;

[0031] FIG. 16 is an application interface of a memo application of a display device according to some embodiments;

[0032] FIG. 17 is a schematic diagram of a waiting interface according to some embodiments;

[0033] FIG. 18 is a schematic diagram of prompt information in a first target area according to some embodiments;

[0034] FIG. 19 is a diagram of interface state changes when a display device is powered off according to some embodiments;

[0035] FIG. 20 is an application interface of a weather forecast function of a display device according to some embodiments;

[0036] FIG. 21 is an application interface of a viewing history function of a display device according to some embodiments;

[0037] FIG. 22 is a memory playing inquiry interface according to some embodiments;

[0038] FIG. 23 is another interface state change diagram of a display device at power-off, according to some embodiments;

[0039] FIG. 24 is a schematic diagram of prompt information in a second target area, according to some embodiments;

[0040] FIG. 25 is an interface state change diagram of displaying prompt information in a second target area, according to some embodiments;

[0041] FIG. 26 is a logic diagram of a control method of a display device, according to some embodiments;

[0042] FIG. 27 is a framework diagram of a voice interactive life service system, according to some embodiments;

[0043] FIG. 28 is an execution logic diagram of a voice interactive system, according to some embodiments;

[0044] FIG. 29 is a signaling timing diagram of a control method of a display device, according to some embodiments;

[0045] FIG. 30 is another flowchart diagram of a control method of a display device, according to some embodiments;

[0046] FIG. 31 is another flowchart diagram of a control method of a display device, according to some embodiments;

[0047] FIG. 32 is another flowchart diagram of a control method of a display device, according to some embodiments;

[0048] FIG. 33 is another flowchart diagram of a control method of a display device, according to some embodiments;

[0049] FIG. 34 is another flowchart diagram of a control method of a display device, according to some embodiments. DETAILED DESCRIPTION

[0050] FIG. 1 is a schematic diagram of an operating scenario between a display device and a control device, according to some embodiments. As shown in FIG. 1, a user can operate the display device 200 through a touch operation, a mobile terminal 300 and a control device 100. For example, the control device 100 can be a remote controller, a stylus, a handle, etc.

[0051] The mobile terminal 300 can serve as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also serve as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install a software application with the display device 200, implement connection communication through a network communication protocol, and achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0052] As further shown in FIG. 1, the display device 200 also communicates data with the server 400 through various communication means. The display device 200 can be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0053] The display device 200 can provide a broadcast receiving television function, and can additionally provide a smart network television function with computer support, including but not limited to a network television, a smart television, an Internet protocol television (IPTV), and the like.

[0054] FIG. 2 is a hardware configuration block diagram of the display device 200 in FIG. 1.

[0055] In some embodiments, the display device 200 can include at least one of a tuning demodulator 210, a communication device 220, a detector 230, a device interface 240, a processor 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0056] In some embodiments, the detector 230 is used to collect signals of an external environment or interaction with the outside. For example, the detector 230 includes a light receiver (not shown in FIG. 2) for collecting an ambient light intensity sensor, or the detector 230 includes an image collector such as a camera, which can be used to collect an external environment scene, a user's attribute, or a user interaction gesture, or the detector 230 includes a sound collector such as a microphone, etc., for receiving external sound.

[0057] In some embodiments, the display 260 can include a display function component for presenting a picture, and a driving component for driving image display. The display 260 is used to receive an image signal output from the processor 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface, as well as a user control UI interface, etc.

[0058] In some embodiments, the communication device 220 is a component for communicating with the external device or the server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 according to different supported communication manners. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 containing WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 containing Bluetooth function.

[0059] The communication device 220 can connect the display device 200 with the external device or the server 400 through wireless or wired connection. The wired connection can connect the display device 200 with the external device through data line, interface, etc. The wireless connection can connect the display device 200 with the external device through wireless signal or wireless network. The display device 200 can directly establish connection relationship with the external device, or indirectly establish connection relationship through gateway, route, connection device, etc.

[0060] In some embodiments, the processor 250 can include at least one of central processor, video processor, audio processor, graphics processor, power processor, first interface to nth interface for input / output, and the processor 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The processor 250 can control the overall operation of the display device 200.

[0061] In some embodiments, the processor 250 and the tuner demodulator 210 can be located in different split devices, i.e. the tuner demodulator 210 can also be in an external device of the main device where the processor 250 is located, such as an external set-top box, etc.

[0062] In some embodiments, the user can input user commands in the graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the graphical user interface (GUI).

[0063] In some embodiments, the audio output device 270 can be a native loudspeaker of the display device 200, or an audio output device connected to the display device 200. For the audio output device connected to the display device 200, the display device 200 can also be provided with an external audio output terminal, and the audio output device can be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0064] In some embodiments, the user input interface 280 can be used to receive instructions from the user input.

[0065] Figure 3 is a hardware configuration block diagram of the control device in Figure 1. As shown in Figure 3, the control device 100 can include a processor 110, a communication interface 130, a user input / output interface, a memory, a power supply.

[0066] The control device 100 is configured to control the display device 200, and can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, acting as an intermediary between the user and the display device 200.

[0067] In some embodiments, the control device 100 can be a smart device. For example, the control device 100 can install various applications for controlling the display device 200 according to user needs.

[0068] In some embodiments, as shown in Figure 1, the mobile terminal 300 or other smart electronic device can perform similar functions to the control device 100 after installing the application for controlling the display device 200.

[0069] The processor 110 can include a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The processor 110 is used to control the operation and operation of the control device 100, and the communication and cooperation between the internal components, as well as the data processing functions of the external and internal.

[0070] The communication interface 130 can communicate control signals and data signals between the display device 200 under the control of the processor 110. The communication interface 130 can include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, and other near field communication modules.

[0071] In some embodiments, the control device 100 can include at least one of the communication interface 130 and the input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC, etc. module, which can encode user input instructions through WiFi protocol, or Bluetooth protocol, or NFC protocol and send them to the display device 200. The input interface of the user input / output interface 140 includes at least one of a microphone 141, a touchpad 142, a sensor 143, a key 144, and other input interfaces.

[0072] The memory 190 is used to store various running programs, data and applications for driving and controlling the control device 100 under the control of the processor. The memory 190 can store various control signal instructions input by the user.

[0073] The power supply 180 is used to provide operating power support for each element of the control device 100 under the control of the processor.

[0074] To perform user interactions, in some embodiments, the display device 200 can run an operating system. The operating system is a computer program used to manage and control hardware and software resources in the display device 200. The operating system can provide a user interface to allow a user to interact with the display device 200 and support running various application programs.

[0075] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0076] The operating system can be divided into different modules or levels according to the implemented functions, for example, as shown in FIG. 4, in some embodiments, the system is divided into four layers, from top to bottom, the Applications layer (referred to as the “application layer”), the Application Framework layer (referred to as the “framework layer”), the system library layer, and the kernel layer.

[0077] In some embodiments, the application layer is used to provide services and interfaces for application programs, so that the display device 200 can run application programs and interact with users based on the application programs. At least one application program can run in the application layer, which can be a window (Window) program, a system setting program, or a clock program provided by the operating system, or an application program developed by a third-party developer. In specific implementation, the application programs in the application layer are not limited to the above examples.

[0078] The framework layer provides application programming interfaces (APIs) and programming frameworks for application programs. The application framework layer includes some pre-defined functions. The application framework layer is equivalent to a processing center that decides which application program in the application layer to act. The application program can access resources in the system and obtain services of the system through the API interface during execution.

[0079] As shown in FIG. 4, the application framework layer in the embodiments of the present disclosure can include a view system (View System), managers (Managers), and a content provider (Content Provider).

[0080] The view system can design and implement the interface and interaction of the application program, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an Activity Manager for interacting with all activities running in the system; a Location Manager for providing system location services to system services or applications; a Package Manager for retrieving various information related to application packages currently installed on the device; a Notification Manager for controlling the display and clearing of notification messages; and a Window Manager for managing icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0081] In some embodiments, the Activity Manager is used to manage the life cycle of each application program and the general navigation back function, such as controlling the exit, opening, back, etc. of the application program. The Window Manager is used to manage all window programs, such as obtaining the size of the display screen, determining whether there is a status bar, locking the screen, intercepting the screen, and controlling the display window change, for example, reducing the display window, shaking the display, and twisting the display.

[0082] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to realize the functions of the framework layer.

[0083] In some embodiments, the kernel layer is a functional layer between the hardware and the software of the display device 200. The kernel layer can realize functions such as hardware abstraction, multitasking, memory management, etc. For example, as shown in FIG. 4, the kernel layer can be configured with a hardware driver, and the driver contained in the kernel layer can be at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc.

[0084] It should be noted that the above examples are only a simple division of the functions of the operating system, and do not constitute a limitation on the specific operating system form of the display device 200 in the embodiments of the present disclosure. According to the function of the display device, the type of the operating system, and other factors, the number and specific type of the layers contained in the operating system can be in other forms.

[0085] Generally, in the process of playing electronic games through a display device, if a user has a demand for querying content related to the game, such as querying game-related strategies or finding specific game content, etc., the user usually needs to use another terminal device, such as a mobile terminal or a computer device, to query, which is time-consuming and seriously affects the efficiency of obtaining information and the user experience.

[0086] It is found through research that the current display device can generally have the functions of interacting with the user and networking. Taking a television device as an example, the television device can be connected to the Internet, so that in the process of playing games through the television device, based on the triggering of user operation, the thumbnail of the video related to the game can be obtained and displayed on the television device, so that the user can know the content of the video related to the game that can be previewed or quickly identified, thereby saving resources and improving efficiency without using other terminal devices. Moreover, by displaying the content of the at least one game video on the display device in the form of a video thumbnail, the user can select at least one game video for playing, which can further meet the individual needs and improve the user experience.

[0087] In some embodiments, the display device can include a display, a memory configured to store computer instructions and data associated with the display device, at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol, and at least one processor connected to the display, the at least one external device interface, and the memory, configured to execute the computer instructions to cause the display device to perform:

[0088] in response to entering a game application control instruction, controlling the display to display a game application interface;

[0089] in response to a control instruction associated with a first button of a remote controller, in the case of entering a game application, determining a first instruction associated with the first button; the first instruction is used to display at least one video thumbnail in a first display area of the display when the display displays the game application interface; the video thumbnail is a thumbnail generated from a video associated with the game application.

[0090] wherein the first button is a button provided on a remote controller capable of interacting with the display device to control the display content of the display device.

[0091] In some embodiments, the first button can be a button specially used to trigger the generation of the first instruction described above, and the button name displayed on the button is not limited, for example, "AI". No matter where the focus of the display device is, when the user presses the first button, the control instruction associated with the first button can be obtained, and the first instruction associated with the first button described above can be determined.

[0092] In some embodiments, the first button can also be an existing button in the remote controller. In some embodiments, a virtual control can be displayed on the game application interface, and the virtual control can be operated by the first button, so as to obtain the control instruction associated with the first button.

[0093] The shape of the virtual control is not limited, for example, circular, square, rectangular, oval, etc., and the color of the virtual control is not limited, as long as it can be distinguished from the preset application interface. Based on actual needs, different transparencies can also be set for the virtual control.

[0094] It can be understood that the first button can also be set in other ways, as long as when the display game application interface is displayed, and the first instruction associated with the first button is received, it can be determined that at least one video thumbnail is displayed in the first display area of the display when the display displays the game application interface, and the video thumbnail is a thumbnail generated by a video associated with the game application.

[0095] The first display area is an area of the display area of the display for displaying the obtained at least one video thumbnail, and the specific position of the first display area is not limited. In some embodiments, it can be an area below the display screen of the display, such as the first display area 501 shown in FIG. 5. It can be understood that in other embodiments, the first display area can also be an area above, left or right of the display screen of the display.

[0096] The video thumbnail displayed in the first display area can be a thumbnail of a frame image of a video associated with the game application. The frame image can be a starting frame, a middle frame or an ending frame of the video. In some embodiments, the video thumbnail displayed in the first display area can also be a preview video formed by thumbnails of multiple frame images of the video. The multiple frame images can be consecutive multiple frame images in the video, or can be discontinuous multiple frame images in the video, but are not limited thereto. Through the displayed video thumbnail, the user can quickly preview and identify the relevant content of the video based on the displayed video thumbnail.

[0097] The number of video thumbnails displayed in the first display area is not limited, and can be determined based on the number of videos associated with the game application, the size of the video thumbnails, and the size of the first display area. The proportion of the display screen of the display occupied by the first display area can be determined in combination with the game experience of watching the game display interface and the clarity of watching the video thumbnails, and the embodiments of the present disclosure do not make specific limitations in this regard.

[0098] In the case where the display device switches from displaying only the game application interface on the display to displaying at least one video thumbnail in the first display area while displaying the game application interface, the control focus of the display device can be on the game application interface, and the control focus can be switched by pressing the direction keys (such as the up, down, left, and right keys) of the remote controller. For example, the control focus can be switched between the game application interface and the video thumbnails by the up and down direction keys, and the control focus can be switched between different video thumbnails by the left and right direction keys. As shown in FIG. 5, when the control focus is on the video thumbnail of video 1, if the right key of the remote controller is received, the control focus can be switched to the video thumbnail of video 2.

[0099] Based on the display device of this embodiment, during the process of displaying the game application interface, the first instruction associated with the first key of the remote controller can be determined based on the control instruction received by the first key of the remote controller, so that the video thumbnail of the video associated with the game application is displayed on the display when the game application interface is displayed. The triggering method is simple and can be triggered by manipulating the first key of the remote controller, so that the video thumbnail associated with the game application can be triggered and displayed simply during the process of playing the game by using the display device, and the efficiency of obtaining the video related to the game process is improved. Moreover, at least one video thumbnail is displayed in the first display area of the display, so that the user can quickly preview and identify the related content of the video based on the displayed video thumbnail, and select the video for playing from the video thumbnail, which can meet the personalized needs of the user and improve the user experience.

[0100] When at least one video thumbnail is displayed in the first display area of the display, the specific display method is not limited.

[0101] In some embodiments, the display device can switch the game application interface to be displayed in the seventh display area of the display, and display at least one video thumbnail in the first display area of the display. The seventh display area does not overlap with the first display area, so that at least one video thumbnail is displayed in the first display area of the display when the game application interface is displayed on the display.

[0102] The seventh display area is a display area of the display, and is used for displaying the game application interface when displaying the at least one video thumbnail, and is not overlapped with the area displaying the at least one video thumbnail.

[0103] The specific position of the seventh display area is not limited, and in some embodiments, can be an area above the display screen of the display, such as the display area above the first display area 501 shown in FIG. 5. It can be understood that in other embodiments, the seventh display area can also be other areas of the display screen of the display, as long as it is not overlapped with the first new urban area.

[0104] Based on this embodiment, by switching the game application interface to be displayed in the seventh display area of the display, the area displaying the at least one video thumbnail is not overlapped with the display area displaying the game application interface, so that all game elements displayed in the game application interface can be viewed, and the need to view the entire game application interface and all game elements can be met.

[0105] In some embodiments, the display area of the game application interface is kept unchanged, a new layer is generated above the layer of the game application interface in the first display area of the display, and the at least one video thumbnail is displayed in the new layer, so that when the display displays the game application interface, the at least one video thumbnail is displayed in the first display area of the display.

[0106] For example, if the display area of the game application interface is kept unchanged and the game application interface is displayed full screen on the display screen, the game application interface is continuously displayed full screen on the display screen.

[0107] Referring to FIG. 5, the game application interface is displayed full screen on the display, and the at least one video thumbnail is displayed in a layer above the layer of the game application interface in the first display area 501 below the display, so that when the at least one video thumbnail can be viewed, the full screen interface of the game can be viewed.

[0108] Based on this embodiment, by superimposing the layer of the video thumbnail on the layer displaying the game application interface and keeping the display area of the game application interface unchanged, the size of the game elements displayed in the game application interface will not change due to the adjustment of the display area.

[0109] In some embodiments, the display device can display the at least one video thumbnail in the first display area in a preset order. The preset order is not limited in specific setting manner, and can include an order from large to small of a play amount of a video corresponding to the video thumbnail, an order from large to small of a relevance of a video corresponding to the video thumbnail to the game application interface, an order from large to small of a favorable degree of a video corresponding to the video thumbnail, an order from recent to remote of an upload time of a video corresponding to the video thumbnail, and the like, but is not limited thereto.

[0110] In some embodiments, when the display device displays the video thumbnail of the at least one game video in the first display area of the display, the display device displays a first control in the first display area of the display; in response to an instruction of operating the first control, the display device keeps the display area of the game application interface unchanged and displays a guide prompt interface in a second display area of the display.

[0111] The first control is a second control for triggering the display of the guide prompt interface, and the size and style of the first control are not limited. For example, the first control in the first display area in some embodiments is shown in FIG. 5 as 5011. The size of the first control can be set to be the same as the size of the video thumbnail, and after the first control 5011 is displayed in the first display area 501, the video thumbnails are arranged in the preset order, i.e., the first control is located at the front of the arrangement order of the video thumbnails.

[0112] The first control can include an image mark and corresponding text content, such as image 1 and text prompt information "game intelligent body" shown in FIG. 5, to prompt that the guide prompt interface can be opened by clicking the first control. It can be understood that in other embodiments, the image mark and text content of the first control can also be set in other manners.

[0113] The guide prompt interface is an interface for prompting how to obtain the video related to the game application, and can include prompt information of functions that can be provided, such as "I am your prompt assistant, skilled in strategy and skills, ask me anytime you have a question" shown in FIG. 6, and can also include some examples of questions, such as "try to say to me: how to refine ****'s refining elixir" and "where can **** find AA" shown in FIG. 6, but is not limited thereto.

[0114] The second display area is an area of the display area of the display for displaying the guide prompt interface, and the specific position of the second display area is not limited. In some embodiments, the second display area can be an area on the right side of the display screen of the display, such as 601 shown in FIG. 6. It can be understood that in other embodiments, the second display area can also be an area on the top, top, or left and right side of the display screen of the display.

[0115] The second display area can have an overlapping area with the display area of the game application interface. Generally, the game application interface can be displayed in full screen on the display screen of the display. The display guide prompt interface of the second display area is superimposed on the layer of the game application interface.

[0116] In some embodiments, when the display device displays at least one video thumbnail in the first display area of the display, the game application interface is switched to be displayed in the seventh display area of the display. After the game application interface is switched to be displayed in full screen, the display device displays the guide prompt interface in the second display area of the display.

[0117] In some embodiments, the layer of the guide prompt interface can be set to have a transparency, so that the user can view the prompt information of the guide prompt interface and watch the overall picture of the game application interface. The size of the transparency is not limited, as long as the part of the game application interface covered by the guide prompt interface can be watched.

[0118] Therefore, by setting the first display area to display the first control, and switching to display the guide prompt interface when the game application interface is displayed in response to the operation of the first control, the user can be conveniently prompted on how to obtain the game video related to the game process, the operation is convenient, and the user experience is improved.

[0119] In some embodiments, as shown in FIG. 5, when the display device displays at least one video thumbnail of the game video in the first display area of the display, the display device displays a prompt component 5012. The prompt component 5012 can be only an image and cannot start any application, but can prompt that the AI view can be automatically performed to obtain the video information through the prompt component 5012, or prompt that the video thumbnails corresponding to the images on the right side of the prompt component 5012, such as video 1 and video 2, are results obtained by automatic identification and search through AI technology. In some embodiments, the prompt component 5012 can also be set as a second control or a trigger interface of related content. By clicking the prompt component 5012, the associated page can be entered. The specific page content and type are not specifically displayed in the embodiments of the present disclosure.

[0120] In some embodiments, when the display device displays the guide prompt interface, the display device displays text information and a preset number of video thumbnails in the sixth display area of the display in response to the received voice input information. The text information is text information associated with the voice input information and the interface element of the game application interface, and the video thumbnail is a thumbnail generated by a video associated with the voice input information and the game application.

[0121] The preset number of specific numbers is not limited, and the preset number can be 2 in some embodiments. Taking the preset number 2 as an example, a schematic diagram of displaying the text information and the preset number of video thumbnails in the sixth display area is shown as 701 in FIG. 7.

[0122] The sixth display area is a display area of the display, which is used to simultaneously display the text information and the preset number of video thumbnails. The specific position of the sixth display area is not limited, and in some embodiments, it can be an area on the right of the display screen of the display, as shown in FIG. 7. It can be understood that in other embodiments, the sixth display area can also be an area in other directions of the display screen of the display.

[0123] The sixth display area can have an overlapping area with the display area of the game application interface, that is, the game application interface can be displayed full screen on the display screen of the display, and the layer of the sixth display area displaying the text information and the preset number of video thumbnails is superimposed on the layer where the game application interface is located.

[0124] Therefore, when the guide prompt interface is displayed, based on the voice input information, the text information associated with the interface element of the game application interface and the thumbnail of the video generated for the voice input information and the game application can be displayed in the sixth display area. Combined with the voice input information and the interface element of the game application interface, the demand can be more fully known, and the information of the text information and the preset number of video thumbnails corresponding to the video is also more accurate, thereby providing a better game experience.

[0125] In some embodiments, the display device can keep the display area of the display displaying the game application interface unchanged in response to the selection operation of selecting one of the video thumbnails, generate a video player in the third display area of the display, and play the video associated with the video thumbnail corresponding to the selection operation through the video player.

[0126] It can be understood that taking the video associated with the video thumbnail as stored in the server as an example, in some embodiments, when the selection instruction of selecting one of the video thumbnails is received, a video acquisition instruction is sent to the server to acquire the video associated with the video thumbnail corresponding to the selection operation from the server.

[0127] In the process of playing the video associated with the video thumbnail in the third display area, the video can be played by calling the built-in player of the display device, or the video can be played by calling the player of a third party through an interface. The specific display of the embodiments of the present disclosure is not limited.

[0128] Based on this embodiment, when a certain video thumbnail is selected, the video associated with the video thumbnail can be displayed in a third display area that has an overlapping area with the display area of the game application interface, so that the specific video can be viewed without affecting the display size of the game application interface, thereby improving the game experience.

[0129] In some embodiments, the display device can update the game application interface to a fifth display area of the display in response to a selection operation of selecting one of the video thumbnails, so that the game application interface is displayed in the fifth display area of the display; generate a video player in a fourth display area of the display, and play the video associated with the video thumbnail corresponding to the selection operation through the video player; the fourth display area does not overlap with the fifth display area.

[0130] Based on this embodiment, when a certain video thumbnail is selected, the video associated with the video thumbnail can be displayed in a fourth display area that does not overlap with the display area of the game application interface, so that the user can view the overall picture of the game application interface and the overall picture of the video, and the user's demand for split-screen or zoned display can be met, thereby improving the game experience.

[0131] The selection operation is an operation for indicating the selection of opening a game video corresponding to a video thumbnail. The selection operation can be obtained in various possible ways.

[0132] For example, in the case where the display device can be controlled by the control device 100 (such as a remote controller), when the control focus of the display device is located in a certain video thumbnail, if a determination instruction is sent through the control device 100, it is determined that the selection operation of selecting the video thumbnail is received.

[0133] In the case where the display device is provided with a sound collector, voice input information can be obtained through the sound collector, and it is determined that the selection operation of selecting a certain video thumbnail is received based on the voice input information. Taking FIG. 5 and FIG. 7 as examples, when voice input information such as "open the second video", "open the second", "watch the second" and the like is received, it is considered that the selection operation of selecting a certain video thumbnail is received, and the display interface shown in FIG. 8 or FIG. 9 is switched to.

[0134] The third display area is an area for displaying game video content when the game video content is superimposed on the game application interface for display, and the specific position of the third display area is not limited. In some embodiments, it can be an area at the upper right of the display screen of the display, such as the third display area 801 shown in FIG. 8, which is located in an area at the upper right of the display screen of the display.

[0135] The fourth display area is an area for displaying the game video content when the game video content and the game application interface are not overlapped, and the fifth display area is an area for displaying the game application interface when the game video content and the game application interface are not overlapped. The specific positions of the fourth display area and the fifth display area are not limited, and in some embodiments, they can be the areas on the left and right sides of the display screen of the display respectively, as shown in FIG. 9, the fourth display area 901 and the fifth display area 902 are located in the areas on the left and right sides of the display screen of the display respectively.

[0136] When the selection operation is received, it can be determined whether the game video content corresponding to the video thumbnail of the selection instruction is played in the third display area, that is, switched to the display interface as shown in FIG. 8, or the game video content corresponding to the video thumbnail of the selection instruction is played in the fourth display area, and the game application interface is played in the fifth display area, that is, switched to the display interface as shown in FIG. 9.

[0137] In some embodiments, the display device can provide a play setting item for setting the playing mode of the video associated with the corresponding video thumbnail when the selection operation is received. For example, the attribute value of the play setting item can be read when the selection operation is received. When the attribute value of the play setting item is a first preset value, the video associated with the video thumbnail corresponding to the selection operation is played in the third display area, and when the attribute value of the play setting item is a second preset value, the video content associated with the video thumbnail corresponding to the selection operation is played in the fourth display area.

[0138] In some embodiments, the display device can record the last playing mode of the video, and use the last playing mode of the video as the playing mode of the video when the selection operation is received. For example, if the last time the video is played, the video is displayed in the third display area and is overlaid on the game application interface, the video associated with the video thumbnail corresponding to the selection operation is played in the third display area. If the last time the video is played, the video is displayed in the fourth display area and is not overlapped with the game application interface, the video associated with the video thumbnail corresponding to the selection operation is played in the fourth display area, and the game application interface is displayed in the fifth display area.

[0139] In some embodiments, during the process of playing the video through the video player generated in the third display area of the display, if a game display mode switching instruction is received: the display area of the video player is adjusted to the fourth display area of the display, so that the video player plays the video in the fourth display area; and the game application interface is updated from the display area of the display to the fifth display area of the display, so that the game application interface is displayed in the fifth display area of the display; wherein the fourth display area and the fifth display area are not overlapped.

[0140] The game display mode switching instruction is an instruction for indicating that the relationship between the display area playing the video and the display area of the game application interface needs to be adjusted. If the game display mode switching instruction is received during the process of playing the video by the video player generated in the third display area of the display, it indicates that the display area playing the video and the display area of the game application interface need to be adjusted not to overlap. At this time, the game display mode switching instruction can also be referred to as a first game display mode switching instruction. Therefore, the game video content is played in the fourth display area, and the game application interface is displayed in the fifth display area, and the fourth display area and the fifth display area do not overlap.

[0141] Therefore, by means of the game display mode switching instruction, the relationship between the display area playing the video and the display area of the game application interface can be switched, different ways can be used to play the video, and different playing requirements can be met.

[0142] The manner of receiving the first game display mode switching instruction is not limited, and some implementation manners are taken as examples for illustration.

[0143] In some embodiments, the display device can display at least one second control in a preset range of the display range of the video player during the process of playing the video associated with the video thumbnail corresponding to the selection operation by the video player, and the second control includes a display mode switching second control.

[0144] In response to the operation of the display mode switching second control, the game display mode switching instruction is received.

[0145] The preset range of the display range of the video player is a range of the area for displaying the second control for controlling the playing of the video. The preset range can be determined based on the interface size of the video player or the boundary of the third display area. As shown in FIG. 8, the preset range can include an area 802 adjacent to the lower boundary of the third display area and not overlapping with the third display area. In other embodiments, the preset range 802 of the display range of the video player can also be a part of the third display area 801. It can be understood that in other embodiments, the preset range of the display range of the video player can also be set differently, for example, an area adjacent to the lower boundary of the third display area and overlapping with the third display area, or an area adjacent to the upper boundary, the left boundary or the right boundary of the third display area and overlapping or not overlapping with the third display area.

[0146] The specific display name of the display mode switching second control can be set differently. For example, as shown in the example in FIG. 8, the display name of the display mode switching second control 8021 can be set as “split screen”.

[0147] Therefore, by setting the display mode switching second control, it can be intuitively and conveniently known how to realize the switching of the display mode, which helps to improve the efficiency of switching and user experience.

[0148] In some embodiments, the display device can receive voice input information through the sound collector, and determine whether a game display mode switching instruction is received based on the received voice input information.

[0149] In the case that the display device is provided with a sound collector, the voice input information can be obtained through the sound collector, and it is determined whether a game display mode switching instruction is received based on the voice input information. For example, when the received voice input information includes "split screen" or similar voice input information, it is considered that a game display mode switching instruction is received.

[0150] In some embodiments, the display device can collect image information through the image collector, and determine whether a game display mode switching instruction is received based on the image information.

[0151] In the case that the display device is provided with an image collector, the first preset image information corresponding to the first game display mode switching instruction can be stored in the memory of the display device. The content of the first preset image information is not limited, for example, it can be preset gesture information. The image collector is used to collect image information, the obtained image information is analyzed and processed to obtain recognized image information, and the recognized image information is compared with the first preset image information. If they are the same (for example, the probability of being the same is greater than a preset probability threshold), it is determined that a game display mode switching instruction is received.

[0152] Referring to FIG. 8, in some embodiments, the second control displayed in the preset range of the third display area can also include a play control second control. The display device can perform play control on the game video content in response to a play control instruction received through the play control component.

[0153] The play control component is a second control for controlling the play of the game video content, for example, a second control for controlling the start of playing the video, a second control for controlling the pause of playing the video. In general, the start of playing the video and the pause of playing the video can be realized through the same second control, such as the second control 8022 shown in FIG. 8. Among them, the second control name displayed by the second control 8022 is dynamically adjusted based on the different play states of the video. For example, as shown in FIG. 8, during the process of playing the video, the second control name displayed by the second control 8022 is "pause", and during the process of pausing the video, the second control name displayed by the second control 8022 can be adjusted to "play".

[0154] In some embodiments, the playback control component further includes a backward playback second control (not shown in FIG. 8) for controlling a backward playback of a first playback duration, and a forward playback second control (not shown in FIG. 8) for controlling a forward playback of a second playback duration. When the user clicks the backward playback second control, the processor can receive an instruction for playing back the first playback duration, and the backward playback of the video can be realized. When the user clicks the forward playback second control, the processor can receive an instruction for playing forward the second playback duration, and the forward playback of the video can be realized. The display manner of the backward playback second control and the forward playback second control is not limited, and in some embodiments, can be set based on the first playback duration and the second playback duration. For example, when the first playback duration is 15 seconds and the second playback duration is 15 seconds, the backward playback second control and the forward playback second control can be displayed as “-15s” and “15s” respectively, but are not limited thereto.

[0155] In some embodiments, as shown in FIG. 8, the playback control component can further include an audio control component, and the display device can control the playback of the audio corresponding to the game video content in response to an audio control instruction received through the audio control component.

[0156] The content of the playback control of the audio is not limited, for example, can be to control the playback or shutdown of the audio associated with the video, can be to control the sound channel of the audio of the video, or can be to control whether to play the audio of the video or the audio corresponding to the game application interface. Based on different playback control contents, the display content of the audio control component can be different. For example, when the control is to control whether to play the audio of the video or the audio corresponding to the game application interface, the display content of the audio control component in some embodiments can be “sound switching” as shown in FIG. 8, but is not limited thereto.

[0157] In some embodiments, as shown in FIG. 9, the second control further includes a hiding control component 8024, and the display device can display the game application interface on the full screen of the display in response to the operation of the hiding control component, control the playback window of the video player to be invisible on the display interface of the display, and display a playback window display manner prompt information at a preset edge position of the display screen of the display.

[0158] The hiding instruction is an instruction for indicating to hide the video played by the player or to hide the player, and therefore, after receiving the hiding instruction, the playback window of the video player can be controlled to be invisible on the display interface of the display, and the game application interface can be displayed on the full screen of the display.

[0159] The manner in which the playback window of the video player is invisible on the display interface of the display is not limited, as long as the content of the played video cannot be viewed within the screen range of the display screen.

[0160] By setting the hidden control component, the playing window of the video player can be invisible on the display interface of the display when the user does not need to watch the video or only needs to listen to the sound of the video. By displaying the playing window display mode prompt information at the preset edge position of the display screen of the display, the user can switch to the playing window of the video player when needed, thereby meeting different needs of the user.

[0161] The specific and detailed playing window display mode prompt information is not limited. For example, after receiving the hiding instruction, the display interface of the display can be as shown in FIG. 10. As shown in FIG. 10, the playing window display mode prompt information “press the left key of the remote control to pull up the strategy” is arranged at the upper right of the display screen of the display. Based on the playing window display mode prompt information shown in FIG. 10, when the user presses the left key of the remote control, the display interface shown in FIG. 8 can be switched back. It can be understood that based on different design needs, other ways can also be used to trigger the switching back to the display interface shown in FIG. 8. At this time, based on the difference in the triggering way, the prompt information displayed in FIG. 10 can also be different.

[0162] In some embodiments, in combination with FIG. 8, the playing control component can further include a closing component 8025. The display device can close the player in response to a closing instruction received through the closing component, and display the game application interface on the full screen of the display.

[0163] The way of closing the playing of the game video content is not limited. For example, the player playing the game video content is closed to achieve the closing of the playing of the game video content.

[0164] In some embodiments, during the playing of the game video content in the fourth display area, when a game display mode switching instruction is received: the display area of the video player is adjusted to be the third display area of the display, so that the video player plays the video in the third display area; and the display area of the game application interface is updated to be the full screen of the display, so that the game application interface is displayed on the full screen of the display.

[0165] During the playing of the game video content in the fourth display area, when a game display mode switching instruction is received, it indicates that the display area playing the video needs to be adjusted to be non-overlapping with the display area of the game application interface. At this time, the game display mode switching instruction can also be called a second game display mode switching instruction, which is an instruction for indicating that the video is displayed on the upper layer of the game application interface. Therefore, the game video content is switched to be played in the third display area, and the third display area has an overlapping area with the display area of the game application interface.

[0166] The manner of receiving the second game display mode switching instruction can be similar to the manner of receiving the first game display mode switching instruction.

[0167] In some embodiments, as shown in FIG. 9, when playing the video in the fourth display area, at least one second control is displayed in the preset range of the fourth display area, and the second control includes a second display mode switching second control 9011, and a second game display mode switching instruction is received through the second display mode switching second control 9011.

[0168] In some embodiments, the first display mode switching second control 8011 and the second display mode switching second control 9011 can be different second controls, or can be one second control in the preset range of the player (i.e., the preset range of the third display area or the preset range of the fourth display area) for playing the game video content. When an instruction is received through the second control in the case that the game video content is displayed on the upper layer of the game application interface, it can be considered that a first game display mode switching instruction is received. When an instruction is received through the second control in the case that the game video content is not overlapped with the display area of the game application interface, it can be considered that a second game display mode switching instruction is received.

[0169] In some embodiments, voice input information is received through a sound collector, and it is determined whether a second game display mode switching instruction is received based on the received voice input information.

[0170] In some embodiments, image information is collected through an image collector, and it is determined whether a second game display mode switching instruction is received based on the image information.

[0171] In some embodiments, as shown in FIG. 9, at least one second control displayed in the preset range of the fourth display area 901 can also include a play control component (such as the play start-stop control component 9002, the backward play second control 9003, and the forward play second control 9004 shown in FIG. 9), an audio control component 9005, a hiding component 9006, and a closing component 9007. The specific manner is the same as in the above embodiments, and will not be described here. It can be understood that, in the example shown in FIG. 9, the step of forward play and the step of backward play are 15 seconds. In the case of using other steps of forward play and backward play, the values displayed by the backward play second control 9003 and the forward play second control 9004 can be different.

[0172] It can be understood that, in some embodiments, the second control in the preset range of the third display area and the second control in the preset range of the fourth display area can both be second controls provided on the player, but are not limited thereto.

[0173] The manner of obtaining information of the video associated with the game application is not limited in the embodiments of the present disclosure.

[0174] In some embodiments, a frame image of a currently displayed game application interface is obtained, a game interface element in the frame image is identified, the game interface element includes at least one of a background, a prop, and a character; game name and game level information are obtained based on a search in a game knowledge graph based on the game interface element, the nodes of the game knowledge graph at least include the game interface element, the game name, and the level; a connection line of the game knowledge graph indicates that the nodes at both ends of the connection line have an association relationship; a query keyword is determined based on the game interface element, the game name, and the game level information; a query request is sent to a server based on the determined query keyword, and at least one video thumbnail returned by the server based on the query request is obtained.

[0175] The manner of obtaining the frame image of the currently displayed game application interface is not limited, for example, the frame image of the currently displayed game application interface can be obtained by taking a screenshot of the current display interface of the display. The frame image of the currently displayed game application interface can also be obtained by obtaining a video stream transmitted to the display based on the video stream.

[0176] When the frame image of the currently displayed game application interface is subjected to image recognition to obtain image information, the image recognition can be performed by an image recognition model. The image recognition model can be pre-trained, and the image recognition model trained by the server or other devices can be deployed in the display device for image recognition.

[0177] The manner of training the image recognition model is not limited, and in some embodiments, the manner of training the image recognition model can adopt the following flow:

[0178] First, game screenshot pictures of different games and different levels are collected, and after image preprocessing of the game screenshot pictures, the game interface elements such as the game name, the level name, the character name involved in the picture, and the prop name (such as props) involved in the picture are labeled in each picture, thereby obtaining an initial sample data set. The manner of image preprocessing is not limited, for example, it can be grayscale, denoising, binarization, edge detection, but is not limited thereto.

[0179] The initial sample data set is subjected to data enhancement processing, and the manner of data enhancement is not limited, for example, image rotation, image scaling, image cropping, and the like, to obtain a sample data set. By performing data enhancement processing, the generalization ability of the obtained image recognition model can be improved.

[0180] Secondly, after obtaining the sample data set, the sample data set is divided into a training set, a validation set and a test set, for example, 70% of the sample data constitutes the training set, 15% of the sample data constitutes the validation set, and the remaining 15% of the sample data constitutes the test set to evaluate the generalization ability of the model.

[0181] After the division of the sample data set, the model training process can be performed. In the model training process, the image recognition model is trained by the training set, and the performance of the image recognition model is evaluated using the validation set during or after the training process. The trained image recognition model is finally tested using the test set to ensure the performance of the model in actual application. It can be understood that in the training process of the image recognition model, the image recognition model recognizes the information of the background name, prop name and character name in the sample data, and compares it with the background name, prop name and character name annotated in the sample data to calculate the training loss, and adjusts the image recognition model combined with the calculated training loss to realize the update of the image recognition model. After the model training is completed, the last updated image recognition model is used as the trained image recognition model.

[0182] The specific model architecture of the image recognition model is not limited, for example, it can be a convolutional neural network (CNN), a YOLOv8 network architecture, etc., but is not limited thereto.

[0183] In some embodiments, the image recognition model can include an image classification sub-model and a target detection sub-model.

[0184] The image classification sub-model can be a deep learning model such as a convolutional neural network (CNN), which classifies game screenshots to identify specific game names and game levels.

[0185] The target detection sub-model is used to identify game characters, level elements, game props, and boss targets in game screenshots and locate the specific positions of the targets. The detection method is not limited, for example, YOLO algorithm or Faster R-CNN algorithm can be used for target detection, but is not limited thereto.

[0186] During the training of the image recognition model, the image classification sub-model and the target detection sub-model can be trained simultaneously, and the first loss of the image classification sub-model and the second loss of the target detection sub-model are combined to determine the training loss of the image recognition model, for example, the training loss can be the weighted sum of the first loss and the second loss.

[0187] The knowledge graph is a graph-based data structure used to describe concepts, entities, and their relationships in the real world. It represents information on the Internet in a way that is closer to human cognition, providing better organization, management, and understanding of massive amounts of information. The game knowledge graph is a knowledge graph for information related to various games. It can include information such as game names, game levels, game backgrounds, game props, and character names. The connection lines in the game knowledge graph represent the association between the nodes at both ends of the connection line. A partial graph structure of a game knowledge graph in one specific example is shown in FIG. 11.

[0188] The manner of determining the query keyword based on the game interface element, the game name, and the game level information is not limited. In some embodiments, the recognized game interface element, game name, and game level can all be used as query keywords. In some embodiments, a language model can be used to identify keywords from the game interface element, game name, and game level information. Some possible language models include large language models such as prompt, but are not limited thereto.

[0189] Based on the determined query keyword, a query request can be sent to the server to obtain information of at least one game video returned by the server based on the query request.

[0190] Thus, the game interface element in the frame image of the game application interface can be combined to determine the query keyword based on the game knowledge graph to query and obtain at least one video thumbnail, which can improve the accuracy of the queried and obtained video thumbnail.

[0191] In the embodiments of the present disclosure, the manner of obtaining information of a video associated with the voice input information and the game application is not limited.

[0192] In some embodiments, in response to the voice input information, a frame image of a currently displayed game application interface is obtained, a game interface element in the frame image is recognized, the game interface element includes at least one of a background, a prop, and a character; game name and game level information are obtained based on the game interface element in a game knowledge graph, the nodes of the game knowledge graph at least include the game interface element, the game name, and the level; the connection lines of the game knowledge graph represent the association between the nodes at both ends of the connection line; a query keyword is determined based on the voice input information, the game name, and the game level information; a query request is sent to a server based on the determined query keyword, and text information and game video information returned by the server based on the query request are obtained.

[0193] The manner of obtaining the frame image of the currently displayed game application interface, the manner of image recognition, and the game knowledge graph can be the same as in the above embodiments, and will not be described again here.

[0194] In the determination of the query keyword based on the voice input information, the game name, and the game level information, the determination can be performed by a language model, for example, inputting the text content corresponding to the voice input information, the game name, and the game level information into the language model, and outputting the query keyword by the language model. The language model can be the same model as the language model in the above embodiments, or can be a different model.

[0195] Thus, the query keyword can be determined based on the game interface elements in the frame image of the game application interface, the received voice input information, and the game knowledge graph, to query the text information and the at least one video thumbnail, which can improve the accuracy of the queried video thumbnail.

[0196] In some embodiments, the display device obtains the voice input information, obtains the received voice input information and the frame image of the displayed game application interface, and obtains the corresponding text information and the information of the preset number of game videos, as shown in FIG. 12.

[0197] In combination with FIG. 12, on the one hand, the voice input information obtained based on the user voice is obtained. When the voice input information is obtained, the frame image of the currently displayed game application interface is obtained. After image preprocessing of the frame image of the currently displayed game application interface, the character name, the prop, the background, and other game interface elements in the frame image of the game application interface are recognized by the image recognition model, and the corresponding game name and level are retrieved in the game knowledge graph through these game interface elements. Then, the voice input information, the game name, the level, and other information are input into the language model, the query keyword is extracted by the language model, and the query keyword is used for web search and video search respectively.

[0198] When the web search result is obtained, a large model can be used to summarize the web search result to obtain the final text information. The specific way of summarizing is not limited, for example, the HTML2Text tool is used to convert the web page into text information, the text information of the web page is summarized by a language model (such as a prompt model) to obtain the final text information, which can be executed by a server or a display device.

[0199] When the video search result is obtained, the preset number of videos (for example, 2 videos) in the video search result can be used as the obtained game video information.

[0200] After obtaining the text information and the game video information, the text information and the video thumbnail of the video are displayed in combination, as shown in FIG. 7.

[0201] In some embodiments, the display device may include: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least the Bluetooth protocol; and at least one processor connected to the display, the at least one external device interface, and the memory, configured to execute computer instructions to cause the display device to perform: in response to a control instruction for entering a game application, controlling the display to display a game application interface; and in response to a control instruction associated with a first button on a remote control, determining a first instruction associated with the first button when entering a game application, the first instruction being used to display text information and a preset number of video thumbnails in a sixth display area of ​​the display when the display shows the game application interface, the text information being text information associated with interface elements of the game application interface, and the video thumbnails being thumbnails generated from videos associated with the game application.

[0202] Based on this embodiment, the display device, during the process of displaying the game application interface, can display text information associated with the game application and thumbnails generated from videos associated with the game application in the sixth display area based on the control command associated with the first button of the remote control received. This allows for a more comprehensive understanding of the requirements, and the text information and video information corresponding to a preset number of video thumbnails obtained on this basis are also more accurate, providing a better gaming experience.

[0203] Other implementations of the display device may be the same as those in the above embodiments.

[0204] In some embodiments, the display device may include: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least the Bluetooth protocol; and at least one processor connected to the display, the at least one external device interface, and the memory, configured to execute computer instructions to cause the display device to perform: in response to a control instruction to enter a game application, controlling the display to display a game application interface; and, in the case of displaying the game application interface, in response to received voice input information, displaying at least one video thumbnail in a first display area of ​​the display while the display is showing the game application interface; the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

[0205] Based on the display device of the embodiment, in the process of displaying the game application interface, the video thumbnail of the video associated with the game application can be triggered and displayed based on the received voice input information when the display displays the game application interface, the triggering manner is simple, the triggering manner is simple, so that the video thumbnail associated with the game application can be triggered and displayed simply in the process of playing the game through the display device, and the efficiency of obtaining the video related to the game process is improved. Moreover, by displaying at least one video thumbnail in the first display area of the display, the user can quickly preview and identify the related content of the video based on the displayed video thumbnail, and select the video for playing from the video, which can meet the personalized needs of the user and improve the user experience.

[0206] In some embodiments, the display device can include a display, a memory configured to store computer instructions and data associated with the display device, at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol, and at least one processor connected to the display, the at least one external device interface, and the memory, configured to execute the computer instructions to cause the display device to perform: in response to entering a control instruction of a game application, controlling the display to display a game application interface; in the case of displaying the game application interface, in response to received voice input information, displaying text information and a preset number of video thumbnails in a sixth display area of the display when the display displays the game application interface, the text information being text information associated with the voice input information and an interface element of the game application interface, and the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

[0207] Based on the display device of the embodiment, in the process of displaying the game application interface, the video thumbnail of the video associated with the game application can be triggered and displayed based on the received voice input information when the display displays the game application interface, the triggering manner is simple, the triggering manner is simple, so that the video thumbnail associated with the game application can be triggered and displayed simply in the process of playing the game through the display device, and the efficiency of obtaining the video related to the game process is improved. Moreover, by displaying at least one video thumbnail in the first display area of the display, the user can quickly preview and identify the related content of the video based on the displayed video thumbnail, and select the video for playing from the video, which can meet the personalized needs of the user and improve the user experience.

[0208] In some embodiments, the display device can include a display, a memory configured to store computer instructions and data associated with the display device, at least one external device interface for communicating with at least one external device according to a communication protocol, the communication protocol including at least a Bluetooth protocol, and at least one processor connected to the display, the at least one external device interface, and the memory, configured to execute the computer instructions to cause the display device to perform: in response to entering a control instruction of a game application, controlling the display to display a game application interface; in the case of displaying the game application interface, in response to received voice input information, displaying text information and a preset number of video thumbnails in a sixth display area of the display when the display displays the game application interface, the text information being text information associated with the voice input information and an interface element of the game application interface, and the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

[0209] In some embodiments, the display device can comprise: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol comprising at least a Bluetooth protocol; and at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform: in response to a game application starting instruction, loading game application data, and controlling the display to display a game application interface; and when the game application data is loaded and the display displays the game application interface, displaying at least one video thumbnail in a first display area of the display, the video thumbnail being a thumbnail generated from a video associated with voice input information and the game application.

[0210] According to the display device of the embodiments, after loading the game application data and controlling the display to display the game application interface in response to the game application starting instruction, the display device can display the video thumbnail of the video associated with the game application when the display displays the game application interface, which can be automatically triggered and improve the efficiency. Moreover, by displaying the at least one video thumbnail in the first display area of the display, the user can quickly preview and identify the relevant content of the video based on the displayed video thumbnail, and select the video for playing, which can meet the personalized needs of the user and improve the user experience.

[0211] According to the display device of the embodiments, other implementation manners of the display device can be the same as those of the display device in the above embodiments, which will not be described herein.

[0212] The manner of triggering the game application starting instruction and the source of the loaded game application data are not limited.

[0213] In some embodiments, the display device can determine that the game application starting instruction is received after the display device is connected to the external device. It can be understood that the display device can also trigger the game application starting instruction by other manners when the display device is connected to the external device. In this case, the display device can load the game application data stored in the connected external device.

[0214] The manner of connecting the display device to the external device is not limited, for example, the display device can be connected to the external device through a preset device interface, and the type of the preset device interface is not limited, which can include but is not limited to an HDMI interface, a CVBS interface, a component interface, a USB interface, etc., as long as the display device can be connected to the external device through the preset device interface, and the display device can obtain the game application data of the external device connected to the preset device interface, and display the game application interface based on the game application data to implement the game application of the external device connected to the preset device interface.

[0215] In some embodiments, the game application starting instruction can be a game application starting instruction sent by a remote controller of the display device or the like, and the game application starting instruction is considered to be triggered when a selection operation of selecting a game application provided by the display device is received. At this time, the game application data stored in the memory of the display device is loaded.

[0216] Based on the above examples of the display device, the relevant interaction flowchart in the processing of the display device is shown in FIG. 13, which is described by taking an example of a user clicking an AI button on a control device (such as a remote controller) to send a preset button instruction to trigger the application of the game agent; as shown in FIG. 13, it can include:

[0217] S1301, the user clicks a first button on the remote controller;

[0218] S1302, the remote controller sends a signal of the first button to the display device;

[0219] After the user clicks the first button on the remote controller, the remote controller transmits the signal of clicking the first button to the display device.

[0220] S1303, the display device starts a television screenshot and calls a game agent;

[0221] S1304, the display device uploads the screenshot and calls the game agent;

[0222] The display device receives the signal of clicking the first button, obtains the displayed game application interface, for example, takes a screenshot of the current game application interface and uploads it, and calls the game agent service in the display device.

[0223] S1305, the agent executes game agent logic;

[0224] S1306, the agent returns relevant video content to the display device;

[0225] The game agent determines that it is under a game interface according to the current scene parameters of the display device, executes the game agent logic, for example, based on the structure shown in FIG. 12, queries at least one game video information matched with the displayed game application interface from the server, and provides the obtained result to the display device, and the display device displays the result, for example, displays the interface shown in FIG. 5. The type of the current scene parameters of the display device is not limited, for example, it can only include the parameters of the current display interface image, or it can include the relevant running parameters of the display device. The way of determining whether it is under a game interface according to the current scene parameters of the display device is not limited, and it can be based on any way that can identify whether it is under a game interface.

[0226] S1308, the user issues a voice instruction to play a video;

[0227] S1309, the instruction is collected and processed, and the voice instruction is sent to the display device;

[0228] S1310, the display device performs voice recognition on the voice instruction;

[0229] S1311, the display device sends the voice recognition text to the intelligent agent;

[0230] S1312, the intelligent agent performs semantic understanding on the voice recognition text;

[0231] S1313, the intelligent agent returns the play instruction to the reality device;

[0232] S1314, the display device performs floating window play based on the play instruction;

[0233] When the interface as shown in the foregoing Figure 5 is displayed, the user can trigger the execution of the video play by clicking or voice instruction through the remote control device, and the video can be played in the form of floating window during the video play, for example, the play interface shown in the foregoing Figure 8.

[0234] S1315, the user issues a voice instruction of the video display type;

[0235] S1316, the instruction is collected and processed, and the voice instruction is sent to the display device;

[0236] S1317, the display device performs voice recognition on the voice instruction;

[0237] S1318, the display device sends the voice recognition text to the intelligent agent;

[0238] S1319, the intelligent agent performs semantic understanding on the voice recognition text;

[0239] S1320, the intelligent agent returns the control instruction to the reality device;

[0240] S1321, the display device displays the video according to the display type indicated by the control instruction;

[0241] During the video play, the user can control the full screen, split screen, and small window play of the video by clicking or voice instruction through the remote control device.

[0242] In some embodiments, the display device can also execute the flow as shown in FIG. 14, so that the display device can accurately respond to the shutdown instruction input by the user's voice during the video playback process, and display prompt information including the information recorded by the matter record function of the display device on the first display area of the display of the display device during the execution of the shutdown instruction. Instead of inserting irrelevant content during the long shutdown waiting time in the prior art, the user can understand the information recorded by the matter record function of the display device based on the prompt information displayed on the display after the shutdown instruction input by the voice, while improving the diversification of the information displayed by the display device during the shutdown process, improving the intelligent level of the voice interaction system, and achieving more personalized and customized information recommendation for the user.

[0243] Specifically, as shown in FIG. 14, it can include:

[0244] S1401, in response to the control instruction of video playback, controlling the display to display the video playback interface.

[0245] S1402, in response to the shutdown instruction input by the voice, displaying the text information input by the voice on the display while displaying the video playback interface, and displaying prompt information in the first target area of the display, the prompt information at least including matter information. The matter information at least includes the information recorded by the matter record function of the display device.

[0246] S1403, when the prompt information displayed in the first target area of the display reaches a preset time, executing the shutdown operation.

[0247] Wherein, the display device can be a television device.

[0248] Wherein, the definition of shutdown refers to the process of switching the television device from the working state to the non-working state, that is, stopping the reception, processing and display of television signals, and closing the operation of all main electronic elements and components inside the television, in order to reduce energy consumption and prolong the service life of the television. This process is usually realized through the shutdown button on the remote control, the physical switch button on the television body, or in some smart televisions, through the shutdown option on the operating system interface.

[0249] Television shutdown is not only to close the screen display of the television, but more importantly, to ensure that all systems inside the television stop running safely and orderly. This includes closing the image processing system, audio processing system, signal receiving and decoding system, etc., and ensuring that the power management system inside the television can correctly cut off or reduce the power supply to the minimum level.

[0250] Unlike the TV standby or sleep state, the TV shutdown is to completely stop all functions of the TV set, which will not consume any power (except for the possible small standby power consumption, which depends on the design and manufacturing standards of the TV set) and will not cause any potential damage to the TV set.

[0251] The control instruction of the video playing can be a control instruction of controlling the display device to play a certain video, for example, a control instruction of controlling the display device to play a certain video segment, or a control instruction of controlling the display device to play a certain TV channel.

[0252] The video playing interface can be an interface of the display 260 playing the video. For the convenience of understanding of those skilled in the art, FIG. 15 provides a schematic diagram of a video playing interface, in which 1510 is a video playing interface. As shown in FIG. 15, the display device can present a picture of a video currently being played through the video playing interface. The video being played by the video playing interface is the video specified by the control instruction of the video playing.

[0253] In the process of the display device displaying the video playing interface through the display 260, the display device can receive a shutdown instruction input by the user voice; specifically, the user can press the voice control button in the control device 100 of the display device, and read out the voice instruction of "shutdown", to realize the voice input of the shutdown instruction to the display device.

[0254] After the display device receives the voice input shutdown instruction, the display device can display the text information input by the voice and display the prompt information in the first target area of the display 260 in response to the voice input shutdown instruction.

[0255] The prompt information at least includes the matter information. The matter information at least includes the information recorded by the matter recording function of the display device. The matter recording function can be a memo application of the display device. For the convenience of understanding of those skilled in the art, please refer to FIG. 16, which is an application interface of the memo application of the display device. As shown in FIG. 16, the matter recording function of the display device records matters such as "get up at 7:00 am tomorrow", "buy vegetables at 4:20 pm tomorrow", "go to bed at 9:30 pm tomorrow", and "go out for exercise at 9:30 am tomorrow".

[0256] For the convenience of the understanding of those skilled in the art, please refer to FIG. 17, which shows that the display device displays the text information "shutdown" 1710 of the voice input in response to the shutdown instruction of the voice input in the case that the display device displays the video playing interface on the display 260; meanwhile, the display device can generate the prompt information according to the information recorded by the matter recording function of the display device. In practical application, the display device may need a period of time to generate the prompt information, and in this period of time, the display device can display the waiting information 1720 on the display 260. For example, the waiting information 1720 can be the text information "generating the prompt information, please wait for a moment".

[0257] As shown in FIG. 18, after the display device generates the prompt information, the display device can display the prompt information on the first target area 1810 of the display. The prompt information can display the information recorded by the matter recording function of the display device. For example, the prompt information can include the text information "going out for exercise at 9:30 to keep healthy, buying food materials at the vegetable market at 4:20 pm to prepare a delicious dinner, going to bed at 9:30 pm, and wishing you a good dream".

[0258] As can be seen, the "going out for exercise at 9:30 to keep healthy, buying food materials at the vegetable market at 4:20 pm to prepare a delicious dinner, going to bed at 9:30 pm, and wishing you a good dream" in the prompt information shown in FIG. 18 matches the "getting up at 7:00 am tomorrow", "buying vegetables at 4:20 pm tomorrow", "going to bed at 9:30 pm tomorrow", and "going out for exercise at 9:30 am tomorrow" matters recorded by the matter recording function of the display device shown in FIG. 16, and the prompt information can be the natural sentence obtained by the large voice model organizing the matter information recorded by the matter recording function.

[0259] When the display of the prompt information on the first target area of the display reaches the preset time, the shutdown operation is performed. For the convenience of the understanding of those skilled in the art, FIG. 19 shows the state change diagram of the display device when the display of the prompt information on the first target area of the display reaches the preset time, and the shutdown operation is performed.

[0260] The display device is provided through the above process, and the display device includes a display configured to display content from a broadcast system or a network and / or a user interface; and at least one processor interfacing with the display and at least one external device, and configured to execute instructions to cause the display device to display a video playing interface in the display in response to a control instruction of the video playing; and in response to a voice inputted shutdown instruction, display text information of the voice input in a case that the video playing interface is displayed in the display, and display prompt information in a first target area of the display, the prompt information at least including matter information; the matter information at least including information recorded by a matter recording function of the display device; and perform a shutdown operation when the prompt information is displayed in the first target area of the display for a preset time. In this way, the display device can accurately respond to the shutdown instruction inputted by the user voice during the video playing, and display the prompt information in the first target area of the display of the display device during the execution of the shutdown instruction, the prompt information including the information recorded by the matter recording function of the display device, instead of inserting irrelevant content to the user in a long shutdown waiting time in the prior art, so that the user can know the information recorded by the matter recording function of the display device based on the prompt information displayed in the display after the shutdown instruction inputted by the voice, and the information type displayed by the display device during the shutdown is diversified, the intelligent level of the voice interaction system is improved, and more personalized and customized information recommendation to the user is realized.

[0261] In some embodiments, the prompt information further includes weather information.

[0262] The weather information at least includes information displayed by a weather forecast function of the display device. For the understanding of those skilled in the art, please refer to FIG. 20, which is an application interface of the weather forecast function of the display device. As shown in FIG. 20, the information displayed by the weather forecast function of the display device includes weather forecast information such as “sunny today, 19-29 degrees Celsius”, “cloudy yesterday, 22-21 degrees Celsius”, “sunny tomorrow, 21-30 degrees Celsius”, and “cloudy the day after tomorrow, 20-30 degrees Celsius”.

[0263] In a specific implementation, the display device can display the prompt information in the first target area 730 of the display. As shown in FIG. 18, the prompt information can display weather information provided by the weather forecast function of the display device. For example, the prompt information can include text information such as “The weather in area A tomorrow will be sunny, with a temperature between 21 and 30 degrees Celsius, and the average temperature is 1 degree higher than today. The air quality is very good!”.

[0264] It can be seen that the "A area tomorrow's weather will be sunny, temperature between 21 to 30 degrees Celsius, average temperature higher than today by 1 degree. Air quality is very good!" in the prompt information shown in FIG. 18 matches the weather forecast information "Sunny today 19-29 degrees Celsius", "Cloudy yesterday 22-21 degrees Celsius", "Sunny tomorrow 21-30 degrees Celsius", "Cloudy the day after tomorrow 20-30 degrees Celsius" displayed by the weather forecast function of the display device shown in FIG. 20, and the weather information can be a natural sentence obtained by the large language model organizing the weather forecast information displayed by the weather forecast function.

[0265] Through the above process, the display device can also display the weather forecast information displayed by the weather forecast function of the display device in the prompt information, so that the user can understand the weather forecast information displayed by the weather forecast function based on the prompt information displayed by the display after inputting the shutdown instruction by voice, and the type of information displayed by the display device during shutdown is diversified.

[0266] In some embodiments, the prompt information further includes viewing history information.

[0267] The viewing history information at least includes a video title of a historical viewing video recorded by a viewing history function of the display device. For the convenience of understanding of those skilled in the art, please refer to FIG. 21, which is an application interface of the viewing history function of the display device. As shown in FIG. 21, the video title of the historical viewing video recorded by the viewing history function of the display device is "Video B".

[0268] In a specific implementation, the display device can display the prompt information in the first target area 730 of the display. As shown in FIG. 18, the prompt information can display the video title of the historical viewing video recorded by the viewing history function of the display device. For example, the prompt information can include the text information "The TV has been set to start after booting, you can continue to enjoy the video picture of "Video B".

[0269] It can be seen that the "The TV has been set to start after booting, you can continue to enjoy the video picture of "Video B"" in the prompt information shown in FIG. 18 matches the video title "Video B" of the historical viewing video recorded by the viewing history function of the display device shown in FIG. 21, and the viewing history information can be a natural sentence obtained by the large language model organizing the video title of the historical viewing video recorded by the viewing history function.

[0270] In actual applications, before displaying the prompt information, the display device can also display a memory playback inquiry interface, as shown in FIG. 22. The user can set, through the memory playback inquiry interface, whether the display device continues to play the historical viewing video recorded by the historical viewing function next time the display device is started. If the user selects the display device to continue to play the historical viewing video recorded by the historical viewing function next time the display device is started, the display device can display the historical viewing information in the prompt information. If the user selects the display device not to continue to play the historical viewing video recorded by the historical viewing function next time the display device is started, the display device can not display the historical viewing information in the prompt information.

[0271] Through the above process, the display device can also display the video title of the historical viewing video recorded by the historical viewing function in the prompt information, so that the user can know, based on the prompt information displayed by the display, that the display device will continue to play the historical viewing video recorded by the historical viewing function next time the display device is started after the voice input shutdown instruction, thereby achieving more personalized and customized information recommendation for the user.

[0272] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to: in a case where the display displays a video playback interface, display a human-computer interaction interface on an upper layer of the video playback interface; the video playback interface has at least part of a video picture that is not blocked by the human-computer interaction interface; in the human-computer interaction interface, display text information of the voice input, and display the prompt information in the first target area of the display.

[0273] In specific implementations, referring again to FIG. 18, in a case where the display device displays a video playback interface on the display 260, the display device can display a human-computer interaction interface 920 on an upper layer of the video playback interface, and the video playback interface has at least part of a video picture that is not blocked by the human-computer interaction interface. In actual applications, the human-computer interaction interface 920 has a preset transparency, and the video picture of the video playback interface can be displayed to the user through the human-computer interaction interface 920. In the human-computer interaction interface 920, the text information of the voice input is displayed, and the prompt information is displayed in the first target area of the display.

[0274] Through the above process, the display device of the present embodiment, in a case where a video playback interface is displayed, displays a human-computer interaction interface on an upper layer of the video playback interface, and the video playback interface has at least part of a video picture that is not blocked by the human-computer interaction interface, and displays the text information of the voice input and the prompt information in the human-computer interaction interface, thereby avoiding the case that the normal playing of the current playing video of the video playback interface is affected due to the display of the prompt information by the display device.

[0275] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to: display the prompt information in the first target area of the display for a preset time, and then the display device exits the human-computer interaction interface; and when the display device exits the human-computer interaction interface, the display device is powered off.

[0276] For the convenience of those skilled in the art, as shown in FIG. 23, when the display of the prompt information in the first target area of the display reaches a preset time, the display device exits the human-computer interaction interface first; when the display device exits the human-computer interaction interface, the display device is powered off, and the display screen is turned off.

[0277] By the above process, when the display in the first target area reaches a preset time, the display device exits the human-computer interaction interface, and when the display device exits the human-computer interaction interface, the display device is powered off, avoiding the display device displaying the human-computer interaction interface for a long time, so that the display device can respond to and successfully execute the user's voice input power-off instruction in time.

[0278] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to: display key information in the first target area of the display; the key information is used to prompt the user to press a target key to trigger the display device to display the prompt information in the second target area of the display.

[0279] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to: display key information in the first target area of the display; the key information is used to prompt the user to press a target key to trigger the display device to display the prompt information in the second target area of the display.

[0280] By the above process, by displaying the key information in the first target area, the user can be prompted to press the target key to quickly trigger the display device to display the prompt information in the second target area of the display, and since the display area of the second target area is larger than that of the first target area, the user can more clearly view the prompt information displayed by the display device to the user.

[0281] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to: display the prompt information in the first target area of the display for a preset time, and then the display device exits the human-computer interaction interface; and when the display device exits the human-computer interaction interface, the display device is powered off.

[0282] For the convenience of those skilled in the art, referring to FIG. 24, 2410 is a second target area, and the display device can display more detailed prompt information in the second target area. In addition, the display device can display recommendation information 2420 matched with the matter information in the second target area. For example, the matter information includes "go to the vegetable market at 4:20 pm", and the recommendation information can include "recommend you to go to the following supermarket" and the description information card 2430 corresponding to the recommendation information.

[0283] Through the above process, the display device of the embodiment can display more information in the first target area than in the second target area, so that the user can understand the prompt information displayed by the display device in more detail.

[0284] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to, in response to the target key being pressed, start displaying the prompt information in the second target area of the display after the display cancels displaying the prompt information in the first target area.

[0285] For the convenience of those skilled in the art, referring to FIG. 25, the display device starts displaying the prompt information in the second target area of the display after the display cancels displaying the prompt information in the first target area in response to the target key (such as the confirmation key) in the control device of the display device being pressed by the user.

[0286] Through the above process, the display device of the embodiment starts displaying the prompt information in the second target area of the display after the display cancels displaying the prompt information in the first target area in response to the target key in the control device of the display device being pressed, and since the display area of the second target area is larger than that of the first target area, the user can clearly view the prompt information displayed by the display device.

[0287] In some embodiments, as shown in FIGS. 1 and 2, the present disclosure provides a display device, which can include:

[0288] a display 260 configured to display content from a broadcast system or a network and / or a user interface;

[0289] and at least one processor 112, which is in interface connection with the display 260 and at least one external device, and is configured to execute instructions to cause the display device to:

[0290] As shown in FIG. 26, the display device can present the following steps:

[0291] S2601, in response to a control instruction of a video play, the display device controls the display to display a video play interface.

[0292] S2601, in response to the shutdown instruction input by voice, in the case that the display displays the video playing interface, display the text information input by voice, and display the prompt information in the first target area of the display, and play the broadcast audio of the prompt information, the prompt information at least includes the matter information; the matter information at least includes the information recorded by the matter record application installed on the display device.

[0293] S2601, in response to the shutdown instruction input by voice, in the case that the display displays the video playing interface, display the text information input by voice, and display the prompt information in the first target area of the display, and play the broadcast audio of the prompt information, the prompt information at least includes the matter information; the matter information at least includes the information recorded by the matter record application installed on the display device.

[0294] Wherein, the display displays the prompt information in the first target area of the display, and plays the broadcast audio of the prompt information. The processor 112 can use a TTS text-to-speech tool to convert the text in the prompt information into natural speech to obtain the broadcast audio of the prompt information.

[0295] It should be noted that the specific limitations of the above steps can be referred to the above, and will not be repeated here.

[0296] The display device of the above-mentioned process includes a display configured to display content from a broadcast system or network and / or a user interface; and at least one processor, which is connected with the display and at least one external device, and is configured to execute instructions to make the display device display a video playing interface in the display in response to a control instruction of video playing; and in response to a shutdown instruction input by voice, in the case that the display displays the video playing interface, display the text information input by voice, and display the prompt information in the first target area of the display, the prompt information at least includes the matter information; the matter information at least includes the information recorded by the matter record function of the display device, and play the broadcast audio of the prompt information; when the display displays the prompt information in the first target area of the display for a preset time, execute the shutdown operation; in this way, the display device can accurately respond to the shutdown instruction input by voice during the process of playing video, and display the prompt information in the first target area of the display during the process of executing the shutdown instruction, the prompt information includes the information recorded by the matter record function of the display device, instead of inserting the content irrelevant to the user in the long shutdown waiting time in the traditional technology, so that the user can understand the information recorded by the matter record function of the display device based on the prompt information displayed by the display after inputting the shutdown instruction by voice, and the type of information displayed by the display device during the shutdown process is diversified, the intelligent level of the voice interaction system is improved, and more personalized and customized information recommendation for the user is realized.

[0297] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to: highlight or bold the text corresponding to the currently played audio in the prompt information during the process of playing the prompt information.

[0298] In specific implementations, for the convenience of understanding of those skilled in the art, please refer to FIG. 27. During the process of playing the prompt information, the display device can highlight or bold the text corresponding to the currently played audio in the prompt information 2710.

[0299] Through the above process, the display device of the present embodiment can highlight or bold the text corresponding to the currently played audio in the prompt information during the process of playing the prompt information, so that when the user is sensitive to environmental noise or watches the prompt information in a noisy environment, highlighting or bolding the text corresponding to the currently played audio can help the user more easily obtain the semantics in the prompt information, and improve the understanding and acquisition efficiency of the prompt information.

[0300] In some embodiments, the at least one processor 112 is further configured to execute instructions to cause the display device to: during the process of playing the prompt information, reduce or turn off the video volume of the video currently played by the video playing interface, or pause the video currently played by the video playing interface.

[0301] In specific implementations, during the process of playing the prompt information, the display device can also reduce or turn off the video volume of the video currently played by the video playing interface, or pause the video currently played by the video playing interface.

[0302] Through the above process, the display device of the present embodiment can reduce or turn off the video volume of the video currently played by the video playing interface, or pause the video currently played by the video playing interface during the process of playing the prompt information, so as to avoid the sound or picture of the video currently played by the video playing interface interfering with the process of listening to the prompt information.

[0303] Some embodiments of the present disclosure also provide a control method of a display device, which can include: in response to a control instruction of video playing, controlling a display to display a video playing interface; in response to a voice inputted shutdown instruction, displaying text information of the voice inputted in the case of displaying the video playing interface by the display, and displaying prompt information in a first target area of the display, the prompt information at least including item information; the item information at least including information recorded by an item recording function of the display device; when the display of the prompt information in the first target area of the display reaches a preset time, performing a shutdown operation.

[0304] The display device includes a display configured to display content from a broadcast system or network and / or a user interface; and at least one processor interfacing with the display and at least one external device and configured to execute instructions to cause the display device to display a video playback interface in the display in response to a control instruction of a video playback; and in response to a shutdown instruction of a voice input, display text information of the voice input in a case where the video playback interface is displayed in the display, and display prompt information in a first target area of the display, the prompt information including at least item information; the item information including at least information recorded by an item recording function of the display device; and perform a shutdown operation when the prompt information is displayed in the first target area of the display for a preset time. In this way, the display device can accurately respond to the shutdown instruction of the voice input of the user during the video playback, and display the prompt information including the information recorded by the item recording function of the display device in the first target area of the display of the display device during the execution of the shutdown instruction, instead of inserting irrelevant content during the long shutdown waiting time in the conventional technology, so that the user can learn the information recorded by the item recording function of the display device based on the prompt information displayed by the display after the shutdown instruction of the voice input, while improving the diversification of the information type displayed by the display device during the shutdown process, improving the intelligent level of the voice interaction system, and achieving more personalized and customized information recommendation for the user.

[0305] In some embodiments, the prompt information further includes weather information; the weather information including at least information displayed by a weather forecast function of the display device.

[0306] The display device can also display weather forecast information displayed by the weather forecast function of the display device in the prompt information, so that the user can learn the weather forecast information displayed by the weather forecast function based on the prompt information displayed by the display after the shutdown instruction of the voice input, while improving the diversification of the information type displayed by the display device during the shutdown process.

[0307] In some embodiments, the prompt information further includes viewing history information; the viewing history information including at least video titles of historically viewed videos recorded by a viewing history function of the display device.

[0308] The display device can also display the video titles of the historically viewed videos recorded by the viewing history function in the prompt information, so that the user can learn that the display device will continue to play the historically viewed videos recorded by the viewing history function after the next boot based on the prompt information displayed by the display after the shutdown instruction of the voice input, while achieving more personalized and customized information recommendation for the user.

[0309] In some embodiments, when the display displays a video playing interface, a human-computer interaction interface is displayed in a suspended manner on an upper layer of the video playing interface; the video playing interface has at least part of a video picture not blocked by the human-computer interaction interface; in the human-computer interaction interface, text information of voice input is displayed, and prompt information is displayed in a first target area of the display.

[0310] The display device of the embodiment displays a human-computer interaction interface in a suspended manner on an upper layer of a video playing interface when the display device displays the video playing interface, and the video playing interface has at least part of a video picture not blocked by the human-computer interaction interface, and displays text information of voice input in the human-computer interaction interface and prompt information, thereby avoiding the case that the normal playing of a currently played video of the video playing interface is affected due to the display device displaying the prompt information.

[0311] In some embodiments, when the display device displays the prompt information in the first target area of the display for a preset time, the display device exits the display of the human-computer interaction interface.

[0312] When the display device exits the display of the human-computer interaction interface, the display device is powered off.

[0313] When the display device displays the prompt information in the first target area of the display for a preset time, the display device exits the display of the human-computer interaction interface, and when the display device exits the display of the human-computer interaction interface, the display device is powered off, thereby avoiding that the display device displays the human-computer interaction interface for a long time, so that the display device can respond to and successfully execute a power-off instruction input by a user in a timely manner.

[0314] In some embodiments, the display device displays key information in the first target area of the display; the key information is used to prompt a user to press a target key to trigger the display device to display the prompt information in a second target area of the display.

[0315] The target key is at least one key in a control device of the display device, and the display area of the second target area is greater than the display area of the first target area.

[0316] The display device displays the key information in the first target area, thereby prompting the user to press the target key to quickly trigger the display device to display the prompt information in the second target area of the display, and since the display area of the second target area is greater than the display area of the first target area, the user can more clearly view the prompt information displayed by the display device.

[0317] In some embodiments, the information amount of the prompt information displayed in the first target area is greater than the information amount of the prompt information displayed in the second target area.

[0318] The display device of the embodiment displays the prompt information in the first target area with a larger amount of information than the prompt information displayed in the second target area, so that the user can understand the prompt information displayed by the display device in more detail.

[0319] In some embodiments, in response to the target key being pressed, the display device starts displaying the prompt information in the second target area of the display after canceling the display of the prompt information in the first target area.

[0320] The display device of the embodiment starts displaying the prompt information in the second target area of the display after canceling the display of the prompt information in the first target area in response to the target key of the control device of the display device being pressed, so that the user can view the prompt information displayed by the display device to the user more clearly due to the larger display area of the second target area than the first target area.

[0321] Some embodiments of the present disclosure further provide a control method of a display device, which can include: in response to a control instruction of a video playing, controlling a display to display a video playing interface; in response to a voice inputted shutdown instruction, displaying text information of the voice input in a case that the display displays the video playing interface, and displaying prompt information in a first target area of the display, and playing a broadcast audio of the prompt information, the prompt information at least including matter information; the matter information at least including information recorded by a matter record application installed by the display device; when the display of the prompt information in the first target area of the display reaches a preset time, performing a shutdown operation.

[0322] The display device includes a display configured to display content from a broadcast system or network and / or a user interface; and at least one processor interfacing with the display and at least one external device and configured to execute instructions to cause the display device to display a video playing interface in the display in response to a control instruction of the video playing, and in response to a voice inputted shutdown instruction, display text information of the voice input in a case that the video playing interface is displayed in the display, and display prompt information in a first target area of the display, the prompt information at least including matter information, the matter information at least including information recorded by a matter recording function of the display device, and playing a broadcast audio of the prompt information; and perform a shutdown operation when the prompt information is displayed in the first target area of the display for a preset time. In this way, the display device can accurately respond to the shutdown instruction inputted by the user voice during the video playing, and display the prompt information including the information recorded by the matter recording function of the display device in the first target area of the display of the display device during the execution of the shutdown instruction, instead of inserting irrelevant content during the long shutdown waiting time in the traditional technology, so that the user can understand the information recorded by the matter recording function of the display device based on the prompt information displayed by the display after the shutdown instruction inputted by the voice, and the type of information displayed by the display device during the shutdown process is diversified, the intelligent level of the voice interaction system is improved, and more personalized and customized information recommendation for the user is achieved.

[0323] In some embodiments, during the playing of the broadcast audio of the prompt information by the display device, the text corresponding to the current broadcast audio is highlighted or bolded in the prompt information.

[0324] During the playing of the broadcast audio of the prompt information by the display device, the text corresponding to the current broadcast audio is highlighted or bolded in the prompt information, which can help the user to easily obtain the semantics in the prompt information when the user is sensitive to environmental noise or watches the prompt information in a noisy environment, and improve the understanding and acquisition efficiency of the prompt information.

[0325] In some embodiments, during the playing of the broadcast audio of the prompt information, the video volume of the video currently played by the video playing interface is reduced or turned off, or the video currently played by the video playing interface is paused.

[0326] The display device of the embodiment can avoid the sound or picture of the video currently played by the video playing interface from interfering with the process of listening to the broadcast audio of the prompt information by reducing or turning off the video volume of the video currently played by the video playing interface, or pausing the video currently played by the video playing interface during the process of playing the broadcast audio of the prompt information.

[0327] In some embodiments, the present disclosure also provides a voice interactive life service system integrated with a large model for business active recommendation and planning. As shown in FIG. 27, FIG. 27 provides a framework diagram of a voice interactive life service system. The system includes:

[0328] The input module 2710 includes a voice input module for receiving voice input from a user, and a voice recognition processing module for converting the voice input into text data.

[0329] The intent analysis module 2720 is configured to understand the single semantic (single intent semantic) input by the user.

[0330] The business active planning module 2730 is configured to associate the single semantic instruction (shutdown instruction) with the actively recommended business services (which can include transaction record function, weather function, historical play record function, etc.).

[0331] The active planning merging module 2740 includes a service execution module for automatically triggering related services according to the analyzed intent, such as querying schedules and weather information microservices, and a large model long text summarization module for summarizing and reasoning the service content (content merging processing) and displaying it to the user.

[0332] The output module 2750 includes a user interface module for displaying prompt information to the user and providing further operation options. The technical effects of responding to the single intent shutdown instruction and displaying the prompt information obtained by fusing multiple business information are achieved.

[0333] Specifically, the multi-intent understanding module can implement the following steps:

[0334] The display device inputs the region where the user is located, the current date, and the user text into the large model in response to the shutdown instruction input by the user.

[0335] The display device extracts key information and information reasoning from the user's voice input "shutdown". Referring to FIG. 28, FIG. 28 provides an execution logic diagram of a voice interactive system.

[0336] 1) The user inputs a shutdown instruction, and the shutdown instruction is analyzed.

[0337] 2) Extract the required information, including: extract the current system information, the current time 2024.08.30 21:00:00.

[0338] 3) Extract the required information, also including: based on the current time, perform multi-service active planning, the current example after 9 o'clock, perform weather and reminder service call, and call the viewing record inquiry next time whether to perform the last viewing picture jump;

[0339] 4) Assemble the information input by the user and the information obtained by the service active recommendation into a prompt word (prompt) input to the large model for reasoning assembly to obtain structured prompt information; the information input to the large model can include:

[0340] Instruction: shutdown

[0341] Current time: 21:00 on August 30, 2024

[0342] Second day schedule reminder:

[0343] 1. Get up at 7:00 tomorrow

[0344] 2. Go out for exercise at 9:31 tomorrow

[0345] 3. Go to the market to buy vegetables at 16:20 tomorrow

[0346] 4. Go to bed at 21:30 tomorrow

[0347] Second day weather reminder: A area tomorrow heavy rain, 23 to 27 degrees Celsius, east wind 4 levels

[0348] Boot display last viewing picture inquiry: yes

[0349] 5) The improved information can include:

[0350] "Tomorrow A area heavy rain, don't forget to take an umbrella! Get up at 7 o'clock, full of energy to meet a new day. Go out for exercise at 9:30, keep healthy. Go to the market to buy vegetables at 16:20, prepare delicious dinner. Go to bed at 21:30, may you have sweet dreams. The TV has been set to continue enjoying the wonderful pictures of Video B. May you have a happy tomorrow!".

[0351] In practical application, the display device can also use the large model for task scheduling and prompt polishing.

[0352] Specifically, it can include the following steps:

[0353] 1. Semantic understanding of main switch-off instruction;

[0354] 2. Use the large model for service active planning, combine with the existing user scene, and perform multi-service call;

[0355] 3. Design a large model prompt. On the selection of business assembly, requirements: such as not to broadcast the business assembly of the reminder, determine which day to broadcast the reminder according to the current time & weather, agree to start the machine before the last viewing screen and no longer ask, etc.

[0356] 4. The text understanding ability and reasoning ability of the large model are arranged for tasks, and the short text supported by the semantic system architecture is disassembled, that is: shutdown, tomorrow's reminder, tomorrow's weather in Qingdao Laoshan, and the last viewing screen.

[0357] 5. The sentence information enters the natural language understanding system for intent understanding.

[0358] 6. The understanding system based on natural speech technology understands each sentence, identifies user intent, and after understanding user intent, calls business microservices to obtain business data, and downgrades the business data to the terminal for information display.

[0359] 7. Use the large model to summarize the output results, and for weather and reminder prompts, including:

[0360] For weather, give abnormal weather reminders, clothing reminders, and umbrella reminders;

[0361] For reminders, give relevant reminders for schedules;

[0362] For the last viewing screen, give the last viewing screen whether to set the machine to start;

[0363] 8. The display device performs the shutdown instruction and broadcasts the reminder information.

[0364] The specific text description of the display effect is as follows:

[0365] Before the display device executes the final shutdown instruction, it will broadcast the presentation according to the induction content time for the shutdown ability;

[0366] "Tomorrow, there will be heavy rain in A area, don't forget to take an umbrella! Get up at 7 o'clock, and have a good start to the new day. Go out for exercise at 9:30 to keep healthy. Go to the market at 16:20 to buy vegetables and prepare a delicious dinner. Go to bed at 21:30 and have a good dream. The TV has been set to start and enjoy the wonderful pictures of Video B. I wish you a happy tomorrow!"

[0367] After the display device displays the prompt information for the preset time, it starts to execute the final shutdown instruction until the display device completes the shutdown.

[0368] The voice interactive life service system provided by the above process can enable the display device to accurately respond to the shutdown instruction input by the user voice during the video playing process, and display prompt information including the information recorded by the matter record function of the display device on the first target area of the display of the display device during the execution of the shutdown instruction, which replaces the irrelevant content inserted in the long shutdown waiting time in the traditional technology, so that the user can understand the information recorded by the matter record function of the display device based on the prompt information displayed by the display after the voice input shutdown instruction, and the information type displayed by the display device during the shutdown process is diversified, the intelligent level of the voice interaction system is improved, and more personalized and customized information recommendation for the user is realized.

[0369] For the convenience of those skilled in the art, FIG. 29 provides a signaling timing diagram of the display device executing the control method of the display device described above. As shown in FIG. 29, it can include:

[0370] S2901, the voice terminal transmits parameters to the semantic gateway;

[0371] After the user inputs the shutdown instruction to the display device by voice, the voice terminal (i.e., the voice system display module) of the display device encrypts the parameters of the shutdown instruction, and transmits the encrypted parameters to the semantic gateway.

[0372] S2902, the semantic gateway verifies and decrypts the parameters;

[0373] S2903, the semantic gateway transmits the decrypted parameters to the single instruction module;

[0374] The semantic gateway verifies and decrypts the parameters transmitted by the voice terminal, and transmits the processed parameters to the single instruction module (i.e., the intent analysis module 2720) of the voice interactive life service system of the display device.

[0375] S2904, the single instruction module understands the intent and analyzes the instruction of the decrypted parameters;

[0376] S2905, the single instruction module transmits the single instruction analysis result to the business active planning module;

[0377] The intent understanding and instruction analysis of the shutdown instruction are realized by the intent analysis module 2720, and the single instruction analysis result is obtained. The intent analysis module 2720 sends the single instruction analysis result to the business active planning module 2730 of the voice interactive life service system.

[0378] S2906, the business active planning module sends the large model planning associated scene to the large model module;

[0379] S2907, the large model module returns the associated scene result;

[0380] The business active planning module 2730 calls the large model module to plan the associated scene, and obtains the associated scene result returned by the large model module.

[0381] S2908, the business active planning module sends the active planning associated business to the single instruction module;

[0382] S2909, the single instruction module returns the service content of the associated business service to the business active planning module;

[0383] S2910, the business active planning module sends the multi-service content merging result to the single instruction module;

[0384] The business active planning module 2730 actively plans the associated business service by using the associated scene result, and executes the associated business service to obtain the service content (such as transaction information, weather information, etc.) provided by the associated business service. Then, the business active planning module 2730 performs multi-content merging on the above service content to obtain a multi-content merging result.

[0385] S2911, the single instruction module sends a media asset playing related query instruction to the television media asset system;

[0386] S2912, the television media asset system returns a query result;

[0387] In the case that the user selects the display device to continue playing the historical playing video at the next boot, the single instruction module can also query the television media asset system for media asset playing, obtain the historical playing video query result returned by the television media asset system, and transmit the parameters of the multi-content merging result and the historical playing video query result back to the voice terminal.

[0388] S2913, the semantic terminal analyzes the query result;

[0389] S2914, the voice terminal transmits the parameters to the semantic gateway;

[0390] The voice terminal encrypts the parameters returned by the single instruction module, and transmits the encrypted parameters to the semantic gateway.

[0391] S2915, the semantic gateway verifies and decrypts the parameters;

[0392] S2916, the semantic gateway transmits the decrypted parameters to the large model module;

[0393] S2907. The large model module returns the broadcast content;

[0394] The semantic gateway checks and decrypts the parameters transmitted by the voice terminal, and transmits the processed parameters to the large model module, so that the large model module assembles the broadcast content containing the prompt information based on the multi-content merging result and the historical video query result. The broadcast content presents the prompt information to the user in a natural language manner. The voice terminal obtains the broadcast content containing the prompt information generated by the large model module, and displays the broadcast content to the user.

[0395] Through the above process, the display device can accurately respond to the shutdown instruction input by the user voice during the video playing process, and display prompt information through the first target area of the display of the display device during the execution of the shutdown instruction. The prompt information includes the information recorded by the matter recording function of the display device, which replaces the irrelevant content inserted in the long shutdown waiting time in the traditional technology, so that the user can understand the information recorded by the matter recording function of the display device based on the prompt information displayed by the display after inputting the shutdown instruction by voice, and the type of information displayed by the display device during shutdown is diversified, the intelligent level of the voice interaction system is improved, and more personalized and customized information recommendation for the user is realized.

[0396] Referring to FIG. 30, in some embodiments, the control method of the display device can include:

[0397] S3001: in response to a control instruction for entering a game application, displaying a game application interface;

[0398] S3002: in response to a control instruction associated with a first key of a remote controller, determining a first instruction associated with the first key in the case of entering the game application;

[0399] S3003: based on the first instruction, displaying at least one video thumbnail in a first display area of the display while displaying the game application interface on the display; the video thumbnail is a thumbnail generated from a video associated with the game application.

[0400] Referring to FIG. 31, in some embodiments, the control method of the display device can include:

[0401] S3101: in response to a control instruction for entering a game application, displaying a game application interface;

[0402] S3102: in response to a control instruction associated with a first key of a remote controller, determining a first instruction associated with the first key in the case of entering the game application;

[0403] S3103: based on the first instruction, while the display displays the game application interface, display text information and a preset number of video thumbnails in a sixth display area of the display, the text information being text information associated with an interface element of the game application interface, and the video thumbnails being thumbnails generated from videos associated with the game application.

[0404] Referring to FIG. 32, in some embodiments, the control method of the display device can include:

[0405] S3201: in response to a control instruction for entering a game application, display a game application interface;

[0406] S3202: while displaying the game application interface, in response to received voice input information, display at least one video thumbnail in a first display area of the display while the display displays the game application interface; the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

[0407] Referring to FIG. 33, in some embodiments, the control method of the display device can include:

[0408] S3301: in response to a control instruction for entering a game application, display a game application interface;

[0409] S3302: while displaying the game application interface, in response to received voice input information, display text information and a preset number of video thumbnails in a sixth display area of the display while the display displays the game application interface, the text information being text information associated with the voice input information and an interface element of the game application interface, and the video thumbnails being thumbnails generated from a video associated with the voice input information and the game application.

[0410] Referring to FIG. 34, in some embodiments, the control method of the display device can include:

[0411] S3401: in response to a game application startup instruction, load game application data and display a game application interface;

[0412] S3402: while displaying the game application interface after the game application data is loaded, display at least one video thumbnail in a first display area of the display; the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

[0413] In some embodiments, the method further includes: switching the game application interface to be displayed in a seventh display area of the display, and displaying the at least one video thumbnail in the first display area of the display, the seventh display area being non-overlapped with the first display area, so that the display displays the game application interface and displays the at least one video thumbnail in the first display area of the display.

[0414] In some embodiments, the method further includes: keeping the display area of the game application interface unchanged, generating a new layer above the layer of the game application interface in the first display area of the display, and displaying the at least one video thumbnail in the new layer, so that the display displays the game application interface and displays the at least one video thumbnail in the first display area of the display.

[0415] In some embodiments, the at least one video thumbnail is arranged and displayed in the first display area according to a preset order.

[0416] In some embodiments, the at least one video thumbnail is arranged and displayed in the first display area according to an order from large to small of a play amount of a video associated with the video thumbnail.

[0417] In some embodiments, the at least one video thumbnail is arranged and displayed in the first display area according to an order of a relevance of a video associated with the video thumbnail to the game application.

[0418] In some embodiments, the at least one video thumbnail is arranged and displayed in the first display area according to an order of a favorable degree of a video associated with the video thumbnail.

[0419] In some embodiments, the at least one video thumbnail is arranged and displayed in the first display area according to an order from recent to remote of an upload time of a video associated with the video thumbnail.

[0420] In some embodiments, a floating layer is displayed in the first display area, and the at least one video thumbnail is displayed on the floating layer.

[0421] In some embodiments, the method further includes: displaying a first control in the first display area of the display when the at least one video thumbnail of the game video is displayed in the first display area of the display; and in response to an instruction of operating the first control, keeping the display area of the display of the game application interface unchanged, and displaying a guide prompt interface in a second display area of the display.

[0422] In some embodiments, the method further includes: in response to the received voice input information, while displaying the game application interface on the display, displaying, in the sixth display area of the display, text information and a preset number of video thumbnails; wherein the text information is text information associated with the voice input information and an interface element of the game application interface, and the video thumbnails are thumbnails of videos associated with the voice input information and the game application.

[0423] In some embodiments, the method further includes: in response to a selection operation of selecting one of the video thumbnails, keeping the display area of the display unchanged while displaying the game application interface, generating a video player in the third display area of the display, and playing, through the video player, a video associated with the video thumbnail corresponding to the selection operation.

[0424] In some embodiments, the method further includes: in the process of playing the video through the video player generated in the third display area of the display, if a game display mode switching instruction is received: adjusting the display area of the video player to the fourth display area of the display, so that the video player plays the video in the fourth display area; and updating the display area of the game application interface to the fifth display area of the display, so that the game application interface is displayed in the fifth display area of the display; wherein the fourth display area and the fifth display area do not overlap.

[0425] In some embodiments, the method further includes: in the process of playing the video associated with the video thumbnail corresponding to the selection operation through the video player, displaying at least one second control within a preset range of the display range of the video player, the second control including a display mode switching second control; and in response to an operation on the display mode switching second control, receiving the game display mode switching instruction.

[0426] In some embodiments, the second control further includes a hiding control component, and the method further includes: in response to an operation on the hiding control component, displaying the game application interface full screen on the display, controlling the playing window of the video player to be invisible on the display interface of the display, and displaying playing window display mode prompt information at a preset edge position of the display screen of the display.

[0427] In some embodiments, the second control further includes a playing control component, and the method further includes: in response to a playing control instruction received through the playing control component, controlling at least one of starting playing, pausing playing, rewinding a first playing duration, or advancing a second playing duration of the video.

[0428] In some embodiments, the second control further includes an audio control component, and the method further includes: in response to an audio control instruction received through the audio control component, controlling playing and stopping playing of audio corresponding to the video.

[0429] In some embodiments, the method further includes: in response to the selection operation of selecting one of the video thumbnails: updating the game application interface in the display region of the display to a fifth display region of the display, so that the game application interface is displayed in the fifth display region of the display; generating a video player in the fourth display region of the display, and playing the video associated with the video thumbnail corresponding to the selection operation through the video player; the fourth display region does not overlap with the fifth display region.

[0430] In some embodiments, the method further includes: during the process of playing the video through the video player generated in the fourth display region of the display, if a game display mode switching instruction is received: updating the game application interface in the display region of the display to a full-screen region of the display, so that the game application interface is displayed full-screen on the display; adjusting the display region of the video player to a third display region of the display, so that the video player plays the video in the third display region.

[0431] In some embodiments, the method further includes: obtaining a frame image of the currently displayed game application interface, identifying game interface elements in the frame image, the game interface elements including at least one of a background, a prop, and a character; retrieving based on the game interface elements in a game knowledge graph to obtain game name and game level information, the nodes of the game knowledge graph including at least the game interface elements, the game name, and the level; the connection lines of the game knowledge graph represent that the nodes at both ends of the connection lines have an association relationship; determining a query keyword based on the game interface elements, the game name, and the game level information; sending a query request to the server based on the determined query keyword, and obtaining at least one video thumbnail returned by the server based on the query request.

[0432] In some embodiments, the method further includes: in response to the voice input information, obtaining a frame image of the currently displayed game application interface, identifying game interface elements in the frame image, the game interface elements including at least one of a background, a prop, and a character; retrieving based on the game interface elements in a game knowledge graph to obtain game name and game level information, the nodes of the game knowledge graph including at least the game interface elements, the game name, and the level; the connection lines of the game knowledge graph represent that the nodes at both ends of the connection lines have an association relationship; determining a query keyword based on the voice input information, the game name, and the game level information; sending a query request to the server based on the determined query keyword, and obtaining text information and game video information returned by the server based on the query request.

[0433] It should be understood that although each step in the flowchart involved in each of the above embodiments is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in the order of the direction of the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential but can be executed in rotation or alternation with at least part of other steps or stages in other steps.

[0434] In some embodiments, a computer readable storage medium is provided, on which a computer program is stored, the computer program being executed by a processor to implement the steps of the control method of the display device in any of the above embodiments.

[0435] In some embodiments, a computer program product is provided, comprising a computer program, the computer program being executed by a processor to implement the steps of the control method of the display device in any of the above embodiments.

[0436] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0437] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments of the present disclosure can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments of the present disclosure can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without limitation. The processor involved in the embodiments of the present disclosure can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without limitation.

Claims

1. A display device, comprising: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol comprising at least a Bluetooth protocol; at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform: controlling the display to display a game application interface in response to a control instruction for entering the game application; in a case of entering the game application, determining a first instruction associated with a first button of a remote controller in response to a control instruction associated with the first button, the first instruction being used to display at least one video thumbnail in a first display area of the display when the display displays the game application interface. The video thumbnail is a thumbnail generated from a video associated with the game application. 2.A display device, comprising: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol comprising at least a Bluetooth protocol; at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform: controlling the display to display a game application interface in response to a control instruction for entering the game application; in a case of entering the game application, determining a first instruction associated with a first button of a remote controller in response to a control instruction associated with the first button, the first instruction being used to display text information and a preset number of video thumbnails in a sixth display area of the display when the display displays the game application interface, the text information being text information associated with an interface element of the game application interface, and the video thumbnail being a thumbnail generated from a video associated with the game application. 3.A display device, comprising: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol comprising at least a Bluetooth protocol; at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform: controlling the display to display a game application interface in response to a control instruction for entering the game application; in a case of displaying the game application interface, displaying at least one video thumbnail in a first display area of the display when the display displays the game application interface in response to received voice input information, the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application. 4.A display device, comprising: a display; ​ a memory configured to store computer instructions and data associated with the display device; at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol comprising at least a Bluetooth protocol; at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform: in response to a control instruction for entering a game application, controlling the display to display a game application interface; in a case where the game application interface is displayed, in response to received voice input information, displaying, on the display, text information and a preset number of video thumbnails in a sixth display area of the display, the text information being text information associated with the voice input information and an interface element of the game application interface, and the video thumbnails being thumbnails of a video associated with the voice input information and the game application.

5. A display device, the display device comprising: a display; a memory configured to store computer instructions and data associated with the display device; at least one external device interface configured to communicate with at least one external device according to a communication protocol, the communication protocol comprising at least a Bluetooth protocol; at least one processor connected to the display, the at least one external device interface, and the memory, and configured to execute the computer instructions to cause the display device to perform: in response to a game application start instruction, loading game application data, and controlling the display to display a game application interface; in a case where the game application data is loaded and the game application interface is displayed on the display, displaying at least one video thumbnail in a first display area of the display; the video thumbnail being a thumbnail of a video associated with the voice input information and the game application.

6. The display device of any one of claims 1, 3, and 5, the at least one processor being further configured to execute the computer instructions to cause the display device to perform: switching the game application interface to be displayed in a seventh display area of the display, and displaying at least one video thumbnail in the first display area of the display, the seventh display area being non-overlapping with the first display area, such that at least one video thumbnail is displayed in the first display area of the display when the game application interface is displayed on the display.

7. The display device of any one of claims 1, 3, and 5, the at least one processor being further configured to execute the computer instructions to cause the display device to perform: keeping a display area of the game application interface unchanged, generating a new layer above the first display area of the display and a layer of the game application interface, and displaying at least one video thumbnail in the new layer, such that at least one video thumbnail is displayed in the first display area of the display when the game application interface is displayed on the display.

8. The display device of any one of claims 1, 3, 5, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: displaying the first control in the first display area of the display while displaying the video thumbnails of the at least one game video in the first display area of the display; in response to an instruction of operating the first control, keeping the display area of the display device displaying the game application interface unchanged, and displaying a guide prompt interface in a second display area of the display.

9. The display device of claim 8, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: in response to receiving voice input information while displaying the guide prompt interface, displaying text information and a preset number of video thumbnails in a sixth display area of the display while displaying the game application interface in the display; wherein the text information is text information associated with the voice input information and an interface element of the game application interface, and the video thumbnails are thumbnails of videos associated with the voice input information and the game application.

10. The display device of any one of claims 1 to 5, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: in response to a selection operation of selecting one of the video thumbnails, keeping the display area of the display device displaying the game application interface unchanged, generating a video player in a third display area of the display, and playing a video associated with the video thumbnail corresponding to the selection operation through the video player.

11. The display device of claim 10, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: in a process of playing a video through the video player generated in the third display area of the display, if a game display mode switching instruction is received: adjusting the display area of the display device displaying the video player to a fourth display area of the display device, so that the video player plays the video in the fourth display area; and updating the display area of the display device displaying the game application interface to a fifth display area of the display device, so that the game application interface is displayed in the fifth display area of the display; the fourth display area and the fifth display area are non-overlapping.

12. The display device of claim 9, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: in a process of playing the video associated with the video thumbnail corresponding to the selection operation through the video player, displaying at least one second control in a preset range of the display range of the video player, the second control including a display mode switching second control; in response to an operation of the display mode switching second control, receiving the game display mode switching instruction.

13. The display device of claim 9, wherein the second control further comprises a hidden control component, and the at least one processor is further configured to execute computer instructions to cause the display device to perform: in response to an operation on the hidden control component, full-screen display the game application interface on the display, control the video player window to be invisible on the display interface of the display, and display a window display mode prompt information at a preset edge position of the display screen of the display.

14. The display device of any one of claims 1 to 5, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: in response to a selection operation of selecting one of the video thumbnails: update the game application interface on the display area of the display to a fifth display area of the display, so as to display the game application interface on the fifth display area of the display; generate a video player on the fourth display area of the display, and play a video associated with the video thumbnail corresponding to the selection operation through the video player; the fourth display area does not overlap with the fifth display area.

15. The display device of claim 14, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: in the process of playing a video through the video player generated on the fourth display area of the display, if a game display mode switching instruction is received: update the game application interface on the display area of the display to a full-screen area of the display, so as to full-screen display the game application interface on the display; adjust the video player on the display area of the display to a third display area of the display, so as to play the video in the third display area.

16. The display device of any one of claims 1, 3, and 4, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: obtain a frame image of the currently displayed game application interface, and identify game interface elements in the frame image, the game interface elements including at least one of a background, a prop, and a character; based on the game interface elements, retrieve in a game knowledge graph to obtain a game name and game level information, the nodes of the game knowledge graph including at least game interface elements, game names, and levels; and the connection lines of the game knowledge graph represent that the nodes at both ends of the connection lines have an association relationship; determine a query keyword based on the game interface elements, the game name, and the game level information; based on the determined query keyword, send a query request to a server, and obtain the at least one video thumbnail returned by the server based on the query request.

17. The display device of claim 2 or 4, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to the voice input information, a frame image of the currently displayed game application interface is acquired, and a game interface element in the frame image is identified, the game interface element including at least one of a background, a prop, and a character; Game name and game level information are obtained based on the game interface element in a game knowledge graph, nodes of the game knowledge graph including at least a game interface element, a game name, and a level, and a connection line of the game knowledge graph indicating that nodes at two ends of the connection line have an association relationship; Based on the voice input information, the game name, and the game level information, a query keyword is determined; Based on the determined query keyword, a query request is sent to a server, and the text information and game video information returned by the server based on the query request are obtained.

18. The display device of any one of claims 1-5, the at least one processor further configured to execute computer instructions to cause the display device to perform: in response to a control instruction of a video play, control the display to display a video play interface; in response to a voice input shutdown instruction, display text information of the voice input on the display while the display displays the video play interface, and display prompt information on a first target area of the display, the prompt information including at least matter information; the matter information including at least information recorded by a matter recording function of the display device; when the display of the prompt information on the first target area of the display reaches a preset time, perform a shutdown operation.

19. The display device of claim 18, the prompt information further including weather information; the weather information including at least information displayed by a weather forecast function of the display device.

20. The display device of claim 18, the prompt information further including viewing history information; the viewing history information including at least a video title of a historically viewed video recorded by a viewing history function of the display device.

21. The display device of claim 18, the at least one processor further configured to execute instructions to cause the display device to: while the display displays the video play interface, display a human-computer interaction interface on an upper layer of the video play interface; the video play interface having at least part of a video picture not being blocked by the human-computer interaction interface; in the human-computer interaction interface, display the text information of the voice input, and display the prompt information on the first target area of the display.

22. The display device of claim 21, the at least one processor further configured to execute instructions to cause the display device to: when the display of the prompt information on the first target area of the display reaches a preset time, the display device exits the display of the human-computer interaction interface; when the display device exits the display of the human-computer interaction interface, the display device shuts down.

23. The display device of claim 18, the at least one processor further configured to execute instructions to cause the display device to: display key information in the first target area of the display; the key information is used to prompt a user to press a target key to trigger the display device to display the prompt information in a second target area of the display; wherein, the target key is at least one key in a control device of the display device, and a display area of the second target area is greater than a display area of the first target area.

24. The display device of claim 23, wherein the at least one processor is further configured to execute instructions to cause the display device to: display the prompt information in the first target area in an amount of information that is greater than an amount of information of the prompt information displayed in the second target area.

25. The display device of claim 23, wherein the at least one processor is further configured to execute instructions to cause the display device to: in response to the target key being pressed, start displaying the prompt information in the second target area of the display after the display cancels displaying the prompt information in the first target area.

26. The display device of any one of claims 1-5, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: in response to a control instruction of a video play, control the display to display a video play interface; in response to a voice inputted shutdown instruction, display text information of the voice input while the display displays the video play interface, and display prompt information in a first target area of the display and play a broadcast audio of the prompt information, the prompt information at least including matter information; the matter information at least including information recorded by a matter record application installed in the display device; perform a shutdown operation when the display displays the prompt information in the first target area for a preset time.

27. The display device of claim 26, wherein the at least one processor is further configured to execute instructions to cause the display device to: highlight or bold display text corresponding to a current broadcast audio in the prompt information during a process in which the display device plays the broadcast audio of the prompt information.

28. The display device of claim 26, wherein the at least one processor is further configured to execute instructions to cause the display device to: decrease or turn off a video volume of a video currently played by the video play interface, or pause playing the video currently played by the video play interface, during a process in which the display device plays the broadcast audio of the prompt information.

29. A control method of a display device, the method comprising: in response to a control instruction of entering a game application, display a game application interface; in response to a control instruction associated with a first key of a remote controller, determine a first instruction associated with the first key while entering the game application; based on the first instruction, display at least one video thumbnail in a first display area of the display while the display displays the game application interface; the video thumbnail is a thumbnail generated from a video associated with the game application.

30. A control method of a display device, the method comprising: displaying a game application interface in response to a control instruction of entering the game application; determining a first instruction associated with a first key of a remote controller in response to a control instruction associated with the first key, in a case that the game application is entered; displaying text information and a preset number of video thumbnails in a sixth display area of the display based on the first instruction, when the display displays the game application interface, the text information being text information associated with an interface element of the game application interface, the video thumbnails being thumbnails generated from videos associated with the game application.

31. A control method of a display device, the method comprising: displaying a game application interface in response to a control instruction of entering the game application; displaying at least one video thumbnail in a first display area of the display when the display displays the game application interface in response to received voice input information; the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

32. A control method of a display device, the method comprising: displaying a game application interface in response to a control instruction of entering the game application; displaying text information and a preset number of video thumbnails in a sixth display area of the display when the display displays the game application interface in response to received voice input information, the text information being text information associated with the voice input information and an interface element of the game application interface, the video thumbnails being thumbnails generated from videos associated with the voice input information and the game application.

33. A control method of a display device, the method comprising: loading game application data and displaying a game application interface in response to a game application start instruction; displaying at least one video thumbnail in a first display area of the display when the display displays the game application interface after the game application data is loaded; the video thumbnail being a thumbnail generated from a video associated with the voice input information and the game application.

Images