Display device and external device joint control method

Abstract

The embodiment of the invention provides a display device and external device joint control method, and the method can respond to a return instruction and generate a current interface image through screenshot when a user interface displayed by the display device comprises a first user interface provided by an external device. Obtaining a historical interface image, and calculating the similarity between the current interface image and the historical interface image; if the similarity is smaller than the similarity threshold value, sending a return instruction to the external equipment through the device interface to display a first user interface after the external equipment executes the return instruction; and if the similarity is greater than or equal to the similarity threshold, executing a return instruction by the display device, and displaying the second user interface. According to the method, on the basis that joint control takes effect, when the external equipment returns to the last interface, the return instruction can exit logic of the interface of the external equipment, so that the interaction continuity is improved, and the problem that return operation does not take effect when the external equipment returns to the last homepage interface is solved.

Description

Technical Field

[0001] This application relates to the technical field of display devices, and in particular, to a display device and a method for jointly controlling an external device. Background Art

[0002] A display device is an intelligent device that can present a user interface and support user interaction. A device interface may be provided on the display device, and the display device can be connected to an external device through the device interface to achieve data interaction with the external device. For example, the display device can be connected to an external device such as a set-top box through a High Definition Multimedia Interface (HDMI). After connecting to an external device such as a set-top box, the display device can obtain a signal source provided by the set-top box through the HDMI interface to display a media resource screen corresponding to the signal source.

[0003] Some device interfaces of display devices also support controlling external devices. For example, the HDMI interface of a display device supports a Consumer Electronics Control (CEC) channel. The display device can transmit control instructions to the external device through the HDMI CEC channel. When the external device supports the CEC protocol, the external device can respond to the control instructions sent by the display device, such as the manipulation of function keys such as arrow keys, OK, and back, to achieve operations such as switching the input source and adjusting the volume of the set-top box by using the remote control of the display device.

[0004] However, when the display device controls an external device, the control instructions are transmitted to the external device without limitation, resulting in some scenarios where the transmitted control instructions cannot be responded to by the external device to change the interface content. For example, when the display device sends a return instruction corresponding to the return key to the external device through the HDMI CEC channel, and the external device has already returned to the last home page interface, but the display device still sends the return key to the external device, then the return key will not take effect at this time, and the display device will stay on the last home page interface for a long time, reducing the user experience. Summary of the Invention

[0005] Some embodiments of this application provide a display device and a method for jointly controlling an external device to solve the problem that the return operation does not take effect when the external device returns to the last home page interface.

[0006] In a first aspect, some embodiments of this application provide a display device, including: a display, a device interface, and a controller. The display is configured to display a user interface, where the user interface includes a first user interface and / or a second user interface. The first user interface is a user interface provided by an external device; the second user interface is a user interface provided by the display device; the device interface is configured to connect to an external device;

[0007] The controller is configured to perform the following steps:

[0008] When the user interface displayed on the display includes a first user interface, in response to a return instruction input by the user, take a screenshot of the current user interface to generate a current interface image;

[0009] Obtain a historical interface image, where the historical interface image is an image generated by taking a screenshot of the user interface before the return instruction;

[0010] Calculate the similarity between the current interface image and the historical interface image;

[0011] If the similarity is less than the similarity threshold, send the return instruction to the external device through the device interface, and control the display to display the first user interface of the external device after executing the return instruction;

[0012] If the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display a second user interface.

[0013] In a second aspect, some embodiments of the present application further provide a method for jointly controlling an external device, which is applied to the display device described in the first aspect. The display device includes a display, a device interface, and a controller. The display device is connected to the external device through the device interface. The method for jointly controlling the external device includes:

[0014] When the user interface displayed on the display includes a first user interface, in response to a return instruction input by the user, take a screenshot of the current user interface to generate a current interface image. The first user interface is a user interface provided by the external device;

[0015] Obtain a historical interface image, where the historical interface image is an image generated by taking a screenshot of the user interface before the return instruction;

[0016] Calculate the similarity between the current interface image and the historical interface image;

[0017] If the similarity is less than the similarity threshold, send the return instruction to the external device through the device interface, and control the display to display the first user interface of the external device after executing the return instruction;

[0018] If the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display a second user interface. The second user interface is a user interface provided by the display device.

[0019] In a third aspect, some embodiments of the present application further provide a display device, including: a display, a device interface, and a controller. The display is configured to display a user interface, where the user interface includes a first user interface and / or a second user interface. The first user interface is a user interface provided by an external device; the second user interface is a user interface provided by the display device; the device interface is configured to connect to the external device;

[0020] The controller is configured to perform the following steps:

[0021] When the user interface displayed on the display includes the first user interface, in response to a return instruction input by the user, take a screenshot of the current user interface to generate a first interface image;

[0022] Send the return instruction to the external device through the device interface, and obtain a second interface image after the external device executes the return instruction;

[0023] Calculate the similarity between the first interface image and the second interface image;

[0024] If the similarity is less than the similarity threshold, control the display to display the first user interface after the external device executes the return instruction;

[0025] If the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display the second user interface.

[0026] In a fourth aspect, some embodiments of the present application further provide an external device control method, which is applied to the display device described in the third aspect. The display device includes a display, a device interface, and a controller. The display device is connected to an external device through the device interface. The external device control method includes:

[0027] When the user interface displayed on the display includes the first user interface, in response to a return instruction input by the user, take a screenshot of the current user interface to generate a first interface image. The first user interface is a user interface provided by the external device;

[0028] Send the return instruction to the external device through the device interface, and obtain a second interface image after the external device executes the return instruction;

[0029] Calculate the similarity between the first interface image and the second interface image;

[0030] If the similarity is less than the similarity threshold, control the display to display the first user interface after the external device executes the return instruction;

[0031] If the similarity is greater than or equal to the similarity threshold, execute the return instruction and control the display to display a second user interface, which is a user interface provided by the display device.

[0032] As can be seen from the above technical solutions, some embodiments of the present application provide a display device and an external device joint control method. When the user interface displayed by the display device includes a first user interface provided by an external device, in response to a return instruction input by the user, a screenshot of the current user interface is taken to generate a current interface image. A historical interface image generated by taking a screenshot of the user interface before the return instruction is obtained, and then the similarity between the current interface image and the historical interface image is calculated. If the similarity is less than the similarity threshold, a return instruction is sent to the external device through the device interface to display the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, the display device executes the return instruction and displays the second user interface. The method can realize that on the basis of the effective joint control, when the external device returns to the last interface, the return instruction can exit the logic of the external device interface, improving the continuity of interaction to solve the problem that the return operation is ineffective when the external device returns to the last home page interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0034] Figure 1 It is a schematic diagram of the usage scenario of the display device in some embodiments of the present application;

[0035] Figure 2 It is a hardware configuration block diagram of the display device provided by some embodiments of the present application;

[0036] Figure 3 It is a schematic diagram of the connection relationship between the display device and the external device provided by some embodiments of the present application;

[0037] Figure 4 It is a hardware configuration block diagram of the control device provided by some embodiments of the present application;

[0038] Figure 5 It is a software configuration schematic diagram of the display device provided by some embodiments of the present application;

[0039] Figure 6 It is a schematic diagram of the HDMI CEC transmission function provided by some embodiments of the present application;

[0040] Figure 7 Schematic diagram of the set-top box control home page interface provided by some embodiments of the present application;

[0041] Figure 8 Schematic diagram of the HDMI signal interface switching provided by some embodiments of the present application;

[0042] Figure 9 Schematic diagram of the process of a method for jointly controlling external devices provided by some embodiments of the present application;

[0043] Figure 10 Schematic diagram of the signal source switching interface provided by some embodiments of the present application;

[0044] Figure 11 Schematic diagram of the process of taking screenshots according to the play window control provided by some embodiments of the present application;

[0045] Figure 12 Schematic diagram of the process of taking screenshots according to the media asset data stream provided by some embodiments of the present application;

[0046] Figure 13 Schematic diagram of the process of obtaining historical interface images provided by some embodiments of the present application;

[0047] Figure 14 Schematic diagram of the display effect of the exit prompt message provided by some embodiments of the present application;

[0048] Figure 15 Schematic diagram of the process of listening for user operation instructions provided by some embodiments of the present application;

[0049] Figure 16 Schematic diagram of the joint control interface provided by some embodiments of the present application;

[0050] Figure 17 Schematic diagram of the process of executing a return instruction according to the joint control function item setting information provided by some embodiments of the present application;

[0051] Figure 18 Schematic diagram of the process of executing a return instruction according to the screenshot time interval provided by some embodiments of the present application;

[0052] Figure 19 Schematic diagram of another method for jointly controlling external devices provided by some embodiments of the present application. Detailed implementation manners

[0053] Embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0054] Embodiments will be described in detail below, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with the present application. They are merely examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0055] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood in their ordinary and common meanings.

[0056] In the present application, the terms "first", "second", "third", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar or like objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0057] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components does not have to be limited to all the components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0058] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or a combination of hardware or / and software code that can perform functions related to the element.

[0059] Figure 1 It is a schematic diagram of the operation scenario between a display device and a control device provided for some embodiments of the present application. As Figure 1 shown, the user can operate the display device 200 through the mobile terminal 300 and the control device 100.

[0060] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through network communication protocols, so as to achieve the purpose of one-to-one control operation and data communication. It is also possible to transmit the audio and video content displayed on the mobile terminal 300 to the display device 200 to achieve the synchronous display function.

[0061] As shown Figure 1 in the figure, the display device 200 also communicates with the server 400 through various communication methods for data communication. The display device 200 is allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0062] In addition to providing the function of receiving broadcast television, the display device 200 can also additionally provide an intelligent network television function that supports computer functions, including but not limited to, network television, smart television, Internet Protocol Television (IPTV), etc.

[0063] Figure 2 For some embodiments of this application Figure 1 is the hardware configuration block diagram of the display device 200 in the figure.

[0064] In some embodiments, the display device 200 includes at least one of a tuner demodulator 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0065] In some embodiments, the detector 230 is used to collect signals from the external environment or interact with the outside. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of environmental light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures. Or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0066] In some embodiments, the display 260 includes a display screen component for presenting a picture, and a driving component for driving image display, which is used to receive an image signal output from the controller, and display video content, image content, and a menu control interface component, as well as a user control UI interface, etc.

[0067] In some embodiments, the communicator 220 is a component for communicating with external devices or the server 400 according to various communication protocol types.

[0068] In some embodiments, the device interface 240 is a component for establishing a communication connection between the display device 200 and an external device 500. The device interface 240 includes, but is not limited to, any one or more of the following interfaces: High Definition Multimedia Interface (HDMI), analog or digital high-definition component input interface (component), composite video input interface (Composite Video Broadcast Signal, CVBS), Universal Serial Bus (USB) input interface, RGB port, etc. It can also be a composite input / output interface formed by the above multiple interfaces.

[0069] Among them, the connection between the display device 200 and the external device 500 means establishing a communication connection, and the display device 200 and the external device 500 that establish the communication connection serve as the receiving end (Sink end) and the sending end (Source end), respectively. For example, the external device 500 can be a set-top box. During the user's use of the set-top box, the set-top box can parse the signal data in the signal source, and output video data and audio data in real time, and send the video data and audio data to the display device 200, so that the display device 200 outputs the video data and audio data as a video picture and sound. At this time, the set-top box device serves as the sending end, and the display device 200 serves as the receiving end.

[0070] A communication connection can be established between the sending end and the receiving end through a specific interface to transfer data. For this purpose, data interfaces with the same interface specification and function should be provided on both the sending end and the receiving end. For example, as Figure 3 shown, both the display device 200 and the external device 500 are provided with a High Definition Multimedia Interface (HDMI). During use, the user can insert both ends of the HDMI interface data cable into the display device 200 and the external device 500 respectively, and after starting the external device 500 and the display device 200, set the signal source of the display device 200 to the HDMI interface, so as to realize data transmission between the display device 200 and the external device 500.

[0071] It should be noted that in order to achieve the communication connection between the display device 200 and the external device 500, other connection methods can also be adopted between the display device 200 and the external device 500. The specific connection method can be a wired connection method, such as a Digital Visual Interface (DVI), a Video Graphics Array (VGA), a USB interface, etc.; it can also be a wireless connection method, such as a wireless local area network, a Bluetooth connection, an infrared connection, etc. Different communication connection methods can adopt different information transfer protocols. For example, when using an HDMI interface to achieve the connection, the HDMI protocol can be used for data transmission.

[0072] In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0073] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, that is, the tuner demodulator 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0074] In some embodiments, the user can input a user command on the graphical user interface (GUI) displayed on the display 260, and then the user input interface receives the user input command through the graphical user interface (GUI).

[0075] In some embodiments, the user interface 280 is an interface that can be used to receive control inputs.

[0076] Figure 4 For some embodiments of the present application Figure 1 The hardware configuration block diagram of the control device in. As Figure 4 shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0077] The control device 100 is configured to control the display device 200, and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing an intermediary role in the interaction between the user and the display device 200.

[0078] In some embodiments, the control device 100 can be an intelligent device. For example, the control device 100 can install various applications for controlling the display device 200 according to user requirements.

[0079] In some embodiments, such as Figure 1 shown, after installing the application for controlling the display device 200, the mobile terminal 300 or other intelligent electronic devices can perform functions similar to those of the control device 100.

[0080] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between internal components and the data processing functions between the external and internal.

[0081] Under the control of the controller 110, the communication interface 130 realizes the communication of control signals and data signals with the display device 200. The communication interface 130 can include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, and other near - field communication modules.

[0082] The user input / output interface 140, where the input interface includes at least one of a microphone 141, a touchpad 142, a sensor 143, a button 144, and other input interfaces.

[0083] In some embodiments, the control device 100 includes at least one of the communication interface 130 and the input / output interface 140. When the communication interface 130 is configured in the control device 100, such as modules like WiFi, Bluetooth, and NFC, it can encode the user input instructions through the WiFi protocol, or the Bluetooth protocol, or the NFC protocol and send them to the display device 200.

[0084] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0085] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0086] Figure 5 For some embodiments provided in this application Figure 1 Schematic diagram of software configuration in the display device, such as Figure 5As shown, in some embodiments, the system is divided into four layers, from top to bottom are the Applications layer (abbreviated as the "application layer"), the Application Framework layer (abbreviated as the "framework layer"), the Android runtime and the System Libraries layer (abbreviated as the "system runtime layer"), and the kernel layer.

[0087] In some embodiments, at least one application runs in the application layer. These applications can be the window programs, system setting programs, or clock programs that come with the operating system; they can also be applications developed by third-party developers. In specific implementation, the application packages in the application layer are not limited to the above examples.

[0088] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that decides the actions of the applications in the application layer. Applications can access the resources in the system and obtain system services through the API interfaces during execution.

[0089] As Figure 5 shown, in the embodiment of the present application, the application framework layer includes a View System, Managers, a Content Provider, etc. Among them, the View System can design and implement the interfaces and interactions of applications. The View System includes lists, grids, text boxes, buttons, etc. The Managers include at least one of the following modules: the Activity Manager is used to interact with all the activities running in the system; the Location Manager is used to provide access to the system location service for system services or applications; the Package Manager is used to retrieve various information related to the application packages currently installed on the device; the Notification Manager is used to control the display and clearing of notification messages; the Window Manager is used to manage the icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0090] In some embodiments, the activity manager is used to manage the life cycles of various applications and the general navigation back function, such as controlling the exit, opening, and back of applications. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, jittering the display, distorting the display, etc.).

[0091] In some embodiments, the system runtime layer provides support for the upper layer, i.e., the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries included in the system runtime layer to implement the functions to be achieved by the framework layer.

[0092] In some embodiments, the kernel layer is the layer between hardware and software. As Figure 5 shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0093] In the above embodiments, the driver based on the device interface 240 can control the device interface 240 to implement image and audio transmission. For example, the HDMI driver can transmit the image signal from the source device to the sink device through an HDMI cable, and use the decoder built into the sink device to convert the signal into an image for display.

[0094] In some embodiments, based on the form of the device interface 240 of the display device 200, the display device 200 can also transmit control instruction data through the device interface 240. For example, as Figure 6 shown, when the HDMI interface of the display device 200 supports the protocol above HDMI 1.3a, the display device 200 can transmit control instructions through the Consumer Electronics Control (CEC) channel. The transmission method based on the HDMI CEC instruction realizes the control communication between the source device and the sink device through an HDMI connection.

[0095] Among them, the HDMI connector includes a CEC pin for transmitting CEC commands and data. When the user operates the remote control of the display device 200, the command signal generated by the remote control will be sent to the display device 200, and then the display device 200 will transmit the signal to the external device 500 through an HDMI cable, so that the external device 500 can perform corresponding operations according to the command signal, such as returning, switching the input source, adjusting the volume, etc., to realize the joint control of the display device 200 and the external device 500.

[0096] To achieve the interlocking control function with the external device 500, in some embodiments, the display device 200 can transmit CEC instruction signals through an HDMI cable. The HDMI cable and its connector contain a dedicated CEC pin. This pin is a key part for transmitting CEC instruction signals. When the user performs an interactive operation based on the display device 200, a control instruction is input based on the interactive operation action. The display device 200 can then generate a CEC instruction signal according to the CEC instruction rules. Then, the CEC instruction signal is transmitted to the external device 500 through the CEC pin of the HDMI cable. When the external device 500 also supports the CEC protocol, it can receive and parse this CEC instruction signal. Moreover, after parsing the CEC instruction signal, the external device 500 will perform corresponding operations according to the received CEC instruction. For example, if the control instruction is to switch the input source, the external device 500 will switch to the specified input source in response to the CEC instruction. If the control instruction is to adjust the volume, the external device 500 will perform the corresponding volume adjustment.

[0097] It can be seen that when controlling the external device 500 based on the HDMI CEC control method, the CEC instruction signal can be transmitted through the CEC pin built in the HDMI cable, so as to achieve the control of the external device 500. The HDMI CEC control method enables the user to control multiple other devices connected by the HDMI cable through the remote control of one device, providing a more convenient operation experience.

[0098] It should be noted that in the above embodiments, the user can use the control device 100 supporting the display device 200 to input control instructions to interlock and control the external device 500. And in some embodiments, the user can also input instructions through the remote control supporting the external device 500 to control the display device 200. That is, the interlocking control process when performing device interlocking control based on the device interface 240 can be bidirectional.

[0099] In some embodiments, after the display device 200 is connected to the external device 500 through the device interface 240, on the one hand, the external device 500 can receive the media signal of the signal source and convert the media signal into a media data stream and send it to the display device 200 through the device interface 240; on the other hand, the external device 500 can also send an interface data stream to the display device 200 through the device interface 240 to present the user interface of the external device 500 on the display device 200.

[0100] For example, as Figure 7As shown, when the display device 200 is connected to an external device 500 through an HDMI interface and the external device 500 is a set-top box or other device, the display device 200 can set the current signal source to the HDMI interface by setting the signal source. After the external device 500 is powered on and running, according to the set-top box operating system settings, it renders the set-top box control home page and sends the screen corresponding to the set-top box control home page to the display device 200. At this time, since the current signal source of the display device 200 is the HDMI interface, the display device 200 can obtain the frame of the screen corresponding to the set-top box control home page from the HDMI interface and display it on the display 260.

[0101] During the process of the display device 200 displaying the user interface of the external device 500, it can also obtain the control instructions input by the user along with the user's interaction operations and send them to the external device 500 through the device interface 240. The external device 500 then responds to the control instructions transmitted through the device interface 240 and performs the control actions corresponding to the control instructions. For the control instructions generated by some interaction operations, the external device 500 will produce corresponding interface changes when executing the control instructions. Therefore, it can send the changed interface content to the display device 200 along with the data stream in the device interface 240, so that the user interface displayed on the display device 200 changes synchronously.

[0102] For example, after a set-top box or other external device 500 is powered on, it will default to display the set-top box control home page. The set-top box control home page includes multiple media asset item controls and a focus cursor. The user can input a cursor movement instruction through the direction keys on the remote control supporting the display device 200. After the display device 200 transmits the cursor movement instruction to the set-top box through the HDMI cable, the set-top box can respond to the cursor movement instruction and change the display position of the focus cursor in the set-top box control home page. At this time, the set-top box can send the process of the focus cursor display position change and the interface content after the focus cursor display position change to the display device 200 so that the display device 200 can display it in real time.

[0103] In some embodiments, some control instructions can cause the user interface provided by the external device 500 to switch, that is, the external device 500 can respond to the control instructions that can cause the user interface to switch and switch from one interface to another. Along with the interface switching process, the display device 200 can also perform real-time display according to the interface data transmitted through the device interface 240.

[0104] For example, as Figure 8As shown, after the user moves the focus cursor to the position of the media asset item control corresponding to Media Asset A based on the set-top box control home page, and presses the confirmation key on the remote control supporting the display device 200, a control instruction for displaying the media asset details interface can be input to the display device 200. The display device 200 then sends the control instruction to the set-top box through the HDMI interface. The set-top box can respond to this control instruction, jump from the set-top box control home page to the media asset details interface, and send the transition animation during the jump process and the screen data corresponding to the media asset details interface after the jump to the display device 200 through the HDMI interface, so that the display device 200 changes from presenting the set-top box control home page interface to displaying the transition animation and the media asset details interface.

[0105] In some embodiments, since there is a hierarchical relationship between the user interfaces provided by the external device 500, when performing an interface jump, the external device 500 will respond to the control instruction and perform the jump according to the hierarchical relationship between the interfaces. For example, the set-top box can provide a set-top box control home page interface, a media asset details interface, and a playback interface respectively. These three interfaces have a hierarchical relationship, that is, when an acknowledgment instruction is input through the media asset item control based on the home page interface, it jumps to the media asset details interface, and when an acknowledgment instruction is input through the playback control based on the media asset details interface, it jumps to the playback interface. Conversely, when the user inputs a return instruction based on the playback interface, it will jump to the media asset details interface; when a return instruction is input based on the media asset details interface, it will jump to the control home page interface.

[0106] It can be seen that since there is a hierarchical relationship between some of the user interfaces provided by the external device 500, and when the display device 200 controls the external device 500, the control instruction will be transmitted to the external device 500 without limitation. Therefore, in some scenarios, the transmitted control instruction cannot be responded to by the external device 500, that is, the interface content cannot be changed. For example, when the display device 200 sends a return instruction corresponding to the return key to the external device 500 through the HDMI CEC channel, if the external device 500 has already returned to the set-top box control home page at the end of the hierarchical relationship, and the display device 200 still transmits the return instruction corresponding to the return key to the external device 500, the return key will not take effect at this time, that is, the set-top box will not change the interface content according to the return key, resulting in the display device 200 staying on the last control home page interface for a long time, reducing the user experience.

[0107] In order to alleviate the problem that the return instruction does not take effect in some scenarios, some embodiments of the present application provide a display device 200. The display device 200 can detect the change situation of the interface content provided by the external device 500, and respond to the return instruction according to different detection results to implement the co-control function with the external device 500.

[0108] Among them, to implement the joint control function, the display device 200 should at least include a display 260, a device interface 240, and a controller 250. Among them, the display 260 is configured to display a user interface, and the user interface includes the user interface provided by the external device 500 and the user interface provided by the display device 200. For the sake of description, in the embodiments of the present application, the user interface provided by the external device 500 is referred to as the first user interface. The user interface provided by the display device 200 is referred to as the second user interface. Therefore, the user interface includes the first user interface and / or the second user interface.

[0109] The device interface 240 is configured to connect to the external device 500, and the device interface 240 can transmit the interface data sent by the external device 500 to the display device 200. Moreover, the device interface 240 can also be used to transmit the control instructions during the joint control between the display device 200 and the external device 500. Therefore, the device interface 240 can be one or a combination of HDMI interface, component interface, CVBS interface, and USB interface.

[0110] Among them, when an interface can satisfy both transmitting interface data and control instructions at the same time, the display device 200 can establish a connection relationship with the external device 500 through one interface. For example, the display device 200 is connected to the external device 500 through an HDMI interface that supports the CEC function. When an interface cannot satisfy both transmitting interface data and control instructions at the same time, the display device 200 can establish a connection relationship with the external device 500 through two or more interfaces. For example, when the display device 200 is connected to the external device 500 through an HDMI interface that does not support the CEC function, since the current HDMI interface cannot transmit control instructions, the display device 200 can also establish a connection with the external device 500 through a USB interface to use the USB interface to transmit control instructions and use the HDMI interface to transmit interface data.

[0111] As Figure 9 shown, the controller 250 is configured to execute the steps corresponding to the method for controlling external devices in a joint manner, including the following:

[0112] S110: When the first user interface is included in the user interface displayed on the display, in response to the return instruction input by the user, take a screenshot of the current user interface to generate a current interface image.

[0113] After the display device 200 is connected to the external device 500, the interface content that can be displayed includes the following three forms, namely, displaying the first user interface entirely, displaying the second user interface entirely, and displaying a combined interface of the first user interface and the second user interface. For example, when the display device 200 is connected to a set-top box device through an HDMI interface and the user sets the signal source of the display device 200 to the HDMI interface, the display device 200 can display the set-top box control home page interface, media asset details interface, and set-top box media asset playback interface provided by the set-top box, etc., that is, the display device 200 can display the first user interface entirely. When the user sets the signal source of the display device 200 not to the HDMI interface, the display device 200 can display its own user interface, including the control home page of the display device, media asset recommendation interface, playback interface of the display device, settings interface, etc., that is, the display device 200 displays the second user interface entirely.

[0114] For some of the second user interfaces displayed by the display device 200, they may also include relevant content of the first user interface. For example, in the signal source switching interface of the display device 200, there may be signal source option controls for multiple device interfaces 240, and on each signal source option control or associated position, a preview window may be provided, and the preview window can display in real time the interface content provided by the external device 500 connected to each device interface 240. As Figure 10 shown, when the HDMI1 interface of the display device 200 is connected to a set-top box device, the set-top box control home page provided by the set-top box, that is, the first user interface, is displayed in the preview window of the signal source control corresponding to the HDMI1 interface.

[0115] When the user interface displayed by the display device 200 is the first user interface or includes the first user interface, the display device 200 can receive control instructions input by the user. Since the connection relationship between the display device 200 and the external device 500 has been established when the display device 200 displays the first user interface or the second user interface that includes the first user interface, the display device 200 can forward control instructions (or forward some control instructions) to the external device 500 according to the driver and protocol specifications corresponding to the device interface 240, so as to realize the co-control function with the external device 500.

[0116] The control instructions received by the display device 200 may have different control functions based on different input methods. In some embodiments, the control instructions may be used to control the display device 200 or an external device 500 to perform a return operation. For example, a user may input a return instruction by pressing the return key on the remote control that comes with the display device 200. When the user presses the return key, the remote control may send a wireless transmission (such as infrared, Bluetooth, radio frequency, etc.) signal carrying the key value of the return key to the display device 200. After receiving the wireless transmission signal carrying the key value of the return key, the display device 200 may read the wireless transmission signal and read out the key value of the return key for response.

[0117] In some embodiments, the return instruction may also be input through different interaction actions according to the interaction methods supported by the display device 200. For example, for a display device 200 that supports touch interaction operations, a user may input a return instruction based on the touch gesture of the display device 200, such as swiping up from the bottom of the display device 200. For a display device 200 that supports voice interaction functions, a user may input a return instruction by voice input such as "return to the previous level".

[0118] After the user inputs a return instruction, the display device 200 may perform an operation response based on the return instruction. When the display device 200 and the external device 500 are jointly controlled, the display device 200 may forward the specific control instruction input by the user to the external device 500. For example, after receiving the return instruction, the display device 200 may convert the return instruction into an instruction that conforms to the interaction specification of the external device 500 based on the HDMI CEC protocol, and transmit the return instruction to the external device 500 based on the HDMI CEC channel. The external device 500 then performs a return operation response based on the return instruction transmitted by the display device 200. However, when the external device 500 returns to the last-level user interface, after the return instruction is transmitted to the external device 500, the user interface of the external device 500 will not be changed, that is, the return operation will not take effect. Therefore, in some embodiments, in order to make the return operation still effective, the display device 200 may first determine the user interface content provided by the external device 500 before forwarding the return instruction to the external device 500 to determine whether the user interface provided by the external device 500 is the last-level user interface.

[0119] That is, the display device 200 may, in response to the return instruction, capture a screenshot of the current user interface to generate a current interface image. In some embodiments, the display device 200 may perform a screenshot operation on the currently displayed user interface by calling a built-in screenshot tool (application).

[0120] For example, the display device 200 can call the system's screenshot process through Android's developer API. That is, the display device 200 first calls the screenshot process through the following instructions: import android.graphics.Bitmap; import android.graphics.Canvas; import android.view.View; import android.widget.ImageView; Then, it obtains the View to be screenshot through the following instructions: View viewToCapture = findViewById(R.id.view_to_capture); Then, it creates a Bitmap object according to the following instructions to store the screenshot image: Bitmap bitmap = Bitmap.createBitmap(viewToCapture.getWidth(), viewToCapture.getHeight(), Bitmap.Config.ARGB_8888); Then, it creates a Canvas object according to the following instructions and uses the Bitmap object as its bitmap: Canvas canvas = new Canvas(bitmap); Then, it draws the View to be screenshot onto the Canvas through the viewToCapture.draw instruction: viewToCapture.draw(canvas); Finally, it saves the screenshot to the album of the display device 200 according to the following instructions: MediaStore.Images.Media.insertImage(getContentResolver(), bitmap, "Screenshot", null); catch (FileNotFoundException e) e.printStackTrace();

[0121] Based on the screenshot method provided in the above embodiments, the display device 200 can take a screenshot of the currently displayed user interface through a screenshot tool. Since the currently displayed user interface on the display device 200 includes the first user interface provided by the external device 500, an image of the interface provided by the external device 500 can be obtained after taking a screenshot of the current user interface. In some embodiments, the user interface displayed by the display device 200 may also be a user interface composed of a combination of the first user interface and the second user interface, that is, in the currently displayed user interface, a part of the area displays the first user interface provided by the external device 500, and another part of the area displays the second user interface provided by the display device 200. At this time, if a screenshot of the entire displayed user interface is taken, the content of the second user interface will affect the judgment of the content provided by the external device 500.

[0122] Therefore, as Figure 11 shown, before performing the screenshot operation, the display device 200 can first traverse the control types in the currently displayed user interface to determine whether the currently displayed user interface includes a playback window control (S1101). If the currently displayed user interface includes a playback window control, a screenshot is taken of the area where the playback window control is located to generate a current interface image (S1102). For example, when the display device 200 displays a signal source switching interface, the signal source switching interface may include a preview window for displaying the content of the HDMI1 interface, that is, the signal source switching interface includes a playback window control. At this time, the display device 200 can read the configuration file of the signal source switching interface, determine the display area of the playback window control from it, and then perform a screenshot based on this display area to obtain a current interface image of the content displayed by the playback window control, that is, obtain a current interface image that only contains the content of the first user interface.

[0123] If the currently displayed user interface does not include a playback window control, a screenshot is taken of the entire user interface area to generate a current interface image (S1103). When the user interface displayed by the display device 200 includes the content of the first user interface and the currently displayed user interface does not include a playback window control, it means that the display device 200 is currently displaying the first user interface provided by the external device 500 in full screen. Therefore, the display device 200 can obtain a current interface image that only contains the content of the first user interface by taking a screenshot of the entire user interface area.

[0124] In some embodiments, when the display device 200 obtains the current interface image, it may also not call the screenshot tool, but extract a specific image frame from the media asset data stream to obtain the current interface image. That is, as Figure 12As shown, when generating the current interface image, the display device 200 can obtain the media asset data stream transmitted by the device interface 240, and then capture the target image frame in the media asset data stream according to the current input time when the user inputs the return instruction, so as to generate the current interface image according to the target image frame.

[0125] Among them, the target image frame is the image frame displayed at the current input time or the adjacent time of the current input time. For example, if the user presses the return key at 18:00:05 on May 5, 2023, the display device 200 can obtain the current input time T1 = 2023-5-5-18:00:05 when the input return instruction is input, and then extract from the media asset data stream sent by the external device 500 the target image frame with the display timestamp of 2023-5-5-18:00:05:000 or the shortest time difference from the current input time 2023-5-5-18:00:05, that is, Pn(2023-5-5-18:00:05:035), as the current interface image.

[0126] In some embodiments, the display device 200 can also obtain the current interface image by extracting the display screen frame in the frame buffer of the current interface. For example, the user interface of the display device 200 is an overlay screen of a video layer and a graphics (On Screen Display, OSD) layer, that is, the image content of the user interface can be rendered and then form a display screen frame in the synthesis module. Among them, the synthesis module can store the screen frames in the frame buffer and the video buffer respectively, and form the screen content in the corresponding layers through the video process module. The media asset screen sent by the external device 500 forms the screen content through the video layer, while the image frames of other image screens (such as the controls and progress bars of the UI interface) form the screen content through the OSD layer (OSD plane). The display device 200 forms the final screen through the blend unit of the video process module for display. Therefore, the display device 200 can extract the screen frame of the final screen in the blend unit, or extract the screen frame in the video buffer corresponding to the video layer to generate the current interface image.

[0127] Since the frame of the final screen extracted by the mixing unit may include the screen content provided by the display device 200. For example, in the HDMI setting interface, it may include an HDMI signal display window and button controls around the HDMI signal display window. The HDMI signal display window displays the screen content provided by the HDMI device, and the button controls are the controls provided by the display device 200 for setting the HDMI connection. Based on the operation logic of the display device 200, the HDMI setting interface belongs to a user interface provided by the display device 200. Therefore, the display device 200 needs to respond to the user's key operation, and such an operation mode is also more in line with the user's usage habits.

[0128] Therefore, in order not to affect the operation logic of the user interface of the display device 200, the display device 200 may execute the inter-control function only when the first user interface provided by the external device 500 is displayed full screen and the inter-control function is enabled. That is, in some embodiments, the display device 200 may read the display mode and the inter-control switch state from the current setting information. Among them, the display mode is used to characterize the display ratio of the screen provided by the external device 500, including the full-screen mode and the window mode. The inter-control switch state is used to characterize whether the inter-control function is enabled, that is, the inter-control switch state includes the on state and the off state.

[0129] If the display mode is the full-screen mode and the inter-control switch state is the on state, extract the display screen frame from the frame buffer of the video layer, and generate the current interface image according to the display screen frame. For example, the display device 200 may read the current setting information. When it reads "display mode = full screen" in the current device information, it determines that the current display mode is the full-screen mode. At the same time, the display device 200 may read the switch state of the HDMI CEC function in the setting information. When it reads "HDMI CEC = on", it determines that the inter-control switch state is the on state. At this time, the display device 200 may determine that the currently displayed content is the screen content provided by the external device 500 and supports the inter-control function. Therefore, it can extract the current display screen frame from the video buffer of the video layer of the current user interface and convert the extracted display screen frame into a picture format to generate the current interface image.

[0130] It can be seen that in the above embodiments, the display device 200 can obtain the current interface image by calling the screenshot API interface, extracting the image frame from the media data stream, and extracting the display screen frame from the frame buffer. At the same time, by detecting the display mode and the inter-control switch state, the current interface image that can reflect the screen content provided by the external device 500 can be accurately obtained, which is convenient for more accurate judgment of the screen content and realizes more accurate inter-control.

[0131] S120: Obtain a historical interface image.

[0132] After the display device 200 obtains the current interface image through screenshot, it can also obtain a historical interface image, where the historical interface image is an image generated by taking a screenshot of the user interface before the return instruction. The display device 200 can set a resident screenshot process after connecting to the external device 500. This screenshot process can monitor the control instructions input by the user or the screen content of the media data sent by the external device 500. Moreover, when a specific control instruction input by the user or a specific change in the screen content is monitored, a screenshot operation can be executed, and the obtained screenshot image can be stored to generate a historical interface image.

[0133] In some embodiments, the historical interface image can be an image obtained by the display device 200 taking a screenshot of the user interface at that time when the user input the return instruction last time. That is, as Figure 13 shown, when the display device 200 executes the operation of obtaining the historical interface image, it can call the screenshot database, where the screenshot database is used to store the images generated by taking screenshots of the user interface when the return instruction is input. Similar to step S100, the display device 200 can, each time the user inputs the return instruction, respond to the return instruction, take a screenshot of the current user interface to obtain the interface image displayed at that time, and store the obtained interface image.

[0134] For the convenience of subsequent calls, the obtained interface images can form a screenshot database according to specific storage rules. For example, in the screenshot database, it can include the storage address of the screenshot file and the execution time of the screenshot operation. According to the execution time of the screenshot operation, the associated storage address of the screenshot file can be queried.

[0135] After calling the screenshot database, the display device 200 can obtain the current input time when the return instruction is input, and query the adjacent input time when the return instruction was input last time according to the current input time. For example, if the user presses the return key at 18:00:05 on May 5, 2023, the display device 200 can obtain the current input time T1 = 2023-5-5-18:00:05 when the return instruction is input, and then, according to the current input time T1, query in the screenshot database for the screenshot data of the execution time of the latest screenshot operation before T1, that is, the adjacent input time T0 = 2023-5-5-18:00:02 when the return instruction was input last time.

[0136] Then, extract the historical interface image from the screenshot database according to the adjacent input time, that is, extract the image generated by taking a screenshot of the user interface when the previous input return instruction was executed. For example, after querying that the adjacent input time T0 = 2023-5-5-18:00:02, the display device 200 can query the storage address corresponding to the screenshot image saved at time T0 in the screenshot database based on the adjacent input time T0, that is, " / storage / Pictures / Screenshots / ××20230505180002.JPG", so as to obtain the historical interface image.

[0137] In some embodiments, the historical interface image may be an image obtained by the display device 200 taking a screenshot of the switched user interface when it monitors that the first user interface switches the interface. For example, the display device 200 can identify the content of the key frames of the first user interface provided by the external device 500 based on an image recognition algorithm. When it is recognized that the first user interface switches from the playback details interface to the set-top box control home page interface, it can take a screenshot of the control home page interface after the interface switch and save the interface screenshot as the historical interface image.

[0138] S130: Calculate the similarity between the current interface image and the historical interface image.

[0139] After obtaining the current interface image and the historical interface image, the display device 200 can calculate the similarity between the current interface image and the historical interface image based on a preset similarity algorithm. Among them, the similarity is used to represent the content difference between the current interface image and the historical interface image, that is, the higher the similarity, the closer the content of the current interface image and the historical interface image; the lower the similarity, the greater the difference in the content of the current interface image and the historical interface image.

[0140] The display device 200 can calculate the similarity between the current interface image and the historical interface image based on methods such as pixel-level comparison, histogram comparison, feature comparison, and deep learning. Among them, pixel-level comparison calculates the similarity by comparing the pixel values of two images. Histogram comparison first calculates the histogram of each image and then compares these two histograms to calculate the similarity. Feature comparison is to extract features from each image and then compare these features. Feature extraction methods such as Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), etc. are used to determine the similarity. Deep learning uses, for example, Convolutional Neural Networks (CNN) to extract the features of an image and then compares these features to determine the similarity.

[0141] S140: If the similarity is less than the similarity threshold, send the return instruction to the external device through the device interface, and control the display to display the first user interface after the external device executes the return instruction.

[0142] After calculating the similarity between the current interface image and the historical interface image, the display device 200 can judge the content difference between the current interface image and the historical interface image based on the calculated similarity. If the similarity is less than the similarity threshold, that is, when the content difference between the current interface image and the historical interface image is large, it indicates that the return instruction transmitted by the display device 200 to the external device 500 can change the interface content of the first user interface. Therefore, the display device 200 can also continue to send the return instruction to the external device 500 through the device interface 240 so that the external device 500 can respond to the return operation corresponding to the return instruction.

[0143] After sending the return instruction to the external device 500, the display device 200 can continue to receive the interface data sent by the external device 500, so as to display the first user interface after the external device 500 executes the return instruction. For example, if the preset similarity threshold of the display device 200 is 90%, after calculating that the similarity between the current interface image and the historical interface image is 47.3%, the display device 200 can determine that the similarity between the current interface image and the historical interface image is less than the similarity threshold. At this time, the display device 200 can, in accordance with the HDMI CEC protocol, transmit the return instruction to the external device 500 through the HDMI interface. After receiving the return instruction, the external device 500 can perform a return operation, that is, switch from the playback interface to the media asset details interface, and send the interface data corresponding to the interface switching process and the media asset details interface after switching to the display device 200 through the HDMI interface, so that the display device 200 can display the interface switching process and the media asset details interface after switching.

[0144] S150: If the similarity is greater than or equal to the similarity threshold, execute the return instruction and control the display to display the second user interface.

[0145] By comparing the similarity with the similarity threshold, if the similarity is greater than or equal to the similarity threshold, that is, the content difference between the current interface image and the historical interface image is small, the display device 200 may not send the return instruction to the external device 500, but the display device 200 executes the return instruction and displays the second user interface after executing the return instruction.

[0146] Since the display device 200 displays the first user interface before executing the return instruction, the display device 200 can, by executing the return instruction, exit the display of the first user interface and instead display the user interface provided by itself. For example, when the display device 200 is displaying the set-top box control home page interface, it receives a return instruction input by the user. In response to the return instruction, the display device 200 can obtain the current interface image and the historical interface image. By calculating, the similarity between the current interface image and the historical interface image is 100%, that is, the interface content in both the current interface image and the historical interface image is the set-top box control home page interface. Since the similarity of 100% between the current interface image and the historical interface image is greater than the similarity threshold of 90%, the display device 200 does not send the return instruction to the external device 500, and the display device 200 executes the return instruction, that is, the display device 200 can exit the display of the set-top box control home page interface and switch to display the default home page interface of the operating system of the display device 200.

[0147] In some embodiments, when the display device 200 executes the return instruction, it may also control the display to display an exit prompt message in response to the return instruction (S1501). The exit prompt message is used to prompt the user whether to stop displaying the first user interface provided by the external device 500. For example, as Figure 14 shown, when it is determined that the similarity between the current interface image and the historical interface image is greater than the similarity threshold, the display device 200 may display a message pop-up window on the current interface. The message pop-up window may include a prompt message such as "Please press the confirmation key to exit the current HDMI device and return to the home page". The user may input an operation instruction based on this information.

[0148] During the process of displaying the exit prompt message, the display device 200 further monitors the operation instruction input by the user based on the exit prompt message (S1502). The operation instruction is a confirmation exit instruction or a cancellation exit instruction. For example, after the display device 200 displays a message pop-up window containing the exit prompt message, it may monitor the operation instruction input by the user based on the remote control. When the user presses the "confirmation key" of the remote control during the display of the message pop-up window by the display device 200, the user input confirmation exit instruction is monitored. On the contrary, when the user presses the "return key" of the remote control during the display of the message pop-up window by the display device 200, the user input cancellation exit instruction is monitored.

[0149] For different operation instructions input by the user, the display device 200 may execute different interaction methods. That is, if the operation instruction is a confirmation exit instruction, it controls the display to display a second user interface (S1503). For example, as Figure 15 shown, when the user presses the confirmation key based on the message pop-up window of the exit prompt message, it indicates that the user wants to control the display device 200 to exit the HDMI signal interface. Therefore, the display device 200 may display the control home page interface default of the operating system of the display device 200, that is, display the second user interface.

[0150] If the operation instruction is a cancellation exit instruction, it controls the display to display the first user interface after the external device executes the return instruction (S1504). For example, when the user presses the return key based on the message pop-up window of the exit prompt message, it indicates that the user does not want to control the display device 200 to exit the HDMI signal interface. Therefore, the return instruction may be sent to the external device 500 according to the return instruction transmission method in the above embodiments, and the external device 500 still executes the return instruction and displays the set-top box control home page interface after the external device 500 executes the return instruction, that is, the first user interface.

[0151] It should be noted that when the operation instruction input by the user is a cancel exit instruction, since it can be determined through the similarity between the current interface image and the historical interface image that the current return instruction will not cause a change in the interface of the external device 500, the display device 200 may also not transfer the return instruction to the external device 500 and directly cancel the operation function corresponding to the return instruction.

[0152] In some embodiments, when the display device 200 displays the second user interface, it may also close the data channel corresponding to the device interface, query the target user interface, and control the display to display the target user interface. Wherein, the target user interface is the second user interface defaultly displayed by the display device, or the second user interface displayed by the display device when the device interface is switched to the signal source.

[0153] For example, after the display device 200 determines that the similarity between the current interface image and the historical interface image is greater than or equal to the similarity threshold, the display device 200 itself responds to the return instruction, closes the HDMI channel interface, and queries the user interface displayed before switching to the HDMI channel signal source, that is, the signal source switching interface, and displays the signal source switching interface. If the display device 200 fails to query the user interface displayed before switching to the HDMI channel signal source, it may obtain the defaultly displayed user interface, that is, control the home page interface, and display the control home page interface.

[0154] As can be seen from the above embodiments, after receiving the return instruction input by the user, the display device 200 can obtain the current interface image by taking a screenshot of the current user interface, obtain the historical interface image, and calculate the similarity between the current interface image and the historical interface image, so as to determine whether the interface content of the current interface image and the historical interface image has changed according to the similarity between the current interface image and the historical interface image, that is, determine whether the external device 500 can effectively respond to the return instruction. When the similarity is greater than or equal to the similarity threshold, the display device 200 no longer transfers the return instruction to the external device 500, but executes the return instruction by itself and displays the second user interface after executing the return instruction. The display device 200 can change the control method in which the return key is given to the external device 500 without limit. When the external device 500 returns to the last control home page interface, according to the intention of the return key, the current HDMI channel is exited at the display device 200 end and returns to the control home page of the display device 200, improving the continuity of the control interaction and the user experience.

[0155] In some embodiments, such as Figure 16As shown, the user can manually set whether to enable the joint control function of the external device 500 through the setting item. When the joint control function of the external device 500 is set to off, the display device 200 will not transmit control instructions to the external device 500, and all control instructions input by the user will be responded to and executed by the display device 200. Therefore, there will be no problem that the return operation described in the above embodiment does not take effect. When the joint control function of the external device 500 is set to on, the display device 200 can transmit control instructions to the external device 500, and some of all control instructions input by the user will be responded to and executed by the external device 500. Therefore, there will be a problem that the return operation described in the above embodiment does not take effect.

[0156] Therefore, as Figure 17 shown, when the user sets whether to enable the joint control function of the external device 500, the display device 200 can also obtain the setting information of the joint control function item (S1701) before taking a screenshot of the current user interface to generate the current interface image. Among them, the setting information of the joint control function item is used to show whether the user enables the joint control function. In different setting states, there are different setting information contents.

[0157] If the setting information is the first information used to represent enabling the joint control function, the display device 200 can take a screenshot of the current user interface to generate the current interface image (S1702), calculate the similarity between the current interface image and the historical interface image according to the joint control method provided in the above embodiment, and then determine whether to transmit the return instruction to the external device 500 according to the comparison result between the similarity and the similarity threshold.

[0158] For example, when the display device 200 obtains the return instruction input by the user, it can read the user's setting item through the configuration file. Among them, the setting item "Joint control options" is used to indicate whether to enable the joint control function of the HDMI CEC protocol. When it is detected that "Joint control options = ON", that is, when the setting information is detected as the first information "ON", the current user interface can be taken a screenshot in the manner described in the above embodiment to generate the current interface image.

[0159] If the setting information is the second information for characterizing the closing of the joint control function, that is, the user has not enabled the joint control function of the external device 500, the display device 200 executes the return instruction (S1703), and controls the display to display the second user interface after executing the return instruction (S1704). For example, if it is detected that "Joint control options=OFF", that is, the setting information is the second information "OFF", the joint control method in the above embodiment can be skipped, and the return instruction can be directly executed to jump back from the set-top box playback interface to the control home interface of the display device 200, and the second user interface after executing the return instruction, that is, the control home interface of the display device 200, is displayed through the display 260.

[0160] For the external device 500, during the interface switching process, generally, the user needs to perform multiple return operations to return to the final control home interface, and it is necessary for the user to perform multiple return operations to determine whether the content of the first user interface has changed. For example, if the user sequentially enters the "control home interface, media asset details interface, playback interface" through interactive operations, then when performing the interface return operation, the return key needs to be pressed continuously 3 times to determine whether the user interface is the final control home interface. And the time intervals between these 3 consecutive return key operations are relatively short. Therefore, the display device 200 can determine whether the user exits the device interface interface based on the input time interval of the return instruction.

[0161] That is, as Figure 18 shown, in some embodiments, the display device 200 can obtain the first screenshot time of the current interface image (S1801), and obtain the second screenshot time of the historical interface image (S1802), and then calculate the screenshot time interval between the first screenshot time and the second screenshot time (S1803). If the screenshot time interval is less than the interval threshold, it indicates that the user continuously inputs the return instruction within a short time, so the step of calculating the similarity between the current interface image and the historical interface image can be executed (S1804). If the screenshot time interval is greater than or equal to the interval threshold, it indicates that the user does not continuously input the return instruction within a short time, that is, the user has no intention of exiting the device interface interface. Therefore, the display device 200 can execute the step of sending the return instruction to the external device through the device interface (S1805), and continue to perform joint control with the external device 500.

[0162] For example, when the user wants to control the exit of the HDMI device, the user will continuously press the return key, and the pressing interval of the return key is less than 1 minute. Therefore, the display device 200 can set the interval threshold to 1 minute. When the user presses the return key at time T2, the display device 200 can obtain the time T1 when the return key was pressed last time, and calculate the time interval T = T2 - T1 between the two presses of the return key. Then, it is judged against the preset interval threshold.

[0163] If the time interval T = 3 minutes, which is greater than the interval threshold of 1 minute, it indicates that the user is only performing a normal return operation and has no intention of exiting the HDMI device. Therefore, according to the linked control function of the external device 500, a return instruction can be sent to the external device 500 through the HDMI CEC channel. If the time interval T = 2 seconds, which is less than the interval threshold of 1 minute, it indicates that the user has the intention of exiting the HDMI device. Therefore, according to the method for linked control of the external device 500 provided in the above embodiment, a screenshot of the current interface can be taken to generate a current interface image, and based on the current interface image and the historical interface image, it is determined whether to send a return instruction to the external device 500 through the HDMI CEC channel.

[0164] Based on the above display device 200, some embodiments of the present application also provide a method for linked control of an external device. The method for linked control of the external device includes:

[0165] S110: When the user interface displayed on the display includes a first user interface, in response to a return instruction input by the user, take a screenshot of the current user interface to generate a current interface image, where the first user interface is a user interface provided by the external device;

[0166] S120: Obtain a historical interface image, where the historical interface image is an image generated by taking a screenshot of the user interface before the return instruction;

[0167] S130: Calculate the similarity between the current interface image and the historical interface image;

[0168] S140: If the similarity is less than the similarity threshold, send the return instruction to the external device through the device interface, and control the display to display the first user interface after the external device executes the return instruction;

[0169] S150: If the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display a second user interface, where the second user interface is a user interface provided by the display device.

[0170] As can be seen from the above technical solutions, some embodiments of the present application provide a display device and an external device joint control method. When the first user interface provided by the external device is included in the user interface displayed by the display device, in response to the return instruction input by the user, the current user interface is captured to generate a current interface image. The historical interface image generated by capturing the user interface before the return instruction is obtained, and then the similarity between the current interface image and the historical interface image is calculated. If the similarity is less than the similarity threshold, the return instruction is sent to the external device through the device interface to display the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, the display device executes the return instruction and displays the second user interface. The method can realize that on the basis of the effective joint control, when the external device returns to the last interface, the return instruction can exit the logic of the external device interface, improving the continuity of the interaction to solve the problem that the return operation is ineffective when the external device returns to the last home page interface.

[0171] In the above embodiment, when the display device 200 includes the first user interface provided by the external device 500 in the displayed user interface, in response to the return instruction, the current interface image and the historical interface image are obtained, and whether to pass the return instruction to the external device 500 for instruction response is determined according to the similarity between the current interface image and the historical interface image. In some embodiments, the display device 200 can also transmit the return instruction to the external device 500 when receiving the return instruction, and determine whether to exit the device interface signal interface according to the user interface content provided by the external device 500 after responding to the return instruction. That is, in some embodiments of the present application, a display device 200 is also provided. Similar to the above embodiment, the display device 200 provided in this embodiment also includes: a display 260, a device interface 240, and a controller 250. Among them, the display 260 is configured to display a user interface, and the user interface is the first user interface or the second user interface. The first user interface is the user interface provided by the external device; the second user interface is the user interface provided by the display device. The device interface 240 is configured to connect to an external device. As Figure 19 shown, the controller 250 is configured to perform the following steps:

[0172] S210: When the first user interface is included in the user interface displayed by the display, in response to the return instruction input by the user, capture the current user interface to generate a first interface image;

[0173] S220: Send the return instruction to the external device through the device interface, and obtain the second interface image after the external device executes the return instruction;

[0174] S230: Calculate the similarity between the first interface image and the second interface image;

[0175] S240: If the similarity is less than the similarity threshold, control the display to display the first user interface after the external device executes the return instruction;

[0176] S250: If the similarity is greater than or equal to the similarity threshold, execute the return instruction and control the display to display the second user interface.

[0177] For example, when the display device 200 displays the playback interface provided by the external device 500, if the user inputs a return instruction through the return key of the remote control supporting the display device 200, then after obtaining the return instruction input by the user, the display device 200 can capture a screenshot of the current user interface to generate the first interface image. At the same time, the return instruction is transmitted to the external device 500 through the HDMI CEC channel.

[0178] After receiving the return instruction, the external device 500 will execute the return instruction to perform an interface switch, switching from the playback interface to the media asset details interface, and sending the interface data of the switched media asset details interface to the display device 200 through the HDMI interface. The display device 200 then performs a screenshot operation based on the received interface data to obtain the second interface image provided by the external device 500 after executing the return instruction.

[0179] Then calculate the similarity between the first interface image and the second interface image, and determine whether the external device 500 has made an effective interface response to the return instruction based on the similarity between the first interface image and the second interface image. That is, when the external device 500 switches from the playback interface to the media asset details interface, the similarity between the first interface image and the second interface image is less than the similarity threshold. Therefore, the display device 200 can display the media asset details interface after the external device 500 executes the return instruction, that is, display the first user interface.

[0180] For another example, when the display device 200 displays the set-top box control home page interface provided by the external device 500, if the user inputs a return instruction through the return key of the remote control supporting the display device 200, then the display device 200 can obtain the first interface image according to the above example and transmit the return instruction to the external device 500 through the HDMI CEC channel. The external device 500 then performs an interface switch after receiving the return instruction. Since the control home page interface is already the last interface, when the external device 500 executes the return instruction, the interface content will not change, that is, the content of the second interface image obtained by the display device 200 after the external device 500 executes the return instruction is still the set-top box control home page interface.

[0181] At this time, the display device 200 calculates the similarity between the first interface image and the second interface image according to the above example. The similarity is greater than the similarity threshold. Therefore, the display device 200 can execute the return instruction, exit the HDMI interface, and display the default control home interface of the display device 200, that is, the second user interface.

[0182] Based on the display device 200 provided in the above embodiment, some embodiments of the present application further provide an external device joint control method, and the external device joint control method includes:

[0183] S210: When the user interface displayed on the display includes a first user interface, in response to a return instruction input by the user, take a screenshot of the current user interface to generate a first interface image, where the first user interface is a user interface provided by an external device;

[0184] S220: Send the return instruction to the external device through the device interface, and obtain a second interface image after the external device executes the return instruction;

[0185] S230: Calculate the similarity between the first interface image and the second interface image;

[0186] S240: If the similarity is less than the similarity threshold, control the display to display the first user interface after the external device executes the return instruction;

[0187] S250: If the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display a second user interface, where the second user interface is a user interface provided by the display device.

[0188] As can be seen from the above technical solutions, some embodiments of the present application provide a display device 200 and an external device joint control method. When the method includes a first user interface in the user interface displayed by the display device 200, in response to a return instruction input by the user, a screenshot of the current user interface is taken to generate a first interface image. Then, a return instruction is sent to the external device through the device interface, and a second interface image after the external device executes the return instruction is obtained. Then, the similarity between the first interface image and the second interface image is calculated. When the similarity is less than the similarity threshold, the first user interface after the external device executes the return instruction is displayed; when the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and the second user interface is displayed. The second user interface is the user interface provided by the display device. The method can determine whether to return to the subsequent interface by detecting the similarity of the user interfaces provided by the external device 500 before and after executing the return instruction, and when it is determined to return to the last interface, the logic of the external device interface can be exited in response to the return instruction, improving the continuity of the interaction, so as to solve the problem that the return operation is ineffective when the external device returns to the last home page interface.

[0189] The same or similar parts among the various embodiments in this specification can be referred to each other, and will not be repeated here.

[0190] Those skilled in the art can clearly understand that the technologies in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods in various embodiments or some parts of the embodiments of the present invention.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0192] For ease of explanation, the above description has been presented in connection with specific embodiments. However, the foregoing exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Numerous modifications and variations are possible in light of the above teachings. The selection and description of the above embodiments were made to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations suitable for specific use considerations.

Claims

1. A display device, characterized in that, comprising: a display configured to display a user interface, the user interface including a first user interface and / or a second user interface, the first user interface being a user interface provided by an external device; the second user interface being a user interface provided by the display device; a device interface configured to connect to an external device; a controller configured to: when the user interface displayed on the display includes a first user interface, in response to a return instruction input by a user, take a screenshot of the current user interface to generate a current interface image; acquire a historical interface image, the historical interface image being an image generated by taking a screenshot of the user interface before the return instruction; calculate the similarity between the current interface image and the historical interface image; if the similarity is less than a similarity threshold, send the return instruction to the external device through the device interface, and control the display to display the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display the second user interface.

2. The display device according to claim 1, characterized in that, when the controller executes to acquire the historical interface image, it is further configured to: invoke a screenshot database for storing images generated by taking screenshots of the user interface when a return instruction is input; acquire the current input time when the return instruction is input; query the adjacent input time when the previous return instruction was input according to the current input time; extract a historical interface image from the screenshot database according to the adjacent input time, the historical interface image being an image generated by taking a screenshot of the user interface when the previous return instruction was input.

3. The display device according to claim 1, characterized in that, when the controller executes to generate the current interface image, it is further configured to: acquire the media data stream transmitted by the device interface; grab a target image frame from the media data stream according to the current input time when the user inputs the return instruction, the target image frame being an image frame displayed at the current input time or a time adjacent to the current input time; generate a current interface image according to the target image frame.

4. The display device according to claim 1, characterized in that, when the controller executes to generate the current interface image, it is further configured to: read the display mode and the status of the linked control switch from the current setting information; if the display mode is the full-screen mode and the status of the linked control switch is the on state, extract a display picture frame from the frame buffer of the video layer; generate a current interface image according to the display picture frame.

5. The display device according to claim 1, characterized in that, when the controller executes the return instruction, it is further configured to: in response to the return instruction, control the display to display an exit prompt message; monitor an operation instruction input by the user based on the exit prompt message, the operation instruction being a confirmation exit instruction or a cancellation exit instruction; if the operation instruction is a confirmation exit instruction, control the display to display the second user interface; If the operation instruction is a cancellation and exit instruction, control the display to display the first user interface after the external device executes the return instruction.

6. The display device according to claim 1, wherein, before the controller performs a screenshot of the current user interface to generate a current interface image, it is further configured to: Obtain the setting information of the joint control function item; If the setting information is the first information, perform a screenshot of the current user interface to generate a current interface image, and the first information is used to represent the activation of the joint control function; If the setting information is the second information, execute the return instruction, and control the display to display the second user interface after executing the return instruction, and the second information is used to represent the deactivation of the joint control function.

7. The display device according to claim 1, wherein, the controller is further configured to: Obtain the first screenshot time of the current interface image and the second screenshot time of the historical interface image; Calculate the screenshot time interval between the first screenshot time and the second screenshot time; If the screenshot time interval is less than the interval threshold, perform the step of calculating the similarity between the current interface image and the historical interface image; If the screenshot time interval is greater than or equal to the interval threshold, perform the step of sending the return instruction to the external device through the device interface.

8. The display device according to claim 1, wherein, when the controller controls the display to display the second user interface, it is further configured to: Close the data channel corresponding to the device interface; Query the target user interface, where the target user interface is the second user interface defaultly displayed by the display device, or the second user interface displayed by the display device when the device interface is switched to the signal source; Control the display to display the target user interface.

9. A display device, wherein, comprising: A display configured to display a user interface, where the user interface is a first user interface or a second user interface, the first user interface is a user interface provided by an external device; the second user interface is a user interface provided by the display device; A device interface configured to connect to an external device; A controller configured to: When the user interface displayed on the display includes the first user interface, in response to a return instruction input by the user, perform a screenshot of the current user interface to generate a first interface image; Send the return instruction to the external device through the device interface, and obtain a second interface image after the external device executes the return instruction; Calculate the similarity between the first interface image and the second interface image; If the similarity is less than the similarity threshold, control the display to display the first user interface after the external device executes the return instruction; If the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display the second user interface.

10. An external device joint control method, wherein, Applied to the display device according to any one of claims 1-8, the display device includes a display, a device interface, and a controller. The display device is connected to an external device through the device interface. The method for jointly controlling the external device includes: When a first user interface is included in the user interface displayed on the display, in response to a return instruction input by the user, a screenshot of the current user interface is taken to generate a current interface image. The first user interface is a user interface provided by the external device; Obtain a historical interface image, which is an image generated by taking a screenshot of the user interface before the return instruction; Calculate the similarity between the current interface image and the historical interface image; If the similarity is less than the similarity threshold, send the return instruction to the external device through the device interface, and control the display to display the first user interface after the external device executes the return instruction; If the similarity is greater than or equal to the similarity threshold, execute the return instruction, and control the display to display a second user interface, where the second user interface is a user interface provided by the display device.

Images