Display device and external device joint control method

Abstract

Embodiments of the present disclosure provide a display device and external device joint control method. The method comprises: when a user interface displayed by a display device comprises a first user interface provided by an external device, in response to a return instruction, generating a current interface image by capturing a screenshot; acquiring a historical interface image, and then calculating a similarity between the current interface image and the historical interface image; if the similarity is less than a similarity threshold, sending the return instruction to the external device by means of an apparatus interface to display the first user interface after the external device executes the return instruction; and otherwise, the display device executing the return instruction and displaying a second user interface. The process implements the logic that, on the basis of effective joint control, when the external device returns to the last interface, the return instruction can exit the interface of the external device, thereby improving the interaction continuity and thus solving the problem that a return operation fails to take effect when the external device returns to the last homepage interface.

Description

Display device and external device joint control method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese application No. 202311703472.9, filed on December 12, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the technical field of display devices, and in particular to a method for jointly controlling a display device and an external device. Background Art

[0004] A display device is an intelligent device that can present a user interface and support user interaction. A device interface can be provided on the display device, and the display device can be connected to an external device through the device interface to realize data interaction with the external device. For example, the display device can be connected to an external device such as a set-top box through a High Definition Multimedia Interface (HDMI). After connecting to an external device such as a set-top box, the display device can obtain the signal source provided by the set-top box through the HDMI interface to display the media image corresponding to the signal source.

[0005] Some display devices also support device interfaces that control external devices. For example, if a display device's HDMI interface supports the Consumer Electronics Control (CEC) channel, the display device can transmit control commands to the external device via the HDMI CEC channel. If the external device supports the CEC protocol, the external device can respond to the control commands sent by the display device, such as the operation of the arrow keys, OK, and Return keys, allowing the display device remote control to control the set-top box to switch input sources, adjust the volume, and so on.

[0006] However, when a display device controls an external device, control commands are passed to the external device without restriction. As a result, in some scenarios, the passed control commands cannot be responded to by the external device to change the interface content. For example, when the display device sends the return command corresponding to the return key to the external device through the HDMI CEC channel, the external device has already returned to the last home page interface, and the display device still passes the return key to the external device. In this case, the return key will not take effect, and the display device will remain on the last home page interface for a long time, which reduces the user experience.

[0007] Summary of the Invention

[0008] In a first aspect, an embodiment of the present disclosure discloses a display device, which may include: a display, which may be configured to display a user interface, wherein the user interface includes a first user interface provided by an external device and / or a second user interface provided by the display device; a device interface, which may be configured to connect to an external device; a memory, which may be configured to store computer instructions and data associated with the display device; at least one processor, connected to the display, the device interface and the memory, and may be configured to execute computer instructions to cause the display device to execute: when the first user interface is included in the user interface displayed on the display, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a current interface image; obtaining a historical interface image, wherein the historical interface image is an image generated by taking a screenshot of the user interface before the user inputs the return instruction; calculating the similarity between the current interface image and the historical interface image; if the similarity is less than a similarity threshold, sending the return instruction to the external device through the device interface, and controlling the display to display the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, executing the return instruction, and controlling the display to display the second user interface.

[0009] In a second aspect, an embodiment of the present disclosure discloses a display device, which may include: a display, which may be configured to display a user interface, wherein the user interface includes a first user interface provided by an external device and / or a second user interface provided by the display device; a device interface, which may be configured to connect to an external device; a memory, which may be configured to store computer instructions and data associated with the display device; at least one processor, connected to the display, the device interface and the memory, and may be configured to execute computer instructions to cause the display device to execute: when the user interface displayed on the display includes a first user interface, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a first interface image; sending the return instruction to the external device through the device interface, and obtaining a second interface image after the external device executes the return instruction; calculating the similarity between the first interface image and the second interface image; if the similarity is less than a similarity threshold, controlling the display to display the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, executing the return instruction, and controlling the display to display the second user interface.

[0010] In a third aspect, an embodiment of the present disclosure discloses a method for joint control of an external device, which is applied to the display device of the first aspect and may include: when a first user interface is included in the displayed user interface, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a current interface image, wherein the first user interface is a user interface provided by an external device, and the display device is connected to the external device through a device interface; obtaining a historical interface image, wherein the historical interface image is an image generated by taking a screenshot of the user interface before the return instruction; calculating the similarity between the current interface image and the historical interface image; if the similarity is less than a similarity threshold, sending the return instruction to the external device through the device interface, and displaying the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, executing the return instruction, and displaying a second user interface, wherein the second user interface is a user interface provided by the display device.

[0011] In a fourth aspect, an embodiment of the present disclosure discloses a method for joint control of an external device, which is applied to the display device of the first aspect and may include: when a first user interface is included in the displayed user interface, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a first interface image; the first user interface is a user interface provided by an external device, and the display device is connected to the external device through a device interface; sending the return instruction to the external device through the device interface, and obtaining a second interface image after the external device executes the return instruction; calculating the similarity between the first interface image and the second interface image; if the similarity is less than a similarity threshold, displaying the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, executing the return instruction, and displaying the second user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG1 is a schematic diagram of a usage scenario of a display device according to some embodiments;

[0013] FIG2 is a block diagram of a hardware configuration of a display device according to some embodiments;

[0014] FIG3 is a schematic diagram of a connection relationship between a display device and an external device according to some embodiments;

[0015] FIG4 is a block diagram of a hardware configuration of a control device according to some embodiments;

[0016] FIG5 is a schematic diagram of software configuration in a display device according to some embodiments;

[0017] FIG6 is a schematic diagram of HDMI CEC transmission functions according to some embodiments;

[0018] FIG7 is a schematic diagram of a set-top box control homepage interface according to some embodiments;

[0019] FIG8 is a schematic diagram of HDMI signal interface switching according to some embodiments;

[0020] FIG9 is a flow chart of a method for joint control of an external device according to some embodiments;

[0021] FIG10 is a schematic diagram of a signal source switching interface according to some embodiments;

[0022] FIG11 is a flowchart of taking screenshots based on playback window controls according to some embodiments;

[0023] FIG12 is a flowchart based on a screenshot of a media asset data stream according to some embodiments;

[0024] FIG13 is a flow chart of obtaining a history interface image according to some embodiments;

[0025] FIG14 is a schematic diagram showing the display effect of the exit prompt information according to some embodiments;

[0026] FIG15 is a flowchart of monitoring user operation instructions according to some embodiments;

[0027] FIG16 is a schematic diagram of a joint control interface according to some embodiments;

[0028] FIG17 is a flowchart illustrating executing a return instruction according to joint control function item setting information according to some embodiments;

[0029] FIG18 is a flowchart illustrating executing a return instruction according to a screenshot time interval according to some embodiments;

[0030] FIG19 is another flowchart of a method for joint control of an external device according to some embodiments. DETAILED DESCRIPTION

[0031] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device. As shown in Figure 1 , a user can operate a display device 200 via a mobile terminal 300 and a control device 100 .

[0032] In some embodiments, the mobile terminal 300 can install software applications on the display device 200 and connect to the display device 200 through a network communication protocol to achieve one-to-one control operations and data communication. It is also possible to transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronous display.

[0033] The display device 200 can communicate data with the server 400 through various communication methods. For example, the display device 200 can allow the display device 200 to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0034] In addition to providing broadcast reception television functions, the display device 200 may also provide intelligent network television functions with computer support functions, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0035] Figure 2 is a hardware configuration block diagram of the display device 200 shown in Figure 1. As shown in Figure 2, the display device 200 may include at least one of a tuner and demodulator 210, a communication device 220, a detector 230, a device interface 240, a processor 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0036] In some embodiments, detector 230 can be used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, such as a sensor for collecting ambient light intensity; or detector 230 may include an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 may include a sound collector, such as a microphone, for receiving external sounds.

[0037] In some embodiments, the display 260 may include a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0038] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types.

[0039] In some embodiments, the device interface 240 is a component used to establish a communication connection between the display device 200 and the external device 500. The device interface 240 includes, but is not limited to, any one or more interfaces such as a High Definition Multimedia Interface (HDMI), an analog or digital high-definition component input interface (component), a composite video broadcast signal (CVBS) input interface, a Universal Serial Bus (USB) input interface, and an RGB port. Alternatively, the device interface 240 may be a composite input / output interface formed by multiple of the aforementioned interfaces.

[0040] Among them, the connection between the display device 200 and the external device 500 refers to establishing a communication connection, and the display device 200 and the external device 500 that establish the communication connection serve as a receiving end (Sink end) and a transmitting end (Source end), respectively. For example, the external device 500 can be a set-top box. When the user uses the set-top box, the set-top box can parse the signal data in the signal source and output video data and audio data in real time, and send the video data and audio data to the display device 200, so that the video data and audio data are output as video images and sounds through the display device 200. In this case, the set-top box device serves as a transmitting end, and the display device 200 serves as a receiving end.

[0041] A communication connection can be achieved between the transmitting end and the receiving end through a specific interface to transmit data. To this end, a data interface with the same interface specifications and functions should be provided on both the transmitting end and the receiving end. For example, as shown in FIG3 , a High Definition Multimedia Interface (HDMI) is provided on both the display device 200 and the external device 500. During use, the user can plug the two ends of the HDMI interface data cable into the display device 200 and the external device 500 respectively, and after starting the external device 500 and the display device 200, set the signal source of the display device 200 to the HDMI interface, thereby realizing data transmission between the display device 200 and the external device 500.

[0042] It should be noted that, in order to achieve a communication connection between the display device 200 and the external device 500, other connection methods can be used between the display device 200 and the external device 500. The specific connection method can be a wired connection method, such as a digital visual interface (DVI), a video graphics array (VGA), a USB interface, etc.; or it can be a wireless connection method, such as a wireless local area network, a Bluetooth connection, an infrared connection, etc. Different communication connection methods can use different information transmission protocols. For example, when an HDMI interface is used for connection, the HDMI protocol can be used for data transmission.

[0043] In some embodiments, the processor 250 may include a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output. The processor 250 may control the operation of the display device and respond to user operations using various software control programs stored in the memory. The processor 250 may control the overall operation of the display device 200.

[0044] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

[0045] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the device interface receives the user input command through the graphical user interface (GUI).

[0046] In some embodiments, the user input interface 280 is an interface that can be used to receive control input.

[0047] Fig. 4 is a block diagram of the hardware configuration of the control device 100 shown in Fig. 1. As shown in Fig. 4, the control device 100 may include a processor 110, a communication device 130, a user input / output interface, a memory, and a power supply.

[0048] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0049] In some embodiments, the control device 100 may be a smart device. For example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0050] In some embodiments, as shown in FIG. 1 , the mobile terminal 300 or other intelligent electronic device can perform similar functions as the control device 100 after installing an application for controlling the display device 200 .

[0051] The processor 110 may include a RAM 113 and a ROM 114, a communication device 130, and a communication bus. The processor 110 is used to control the operation and operation of the control device 100, as well as the communication and coordination between internal components and external and internal data processing functions.

[0052] Under the control of the processor 110, the communication device 130 communicates control signals and data signals with the display device 200. The communication device 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, or other near field communication modules.

[0053] The user input / output interface 140 includes at least one of a microphone 141 , a touch panel 142 , a sensor 143 , a button 144 and other input interfaces.

[0054] In some embodiments, the control device 100 includes at least one of a communication device 130 and an input / output interface 140. The control device 100 is configured with the communication device 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via the WiFi protocol, Bluetooth protocol, or NFC protocol and send them to the display device 200.

[0055] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the processor. The memory 190 can store various control signal instructions input by the user.

[0056] The power supply 180 is used to provide operating power support for various components of the control device 100 under the control of the processor.

[0057] FIG5 is a block diagram of the software configuration of display device 200 shown in FIG1. ​​As shown in FIG5, the software system of display device 200 can be divided into four layers: from top to bottom, the application layer (hereinafter referred to as the "application layer"), the application framework layer (hereinafter referred to as the "framework layer"), the Android runtime and system library layer (hereinafter referred to as the "system runtime library layer"), and the kernel layer.

[0058] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0059] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0060] Next, as shown in Figure 5, the application framework layer may include a view system, managers, content providers, and more. The view system, which designs and implements the application interface and interactions, includes lists, grids, text boxes, buttons, and more. The manager includes at least one of the following modules: an activity manager, which interacts with all running activities in the system; a location manager, which provides system services or applications with access to the system's location service; a package manager, which retrieves various information related to application packages currently installed on the device; a notification manager, which controls the display and removal of notification messages; and a window manager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0061] In some embodiments, the activity manager can be used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and back of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (for example, shrinking the display window, shaking the display, distorting the display, etc.).

[0062] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0063] In some embodiments, the kernel layer is a layer between hardware and software. Continuing with FIG5 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (e.g., a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0064] In the above embodiment, the driver based on the device interface 240 can control the device interface 240 to realize image and audio transmission. For example, the HDMI driver can transmit the image signal from the source end device to the sink end device via the HDMI cable, and use the built-in decoder of the sink end device to convert the signal into an image screen for display.

[0065] In some embodiments, based on the form of the device interface 240 of the display device 200, the display device 200 can also transmit control command data through the device interface 240. For example, as shown in Figure 6, when the HDMI interface of the display device 200 supports the HDMI 1.3a or higher version protocol, the display device 200 can transmit control commands through the Consumer Electronics Control (CEC) channel. The transmission method based on HDMI CEC commands is to achieve control communication between the source device and the sink device through the HDMI connection.

[0066] The HDMI connector includes a CEC pin for transmitting CEC commands and data. When a user operates the remote control of display device 200, the remote generates a command signal that is sent to display device 200. Display device 200 then transmits the signal to external device 500 via the HDMI cable, allowing external device 500 to perform corresponding operations based on the command signal, such as returning, switching input sources, and adjusting volume, thus achieving joint control of display device 200 and external device 500.

[0067] To enable joint control with the external device 500, in some embodiments, the display device 200 can transmit CEC command signals via an HDMI cable. The HDMI cable and its connector contain a dedicated CEC pin. This pin is a key component for transmitting CEC command signals. When a user interacts with the display device 200, they input control commands based on the interaction. The display device 200 then generates a CEC command signal based on the CEC command rules. This CEC command signal is then transmitted to the external device 500 via the CEC pin of the HDMI cable. If the external device 500 also supports the CEC protocol, it can receive and interpret this CEC command signal. After interpreting the CEC command signal, the external device 500 performs the corresponding operation according to the received CEC command. For example, if the control command is to switch the input source, the external device 500 will switch to the specified input source in response to the CEC command. If the control command is to adjust the volume, the external device 500 will adjust the volume accordingly.

[0068] As can be seen, when the external device 500 is controlled in a linked manner based on the HDMI CEC control method, the CEC command signal can be transmitted through the CEC pin built into the HDMI cable, thereby achieving control of the external device 500. The HDMI CEC control method allows users to control multiple other devices connected to the HDMI cable using the remote control of one device, providing a more convenient operating experience.

[0069] It should be noted that in the above embodiment, the user can use the control device 100 associated with the display device 200 to input control commands to jointly control the external device 500. In some embodiments, the user can also input commands to control the display device 200 using a remote control associated with the external device 500. In other words, the joint control process of devices based on the device interface 240 can be bidirectional.

[0070] In some embodiments, after the display device 200 is connected to the external device 500 through the device interface 240, on the one hand, the external device 500 can receive the media signal of the signal source, and convert the media signal into a media data stream and send it to the display device 200 through the device interface 240; on the other hand, the external device 500 can also send an interface data stream to the display device 200 through the device interface 240 to present the user interface of the external device 500 on the display device 200.

[0071] For example, as shown in FIG7 , when the display device 200 is connected to the external device 500 via an HDMI interface, and the external device 500 is a device such as a set-top box, the display device 200 can set the current signal source to the HDMI interface by setting the signal source. After the external device 500 is turned on and running, it renders the set-top box control homepage according to the set-top box operating system settings and sends the screen corresponding to the set-top box control homepage to the display device 200. At this time, since the current signal source of the display device 200 is the HDMI interface, the display device 200 can obtain the screen frame corresponding to the set-top box control homepage from the HDMI interface and display it on the display 260.

[0072] While displaying the user interface of the external device 500, the display device 200 can also receive control commands input by the user in response to user interactions and transmit them to the external device 500 via the device interface 240. The external device 500 then executes the control action corresponding to the control command transmitted by the device interface 240. For some control commands generated by interactive operations, the external device 500 will generate corresponding interface changes when executing the control command. Therefore, the changed interface content can be transmitted to the display device 200 along with the data stream in the device interface 240, so that the user interface displayed on the display device 200 will change synchronously.

[0073] For example, after the external device 500 such as a set-top box is turned on, the set-top box control homepage will be displayed by default. The set-top box control homepage includes multiple media resource item controls and a focus cursor. The user can input a cursor movement instruction through the direction keys on the remote control of the display device 200. After the display device 200 transmits the cursor movement instruction to the set-top box via the HDMI cable, the set-top box can change the focus cursor display position in the set-top box control homepage in response to the cursor movement instruction. At this time, the set-top box can send the focus cursor display position change process and the interface content after the focus cursor display position changes to the display device 200, so that the display device 200 can display it in real time.

[0074] In some embodiments, some control commands can cause the user interface provided by the external device 500 to switch. That is, the external device 500 can switch from one interface to another in response to a control command that causes the user interface to switch. During the interface switching process, the display device 200 can also display the interface data transmitted by the device interface 240 in real time.

[0075] For example, as shown in FIG8 , after the user controls the focus cursor to the media item control position corresponding to media asset A based on the set-top box control homepage, and presses the confirmation key on the remote control of the display device 200, a control instruction for displaying the media asset details interface can be input to the display device 200. The display device 200 then sends the control instruction to the set-top box via the HDMI interface. The set-top box can respond to the control instruction and jump from the set-top box control homepage to the media asset details interface. The transition animation of the jump process and the screen data corresponding to the media asset details interface after the jump are sent to the display device 200 via the HDMI interface, so that the display device 200 changes from presenting the set-top box control homepage interface to displaying the transition animation and the media asset details interface.

[0076] In some embodiments, since the user interfaces provided by the external device 500 have a hierarchical relationship, when jumping between interfaces, the external device 500 will respond to the control instruction and jump according to the hierarchical relationship between the interfaces. For example, the set-top box can provide a set-top box control homepage interface, a media asset details interface, and a playback interface respectively. These three interfaces have a hierarchical relationship, that is, when a confirmation instruction is input based on the media asset item control of the homepage interface, the interface jumps to the media asset details interface, and when a confirmation instruction is input based on the playback control of the media asset details interface, the interface jumps to the playback interface. Conversely, when the user inputs a return instruction based on the playback interface, the interface jumps to the media asset details interface; when a return instruction is input based on the media asset details interface, the interface jumps to the control homepage interface.

[0077] It can be seen that since some user interfaces provided by the external device 500 have a hierarchical relationship, and when the display device 200 controls the external device 500, the control instructions will be passed to the external device 500 without restriction. Therefore, in some scenarios, the passed control instructions cannot be responded to by the external device 500, that is, the interface content cannot be changed. For example, when the display device 200 sends a return instruction corresponding to the return key to the external device 500 through the HDMI CEC channel, if the external device 500 has returned to the set-top box control homepage interface at the end of the hierarchical relationship, and the display device 200 also passes the return instruction corresponding to the return key to the external device 500, then the return key at this time will not take effect, that is, the set-top box will not change the interface content according to the return key, causing the display device 200 to stay on the last control homepage interface for a long time, reducing the user experience.

[0078] To alleviate the problem of return commands not taking effect in some scenarios, embodiments of the present disclosure provide a display device that can detect changes in the interface content provided by an external device 500 and respond to return commands based on different detection results, thereby achieving a joint control function with the external device 500. To achieve the joint control function, the display device should include at least a display 260, a device interface 240, and a processor 250.

[0079] In some embodiments, the display 260 may be configured to display a user interface, which may include: a first user interface provided by an external device, and / or a second user interface provided by a display device.

[0080] The device interface 240 can be configured to connect to the external device 500. The device interface 240 can transmit interface data sent by the external device 500 to the display device. The device interface 240 can also be used to transmit control commands for the joint control between the display device and the external device 500. The device interface 240 can be one or a combination of an HDMI interface, a component interface, a CVBS interface, and a USB interface.

[0081] In some embodiments, if only one interface is used to simultaneously satisfy the requirements for transmitting interface data and control instructions, the display device can establish a connection relationship with the external device 500 through one interface. For example, the display device is connected to the external device 500 through an HDMI interface that supports the CEC function. Correspondingly, if only one interface is used that cannot simultaneously satisfy the requirements for transmitting interface data and control instructions, the display device can establish a connection relationship with the external device 500 through two or more interfaces. For example, when the display device is connected to the external device 500 through an HDMI interface that does not support the CEC function, since the current HDMI interface cannot transmit control instructions, the display device can also establish a connection with the external device 500 through a USB interface, so as to use the USB interface to transmit control instructions and use the HDMI interface to transmit interface data.

[0082] FIG9 is a flow chart of a method for joint control of an external device according to some embodiments. The display device can solve the problem that the return operation of the external device does not take effect when returning to the last home page interface by executing the method flow shown in FIG9 .

[0083] Specifically, as shown in FIG9 , the following steps may be included:

[0084] S110: When the user interfaces displayed on the display include the first user interface, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a current interface image.

[0085] After the display device is connected to the external device 500, the interface content that can be displayed includes the following three forms, namely, displaying the first user interface in its entirety, displaying the second user interface in its entirety, and displaying a combination of the first and second user interfaces. For example, when the display device is connected to a set-top box device via an HDMI interface, and the user sets the signal source of the display device to the HDMI interface, the display device can display the set-top box control homepage interface, media resource details interface, and set-top box media resource playback interface provided by the set-top box, i.e., the display device can display the first user interface in its entirety. When the user sets the signal source of the display device to something other than the HDMI interface, the display device can display its own user interface, including the display device control homepage, media resource recommendation interface, display device playback interface, settings interface, etc., i.e., the display device displays the second user interface in its entirety.

[0086] For part of the second user interface displayed by the display device, the page may also include relevant content of the first user interface. For example, the signal source switching interface of the display device may include signal source option controls for multiple device interfaces 240. A preview window may be provided on each signal source option control or in an associated position. The preview window may display in real time the interface content provided by the external device 500 connected to each device interface 240. As shown in Figure 10, when the HDMI1 interface of the display device is connected to a set-top box device, the set-top box control homepage provided by the set-top box, i.e., the first user interface, is displayed in the signal source control preview window corresponding to the HDMI1 interface.

[0087] When the user interface displayed by the display device is the first user interface or includes the first user interface, the display device can receive control commands input by the user. Since the connection between the display device and the external device 500 has been established when the display device displays the first user interface or the second user interface including the first user interface, the display device can forward the control command (or forward part of the control command) to the external device 500 in accordance with the driver and protocol specifications corresponding to the device interface 240, thereby realizing the joint control function with the external device 500.

[0088] The control instructions received by the display device can have different control functions based on different input methods. In some embodiments, the control instructions can be used to control the display device or the external device 500 to perform a return operation. For example, the user can input a return instruction by pressing the return key of the remote control provided with the display device. When the user presses the return key, the remote control can send a wireless transmission signal (such as infrared, Bluetooth, radio frequency, etc.) carrying the return key value to the display device. After receiving the wireless transmission signal carrying the return key value, the display device can read the wireless transmission signal and read the return key value to respond.

[0089] In some embodiments, the return command can also be input through different interactive actions based on the interaction methods supported by the display device. For example, for a display device that supports touch interaction, the user can input the return command based on a touch gesture of the display device, such as swiping up from the bottom of the display device. For a display device that supports voice interaction, the user can input the return command by voice input, such as "return to the previous level."

[0090] After the user enters a return command, the display device can respond based on the return command. When the display device and external device 500 are jointly controlled, the specific control command entered by the user is forwarded to the external device 500. For example, after receiving the return command, the display device can convert the return command into a command that complies with the interaction specifications of the external device 500 based on the HDMI CEC protocol, and transmit the return command to the external device 500 based on the HDMI CEC channel. The external device 500 then responds with a return operation based on the return command transmitted by the display device. However, when the external device 500 returns to the last level of user interface, the return command will not change the user interface of the external device 500 after being transmitted to the external device 500, that is, the return operation will not take effect. Therefore, to ensure that the return operation can still take effect, in some embodiments, the display device can first determine the content of the user interface provided by the external device 500 before forwarding the return command to the external device 500 to determine whether the user interface provided by the external device 500 is the last level of user interface.

[0091] That is, the display device can perform a screenshot on the current user interface in response to the return instruction to generate the current interface image. In some embodiments, the display device can perform a screenshot operation on the currently displayed user interface by calling a built-in screenshot tool (application).

[0092] For example, the display device can call the system's screenshot process through the Android developer API. That is, the display device first calls the screenshot process through the following instructions: import android.graphics.Bitmap; import android.graphics.Canvas; import android.view.View; import android.widget.ImageView; then obtain the View to be screenshoted through the following instructions: View viewToCapture = findViewById(R.id.view_to_capture); then create a Bitmap object according to the following instructions to store the screenshot image: Bitmap bitmap = Bitmap.createBitmap(viewToCapture.getWidth(), viewToCapture.getHeight(), Bitmap.Config.ARGB_8888); then create a Canvas object according to the following instructions and use the Bitmap object as its bitmap: Canvas canvas = new Canv as(bitmap); then draw the View to be screenshoted onto the Canvas through the viewToCapture.draw instruction: viewToCapture.draw apture.draw(canvas); Finally, follow the following instructions to save the screenshot to the album of the display device: MediaStore.Images.Media.insertImage(getContentResolver(),bitmap,"Screenshot",null); catch (FileNotFoundException e) e.printStackTrace().

[0093] Based on the screenshot method provided in the above embodiment, the display device can use the screenshot tool to take a screenshot of the currently displayed user interface. Since the user interface currently displayed by the display device includes the first user interface provided by the external device 500, after taking a screenshot of the current user interface, an image of the interface provided by the external device 500 can be obtained. In some embodiments, the user interface displayed by the display device may also be a user interface composed of a first user interface and a second user interface. That is, in the currently displayed user interface, a portion of the area displays the first user interface provided by the external device 500, and another portion of the area displays the second user interface provided by the display device. In this case, if a screenshot is taken of the entire displayed user interface, the content of the second user interface will affect the judgment of the content provided by the external device 500.

[0094] Therefore, as shown in FIG11 , the display device can first traverse the control types in the current display user interface before executing the screenshot operation, and determine whether the current display user interface includes a play window control (S1101). If the current display user interface includes a play window control, a screenshot is taken of the area where the play window control is located to generate a current interface image (S1102). For example, when the display device displays a signal source switching interface, the signal source switching interface may include a preview window for displaying the content of the HDMI1 interface, that is, the signal source switching interface includes a play window control. At this time, the display device can read the configuration file of the signal source switching interface and determine the display area of ​​the play window control therefrom, and then perform a screenshot based on the display area, thereby obtaining a current interface image of the content displayed by the play window control, that is, obtaining a current interface image containing only the content of the first user interface.

[0095] If the currently displayed user interface does not include a play window control, a screenshot is taken of the entire user interface area to generate a current interface image (S1103). If the user interface displayed by the display device includes the first user interface content, and the currently displayed user interface does not include a play window control, it means that the display device is currently displaying the first user interface provided by the external device 500 in full screen. Therefore, the display device can obtain a current interface image that only includes the first user interface content by taking a screenshot of the entire user interface area.

[0096] In some embodiments, the display device may not invoke a screenshot tool when acquiring the current interface image, but may instead extract a specific image frame from the media data stream to acquire the current interface image. Specifically, as shown in FIG12 , when generating the current interface image, the display device may acquire the media data stream transmitted by the device interface 240 and then, based on the current input time when the user inputs the return command, capture the target image frame from the media data stream to generate the current interface image based on the target image frame.

[0097] The target image frame is an image frame displayed at the current input time or at a time adjacent to the current input time. For example, if the user presses the return key at 18:00:05 on May 5, 2023, the display device can obtain the current input time T1 = 2023-5-5-18:00:05 when the return command was entered. Then, from the media resource data stream sent by the external device 500, the target image frame with a display timestamp of 2023-5-5-18:00:05:000 or the shortest time difference with the current input time 2023-5-5-18:00:05, i.e., Pn(2023-5-5-18:00:05:035), is extracted as the current interface image.

[0098] In some embodiments, the display device can also obtain the current interface image by extracting the display screen frame from the frame buffer of the current interface. For example, the user interface of the display device is an overlay screen of the video layer (video plane) and the graphics (On Screen Display, OSD) layer, that is, the image content of the user interface can be rendered and then formed into a screen frame for display in the synthesis module. Among them, the synthesis module can store the screen frames through the frame buffer (frame buffer) and the video buffer (video buffer) respectively, and form the screen content in the corresponding layers respectively through the video process module. The media screen sent by the external device 500 through the device interface 240 forms the screen content through the video layer (video plane), while the image frames of other image screens (such as controls, progress bars, and other graphics of the UI interface) form the screen content through the OSD layer (OSD plane). The display device forms the final screen for display through the blending unit (blend) of the video process module. Therefore, the display device can extract the screen frame of the final screen in the blending unit, or extract the screen frame from the video buffer (video buffer) corresponding to the video layer to generate the current interface image.

[0099] Because the picture frame of the final picture extracted in the mixing unit may include the picture content provided by the display device. For example, in the HDMI setting interface, it can include an HDMI signal display window and button controls around the HDMI signal display window. The HDMI signal display window displays the picture content provided by the HDMI device, and the button controls are controls provided by the display device for setting the HDMI connection. Based on the operating logic of the display device, the HDMI setting interface is a user interface provided by the display device, so the display device is required to respond to user button operations, and such an operating mode is more in line with user habits.

[0100] Therefore, to avoid affecting the user interface operation logic of the display device, the display device may only perform the joint control function when the first user interface provided by the external device 500 is displayed in full screen and the joint control function is enabled. That is, in some embodiments, the display device can read the display mode and joint control switch status from the current setting information. The display mode represents the display ratio of the image provided by the external device 500, including full-screen mode and window mode. The joint control switch status represents whether the joint control function is enabled, i.e., the joint control switch status includes an on state and an off state.

[0101] If the display mode is full-screen mode and the joint control switch state is on, the display screen frame is extracted from the frame buffer of the video layer, and the current interface image is generated based on the display screen frame. For example, the display device can read the current setting information, and when "display mode = full screen" is read in the current device information, it is determined that the current display mode is full-screen mode. At the same time, the display device can read the switch state of the HDMI CEC function in the setting information, and when "HDMI CEC = on" is read, it is determined that the joint control switch state is on. At this time, the display device can determine that the currently displayed content is the screen content provided by the external device 500 and supports the joint control function, so it can extract the current display screen frame from the video buffer of the video layer of the current user interface, and convert the extracted display screen frame into a picture format to generate the current interface image.

[0102] As can be seen, in the above embodiment, the display device can obtain the current interface image by calling the screenshot API interface, extracting image frames from the media data stream, and extracting display screen frames from the frame buffer. At the same time, by detecting the display mode and the state of the joint control switch, the current interface image that can reflect the screen content provided by the external device 500 can be accurately obtained, facilitating more accurate judgment of the screen content and achieving more accurate joint control.

[0103] S120: Acquire historical interface images;

[0104] After obtaining the current interface image through screenshot, the display device can also obtain the historical interface image. The historical interface image is the image generated by taking a screenshot of the user interface before the user inputs the return instruction this time.

[0105] In some embodiments, the display device can set up a resident screenshot process after connecting to the external device 500. The screenshot process can monitor the control instructions input by the user, or monitor the screen content of the media data sent by the external device 500. When it monitors the user inputting specific control instructions or the screen content undergoes specific changes, it can perform a screenshot operation and store the image obtained by the screenshot to generate a historical interface image.

[0106] In some embodiments, the historical interface image can be an image obtained by the display device after taking a screenshot of the user interface at the time the user last entered the return command. That is, as shown in Figure 13, when executing to obtain the historical interface image, the display device can call a screenshot database, wherein the screenshot database is used to store images generated by taking screenshots of the user interface when the return command is entered. Similar to step S100, the display device can respond to the return command each time the user enters the return command and take a screenshot of the current user interface to obtain the interface image displayed at that time, and store the obtained interface image.

[0107] To facilitate subsequent access, the interface images obtained by screenshot can be stored in a screenshot database according to specific storage rules. For example, the screenshot database can include the storage address of the screenshot file and the execution time of the screenshot operation. According to the execution time of the screenshot operation, the storage address of the associated screenshot file can be queried.

[0108] After calling the screenshot database, the display device can obtain the current input time when the return command is entered, and query the adjacent input time when the return command was entered based on the current input time. For example, if the user presses the return key at 18:00:05 on May 5, 2023, the display device can obtain the current input time when the return command is entered, T1 = 2023-5-5-18:00:05. Then, based on the current input time T1, the display device queries the screenshot database for the latest screenshot operation execution time before T1, that is, the adjacent input time when the return command was entered, T0 = 2023-5-5-18:00:02.

[0109] Then, based on the adjacent input time, the display device retrieves the historical interface image from the screenshot database. Specifically, it retrieves the image generated by taking a screenshot of the user interface during the previous return command input, and uses that image as the historical interface image. For example, if the adjacent input time T0 = 2023-5-5-18:00:02, the display device can retrieve the storage address corresponding to the screenshot image saved at T0 in the screenshot database, namely, " / storage / Pictures / Screenshots / ××20230505180002.JPG," based on the adjacent input time T0, to obtain the historical interface image.

[0110] In some embodiments, the historical interface image can be an image obtained by taking a screenshot of the first user interface after the display device monitors the first user interface switching. For example, the display device can perform screen content recognition on the key frames of the first user interface provided by the external device 500 based on an image recognition algorithm. When it is recognized that the first user interface switches from the playback details interface to the set-top box control homepage interface, the display device can take a screenshot of the control homepage interface after the interface switching and save the interface screenshot as the historical interface image.

[0111] S130: Calculate the similarity between the current interface image and the historical interface images.

[0112] After obtaining the current interface image and the historical interface images, the display device can calculate the similarity between the current interface image and the historical interface images based on a preset similarity algorithm. The similarity is used to indicate the content difference between the current interface image and the historical interface images. That is, the higher the similarity, the closer the content of the current interface image and the historical interface images are; the lower the similarity, the greater the content difference between the current interface image and the historical interface images.

[0113] The display device can calculate the similarity between the current interface image and the historical interface image based on pixel-level comparison, histogram comparison, feature comparison and deep learning. Among them, pixel-level comparison is to calculate the similarity by comparing the pixel values ​​of the two images. Histogram comparison first calculates the histogram of each image and then compares the two histograms to calculate the similarity. Feature comparison is to extract features from each image and then compare these features. Feature extraction methods such as Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB) are used to determine the similarity. Deep learning uses methods such as Convolutional Neural Networks (CNN) to extract features of the image and then compare these features to determine the similarity.

[0114] S140: If the similarity is less than the similarity threshold, a return instruction is sent to the external device through the device interface, and the display is controlled to display the first user interface after the external device executes the return instruction.

[0115] After calculating the similarity between the current interface image and the historical interface image, the display device can determine the content difference between the current interface image and the historical interface image based on the calculated similarity. If the similarity is less than the similarity threshold, that is, the content difference between the current interface image and the historical interface image is significant, it means that the return instruction transmitted by the display device to the external device 500 can change the interface content of the first user interface. Therefore, the display device can continue to send the return instruction to the external device 500 via the device interface 240, so that the external device 500 can respond to the return operation corresponding to the return instruction.

[0116] After sending the return instruction to the external device 500, the display device can continue to receive interface data sent by the external device 500, thereby displaying the first user interface after the external device 500 executes the return instruction. For example, if the display device presets a similarity threshold of 90%, then after calculating that the similarity between the current interface image and the historical interface image is 47.3%, the display device can determine that the similarity between the current interface image and the historical interface image is less than the similarity threshold. At this time, the display device can transmit the return instruction to the external device 500 via the HDMI interface in accordance with the HDMI CEC protocol. After receiving the return instruction, the external device 500 can perform a return operation, that is, switch from the playback interface to the media resource details interface, and send the interface switching process and the interface data corresponding to the media resource details interface after the switch to the display device via the HDMI interface, so that the display device displays the interface switching process and the media resource details interface after the switch.

[0117] S150: If the similarity is greater than or equal to the similarity threshold, execute a return instruction, and control the display to display a second user interface.

[0118] By comparing the similarity with the similarity threshold, if the similarity is greater than or equal to the similarity threshold, that is, the content difference between the current interface image and the historical interface image is small, the display device may not send the return instruction to the external device 500, but the display device itself may execute the return instruction and display the second user interface after executing the return instruction.

[0119] Since the display device displays the first user interface before executing the return instruction, the display device can exit the display of the first user interface and display the user interface provided by itself by executing the return instruction. For example, when the display device is displaying the set-top box control homepage interface, it receives a return instruction input by the user. In response to the return instruction, the display device can obtain the current interface image and the historical interface image, and the similarity between the current interface image and the historical interface image is 100% by calculation, that is, the interface content in the current interface image and the historical interface image is the set-top box control homepage interface. Because the similarity between the current interface image and the historical interface image is 100% greater than the similarity threshold of 90%, the display device does not send the return instruction to the external device 500, but executes the return instruction by the display device itself, that is, the display device can exit the currently displayed set-top box control homepage interface and switch to the default homepage interface of the operating system of the display device.

[0120] In some embodiments, when executing a return instruction, the display device may control the display to display an exit prompt message in response to the return instruction (S1501). The exit prompt message is used to prompt the user whether to stop displaying the first user interface provided by the external device 500. For example, as shown in Figure 14, when it is determined that the similarity between the current interface image and the historical interface image is greater than the similarity threshold, the display device may display a message pop-up window in the current interface. The message pop-up window may include prompt information such as "Please press the confirmation key to exit the current HDMI device and return to the home page." The user can enter an operation instruction based on this information.

[0121] During the process of displaying the exit prompt information, the display device monitors the operation instructions input by the user based on the exit prompt information (S1502). For example, after the display device displays a message pop-up window containing the exit prompt information, the operation instructions input by the user based on the remote control can be monitored.

[0122] The operation instruction may include a first instruction indicating confirmation of exit and a second instruction indicating cancellation of exit. For example, if a user presses the "confirm key" on a remote control while a message pop-up window is displayed on a display device, the first instruction indicating confirmation of exit is detected. Conversely, if a user presses the "return key" on a remote control while a message pop-up window is displayed on a display device, the second instruction indicating cancellation of exit is detected.

[0123] The display device can execute different interaction methods for different operation instructions input by the user. That is, if the operation instruction is a first instruction representing confirmation of exit, the display is controlled to display the second user interface (S1503). For example, as shown in Figure 15, when the user presses the confirmation key based on the pop-up message window of the exit prompt information, it indicates that the user wants to control the display device to exit the HDMI signal interface. Therefore, the display device can display the default control homepage interface of the display device operating system, that is, display the second user interface.

[0124] Correspondingly, if the operation instruction is a second instruction representing cancellation of exit, the display is controlled to display the first user interface after the external device executes the return instruction (S1504). For example, when the user presses the return key based on the message pop-up window of the exit prompt information, it indicates that the user does not want to control the display device to exit the HDMI signal interface. Therefore, the return instruction can be sent to the external device 500 according to the return instruction transmission method in the above embodiment. The return instruction is still executed by the external device 500, and the set-top box control homepage interface after the external device 500 executes the return instruction, i.e., the first user interface, is displayed.

[0125] It should be noted that when the operation instruction input by the user is a cancel exit instruction, since the similarity between the current interface image and the historical interface image can determine that the current return instruction will not cause the interface of the external device 500 to change, the display device may not pass the return instruction to the external device 500 and directly cancel the operation function corresponding to the return instruction.

[0126] In some embodiments, when displaying the second user interface, the display device may also close the data channel corresponding to the device interface, query the target user interface, and control the display to display the target user interface. The target user interface may be the second user interface displayed by the display device by default, or the target user interface may be the second user interface displayed by the display device when the device interface is switched as the signal source.

[0127] In some embodiments, the display device may preferentially detect whether it can query the second user interface displayed by the display device when switching to the device interface as the signal source. For example, after the display device determines that the similarity between the current interface image and the historical interface image is greater than or equal to the similarity threshold, it can respond to the return instruction to close the HDMI channel, that is, close the HDMI channel interface. Then, query the user interface displayed before the user switches to the HDMI channel signal source, that is, the signal source switching interface. If the user interface displayed by the display device before switching to the HDMI channel signal source is not queried, the default displayed user interface, that is, the control homepage interface, can be obtained, and the control homepage interface can be displayed.

[0128] As can be seen from the above embodiments, after receiving the return instruction input by the user, the display device can obtain the current interface image by taking a screenshot of the current user interface, obtain the historical interface image, and calculate the similarity between the current interface image and the historical interface image, thereby judging whether the interface content of the current interface image and the historical interface image has changed based on the similarity between the current interface image and the historical interface image, that is, determining whether the external device 500 can effectively respond to the return instruction. When the similarity is greater than or equal to the similarity threshold, the display device no longer passes the return instruction to the external device 500, but executes the return instruction by itself. In this way, the aforementioned joint control method of giving the return key to the external device 500 without restriction can be changed. When the external device 500 returns to the last control homepage interface, the display device can exit the current HDMI channel according to the intention of the return key and return to the control homepage of the display device, thereby improving the continuity of the joint control interaction and improving the user experience.

[0129] In some embodiments, as shown in FIG16 , the user can manually set whether to enable the linked control function of the external device 500 through a setting item. When the linked control function of the external device 500 is disabled, the display device does not transmit control instructions to the external device 500, and all control instructions input by the user are responded to and executed by the display device. Therefore, the issue of the return operation not being effective in the above embodiment does not arise. However, when the linked control function of the external device 500 is enabled, the display device can transmit control instructions to the external device 500, and some of all control instructions input by the user are responded to and executed by the external device 500. Therefore, the issue of the return operation not being effective in the above embodiment does not arise.

[0130] Based on this, as shown in FIG17 , when a user sets whether to enable the linked control function of an external device 500, the display device may also obtain setting information for the linked control function item ( S1701 ) before taking a screenshot of the current user interface to generate the current interface image. The setting information for the linked control function item is used to indicate whether the user has enabled the linked control function. Different setting states provide different setting information contents.

[0131] If the setting information is the first information used to represent the activation of the joint control function, the display device takes a screenshot of the current user interface to generate a current interface image (S1702), and calculates the similarity between the current interface image and the historical interface image according to the joint control method provided in the above embodiment, and then determines whether to pass the return instruction to the external device 500 based on the comparison result of the similarity and the similarity threshold.

[0132] For example, when the display device receives a return instruction input by the user, it can read the user's settings from the configuration file, where the setting item "Joint control options" is used to indicate whether the joint control function of the HDMI CEC protocol is enabled. When "Joint control options = ON" is detected, that is, the setting information is detected as the first information "ON", a screenshot of the current user interface can be performed in the manner described in the above embodiment to generate an image of the current interface.

[0133] If the setting information is the second information for representing the closing of the joint control function, it means that the user has not turned on the joint control function of the external device 500, and the display device will not transmit the control instruction to the external device 500. Even if the similarity between the current interface image and the historical interface image is less than the similarity threshold, the display device cannot send a return instruction through the device interface item external device. Therefore, at this time, the display device can execute the return instruction and control the display to display the second user interface after executing the return instruction (S1703). For example, if "Joint control options = OFF" is detected, that is, the setting information is the second information "OFF", then the joint control method in the above embodiment can be skipped, and the return instruction can be executed directly, jumping from the set-top box playback interface back to the control homepage interface of the display device, and displaying the second user interface after executing the return instruction, that is, the control homepage interface of the display device, through the display 260.

[0134] During interface switching on the external device 500, the user typically needs to perform multiple return operations to return to the last control homepage interface. Furthermore, multiple return operations are required to determine whether the content of the first user interface has changed. For example, if a user interactively enters the "control homepage interface," the media resource details interface, and the playback interface, the user needs to press the return key three times in succession to determine whether the user interface is the last control homepage interface. The intervals between these three consecutive return key presses are relatively short. Therefore, the display device can determine whether the user has exited the device interface based on the time interval between return command inputs.

[0135] That is, as shown in Figure 18, in some embodiments, the display device can obtain the first screenshot time of the current interface image (S1801), and obtain the second screenshot time of the historical interface image (S1802), and then calculate the screenshot time interval between the first screenshot time and the second screenshot time (S1803).

[0136] If the screenshot time interval is less than the interval threshold, it means that the user has continuously input the return instruction in a short period of time. Therefore, the aforementioned similarity calculation can be executed, and the corresponding user interface process can be displayed based on the comparison result of the similarity and the similarity threshold. That is, the similarity between the current interface image and the historical interface image is calculated (S1804). If the similarity is less than the similarity threshold, the return instruction is sent to the external device through the device interface, and the display is controlled to display the first user interface after the external device executes the return instruction (S1805). If the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and the display is controlled to display the second user interface (S1806).

[0137] If the screenshot interval is greater than or equal to the interval threshold, it indicates that the user has not continuously input the return command within a short period of time, that is, the user has no intention of exiting the device interface. Therefore, the display device 200 can execute the step of sending the return command to the external device through the device interface and continue to control the external device 500. In other words, the process returns to step S1805.

[0138] For example, when a user wants to exit an HDMI device, they press the back key repeatedly, with the interval between presses being less than 1 minute. Therefore, the display device can set the interval threshold to 1 minute. When the user presses the back key at time T2, the display device can obtain the time T1 of the last press of the back key, calculate the time interval between the two presses of the back key (T = T2 - T1), and compare it with the preset interval threshold.

[0139] If the time interval T = 3 minutes, which is greater than the interval threshold of 1 minute, it indicates that the user is only performing a normal return operation and has no intention of exiting the HDMI device. Therefore, a return instruction can be sent to the external device 500 via the HDMI CEC channel according to the joint control function of the external device 500. If the time interval T = 2 seconds, which is less than the interval threshold of 1 minute, it indicates that the user has the intention of exiting the HDMI device. Therefore, according to the joint control method of the external device 500 provided in the above embodiment, a screenshot of the current interface can be taken to generate the current interface image. Based on the current interface image and the historical interface images, it is determined whether to send a return instruction to the external device 500 via the HDMI CEC channel.

[0140] Based on the above-mentioned display device, an external device joint control method provided by an embodiment of the present disclosure may include the following steps: when a first user interface is included in the user interface displayed on the display, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a current interface image, where the first user interface is a user interface provided by the external device; obtaining a historical interface image, where the historical interface image is an image generated by taking a screenshot of the user interface before the return instruction; calculating the similarity between the current interface image and the historical interface image; if the similarity is less than a similarity threshold, sending a return instruction to the external device through the device interface, and controlling the display to display the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, executing the return instruction, and controlling the display to display the second user interface, where the second user interface is a user interface provided by the display device.

[0141] The above process can be executed in response to the return instruction input by the user when the user interface displayed by the display device includes the first user interface provided by the external device, and the current user interface is screenshotted to generate the current interface image. And the historical interface image generated by taking the screenshot of the user interface before the return instruction is obtained, and then the similarity between the current interface image and the historical interface image is calculated. If the similarity is less than the similarity threshold, the return instruction is sent to the external device through the device interface to display the first user interface after the external device executes the return instruction; if the similarity is greater than or equal to the similarity threshold, the display device executes the return instruction and displays the second user interface. The method can be implemented on the basis of the joint control being effective, when the external device returns to the last interface, the return instruction can exit the logic of the external device interface, improve the continuity of the interaction, and solve the problem that the return operation does not take effect when the external device returns to the last home page interface.

[0142] In the above embodiment, when the displayed user interface includes a first user interface provided by the external device 500, the display device can respond to a return instruction, obtain a current interface image and a historical interface image, and determine whether to pass the return instruction to the external device 500 for instruction response based on the similarity between the current interface image and the historical interface image. In some embodiments, the display device can also pass the return instruction to the external device 500 upon receiving the return instruction, and determine whether to exit the device interface signal interface based on the user interface content provided by the external device 500 in response to the return instruction. That is, the embodiment of the present disclosure also provides a display device, as shown in Figure 19, which may include: a display 260, a device interface 240, a memory (not shown), and at least one processor 250 connected to the display 260, the device interface 240, and the memory. The display 260 is configured to display a user interface, the user interface including the first user interface provided by the external device and / or the second user interface provided by the display device; the device interface 240 is configured to connect to the external device; the memory is configured to store computer instructions and data associated with the display device, and the at least one processor 250 is configured to execute the computer instructions so that the display device performs the following steps:

[0143] S210: When the user interfaces displayed on the display include the first user interface, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a first interface image;

[0144] S220: Sending a return instruction to the external device through the device interface, and obtaining a second interface image after the external device executes the return instruction;

[0145] S230: Calculating the similarity between the first interface image and the second interface image;

[0146] S240: If the similarity is less than the similarity threshold, controlling the display to display the first user interface after the external device executes the return instruction;

[0147] S250: If the similarity is greater than or equal to the similarity threshold, execute a return instruction, and control the display to display a second user interface.

[0148] For example, when the display device is displaying the playback interface provided by the external device 500, the user enters a return command using the return key of the remote control provided with the display device. After receiving the return command entered by the user, the display device can take a screenshot of the current user interface to generate the first interface image. At the same time, the return command is transmitted to the external device 500 via the HDMI CEC channel.

[0149] After receiving the return command, the external device 500 executes the return command to switch the interface from the playback interface to the media resource details interface, and sends the interface data of the switched media resource details interface to the display device via the HDMI interface. The display device then takes a screenshot based on the received interface data to obtain the second interface image provided by the external device 500 after executing the return command.

[0150] Next, the similarity between the first interface image and the second interface image is calculated, and based on the similarity between the first interface image and the second interface image, it is determined whether the external device 500 has effectively responded to the return instruction. That is, when the external device 500 switches from the playback interface to the media resource details interface, the similarity between the first interface image and the second interface image is less than the similarity threshold. Therefore, the display device can display the media resource details interface after the external device 500 executes the return instruction, that is, display the first user interface.

[0151] For another example, when the display device displays the set-top box control homepage interface provided by the external device 500, the user enters a return command using the return button on the remote control provided with the display device. The display device can then obtain the first interface image according to the above example and transmit the return command to the external device 500 via the HDMI CEC channel. Upon receiving the return command, the external device 500 switches interfaces. Since the control homepage interface is already the last interface, the interface content will not be changed when the external device 500 executes the return command. That is, the content of the second interface image obtained by the display device after the external device 500 executes the return command is still the set-top box control homepage interface.

[0152] At this time, the display device calculates the similarity between the first interface image and the second interface image according to the above example, and the similarity is greater than the similarity threshold. Therefore, the display device can execute the return instruction, exit the HDMI interface, and display the display device's default control homepage interface, that is, the second user interface.

[0153] Based on the display device provided in the above embodiment, an embodiment of the present disclosure further provides a method for joint control of an external device, which may include:

[0154] When the displayed user interface includes a first user interface, in response to a return instruction input by the user, a screenshot is taken of the current user interface to generate a first interface image, where the first user interface is a user interface provided by an external device, and the display device is connected to the external device through a device interface; a return instruction is sent to the external device through the device interface, and a second interface image is obtained after the external device executes the return instruction; the similarity between the first interface image and the second interface image is calculated; if the similarity is less than a similarity threshold, the first user interface after the external device executes the return instruction is displayed; if the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and the second user interface is displayed, where the second user interface is a user interface provided by the display device.

[0155] The above process can be performed when the user interface displayed by the display device includes a first user interface, and in response to the return instruction input by the user, a screenshot of the current user interface is executed to generate a first interface image. Then, a return instruction is sent to the external device through the device interface, and a second interface image after the external device executes the return instruction is obtained. Then, the similarity between the first interface image and the second interface image is calculated, and when the similarity is less than the similarity threshold, the first user interface after the external device executes the return instruction is displayed; when the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and the second user interface is displayed, and the second user interface is the user interface provided by the display device. The method can determine whether to return to the next interface by detecting the similarity of the user interfaces provided by the external device 500 before and after executing the return instruction, and when it is determined to return to the last interface, the logic of the external device interface can be exited in response to the return instruction, thereby improving the continuity of the interaction and solving the problem that the return operation does not take effect when the external device returns to the last homepage interface.

[0156] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be repeated here.

Claims

1. A display device, comprising: A display configured to display a user interface, wherein the user interface includes a first user interface provided by an external device and / or a second user interface provided by the display device; A device interface configured to connect an external device; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, device interface and memory, is configured to execute computer instructions to cause the display device to perform: When the user interfaces displayed on the display include the first user interface, in response to a return instruction input by a user, taking a screenshot of the current user interface to generate a current interface image; Acquire a historical interface image, where the historical interface image is an image generated by taking a screenshot of the user interface before the user inputs the return instruction; Calculating the similarity between the current interface image and the historical interface image; If the similarity is less than the similarity threshold, sending the return instruction to the external device through the device interface, and controlling the display to display a first user interface after the external device executes the return instruction; If the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and the display is controlled to display a second user interface.

2. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device performs the acquisition of the historical interface image in the following manner: Calling a screenshot database, wherein the screenshot database is used to store images generated by taking screenshots of the user interface when a return instruction is input; Get the current input time when the input return instruction is entered; According to the current input time, query the adjacent input time when the return instruction was input last time; An image generated by taking a screenshot of the user interface when a return instruction was input last time is extracted from the screenshot database according to the adjacent input time, and the image is used as the historical interface image.

3. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device generates the current interface image in the following manner: Acquire the media data stream transmitted by the device interface; According to the current input time when the user inputs a return instruction, a target image frame is captured in the media asset data stream, wherein the target image frame is an image frame displayed at the current input time or at a time adjacent to the current input time; The current interface image is generated according to the target image frame.

4. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device generates the current interface image in the following manner: Read the display mode and joint control switch status from the current setting information; If the display mode is a full-screen mode and the joint control switch state is an on state, extracting a display picture frame from a frame buffer of a video layer; The current interface image is generated according to the display picture frame.

5. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device executes the return instruction and controls the display to display the second user interface in the following manner: In response to the return instruction, controlling the display to display an exit prompt message; The exit prompt information indicates whether to stop displaying the first user interface provided by the external device; Monitoring the operation instructions input by the user based on the exit prompt information; If the operation instruction is a first instruction representing confirmation of exit, the display is controlled to display a second user interface.

6. The display device according to claim 5, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If the operation instruction is a second instruction representing canceling the exit, the display is controlled to display a first user interface after the external device executes the return instruction.

7. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Before taking a screenshot of the current user interface to generate the current interface image, the setting information of the joint control function item is determined to be the first information representing the activation of the joint control function.

8. The display device according to claim 7, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If the setting information is the second information indicating closing the joint control function, the return instruction is executed, and the display is controlled to display a second user interface after the return instruction is executed.

9. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Before executing the calculation of the similarity between the current interface image and the historical interface image, determining that a screenshot time interval between a first screenshot time and a second screenshot time is less than an interval threshold; in, The first screenshot time is the screenshot time of the current interface image, and the second screenshot time is the screenshot time of the historical interface image.

10. The display device according to claim 9, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If the screenshot time interval is greater than or equal to the interval threshold, the return instruction is sent to the external device through the device interface.

11. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device controls the display to display the second user interface in the following manner: Closing the data channel corresponding to the device interface; Querying a target user interface, where the target user interface is a second user interface displayed by the display device by default, or the target user interface is a second user interface displayed by the display device when the device interface is switched to be a signal source; The display is controlled to display the target user interface.

12. A display device comprising: A display configured to display a user interface, wherein the user interface includes a first user interface provided by an external device and / or a second user interface provided by the display device; A device interface configured to connect an external device; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, device interface and memory, is configured to execute computer instructions to cause the display device to perform: When the user interfaces displayed on the display include the first user interface, in response to a return instruction input by the user, taking a screenshot of the current user interface to generate a first interface image; Sending the return instruction to the external device through the device interface, and acquiring a second interface image after the external device executes the return instruction; Calculating the similarity between the first interface image and the second interface image; If the similarity is less than the similarity threshold, controlling the display to display a first user interface after the external device executes the return instruction; If the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and the display is controlled to display a second user interface.

13. A method for joint control of an external device, applied to the display device according to any one of claims 1 to 11, comprising: When the displayed user interfaces include a first user interface, in response to a return instruction input by a user, taking a screenshot of the current user interface to generate a current interface image, wherein the first user interface is a user interface provided by an external device, and the display device is connected to the external device via a device interface; Acquire a historical interface image, where the historical interface image is an image generated by taking a screenshot of the user interface before the return instruction; Calculating the similarity between the current interface image and the historical interface image; If the similarity is less than a similarity threshold, sending the return instruction to the external device through the device interface, and displaying a first user interface after the external device executes the return instruction; If the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and a second user interface is displayed, where the second user interface is a user interface provided by a display device.

14. A method for joint control of an external device, applied to the display device according to claim 12, comprising: When the displayed user interfaces include a first user interface, in response to a return instruction input by the user, a screenshot is executed on the current user interface to generate a first interface image; the first user interface is a user interface provided by an external device, and the display device is connected to the external device through a device interface; Sending the return instruction to the external device through the device interface, and acquiring a second interface image after the external device executes the return instruction; Calculating the similarity between the first interface image and the second interface image; If the similarity is less than the similarity threshold, displaying a first user interface after the external device executes the return instruction; If the similarity is greater than or equal to the similarity threshold, the return instruction is executed, and a second user interface is displayed.

Images