Display device and control method of two-way window

Abstract

The invention provides a display device and a control method for a two-way window, and the method comprises the steps: obtaining an anchor point coordinate and key information of a key instruction by the display device in response to the key instruction input by a user based on a floating window; determining an operation area of the key instruction according to the anchor point coordinates; and generating control information according to the operation area and the key information, and executing reverse control on the first terminal equipment or the second terminal equipment according to the control information. The display device can respond to a key instruction input by a user based on a floating window, determines a reverse control device by judging an operation area of the key instruction, and sends control information to different terminal devices through different applications. Therefore, reverse control of the first terminal equipment or the second terminal equipment in the two-way control process is realized, and the problem that control cannot be executed on media asset playing pictures and screen projection pictures of the terminal equipment is solved.

Description

Technical Field

[0001] This application relates to the technical field of display devices, and in particular, to a display device and a control method for a dual-channel window. Background Art

[0002] A display device is an intelligent device that can present a user interface and support user interaction. Taking a smart TV as an example, a smart TV is a display device based on Internet application technology, with an open operating system and a chip, and has an open application platform, enabling two-way human-computer interaction functions. A smart TV can integrate various functions such as audio-visual, entertainment, and data to meet the diverse and personalized needs of users. For example, users can watch various movies, TV dramas, variety shows, etc. on a smart TV, and can also connect terminal devices such as mobile phones and tablets to the smart TV to achieve functions such as screen mirroring and screen projection.

[0003] To meet the user's demand for multi-device synchronous control, due to the improvement of device chips, performance, etc., some display devices support the function of dual-channel synchronous control. Among them, one channel can access a High Definition Multimedia Interface (HDMI) device for reverse control of the same screen. For example, the HDMI device can be a computer device, and the other channel is used to connect mobile terminal devices such as mobile phones and tablets to achieve the direction control function in screen projection. During the dual-channel control process, users can project the media playback screen of terminal devices such as computers, mobile phones, and tablets or the screen projection screen of mobile terminals onto the screen of the display device. When users perform operations on the screen, such as handwriting, adding annotations, panning, etc., these operations can also be real-time fed back to the terminal device or mobile terminal to achieve a two-way interaction effect and improve the user experience.

[0004] However, when implementing the dual-channel control function, it can only be executed on a touch-supported display device or by connecting an external mouse to the display device, and it is impossible to control the media playback screen in the terminal device or the screen projection screen in the mobile terminal. In this way, when users use the remote control to operate the screen interface of the display device, they can only view the content in the interface, and it is impossible to control the media playback screen in the terminal device and the screen projection screen in the mobile terminal in dual-channel control. Summary of the Invention

[0005] Some embodiments of this application provide a display device and a control method for a dual-channel window to solve the problem that the media playback screen and the screen projection screen of the terminal device cannot be controlled during the dual-channel control process.

[0006] In a first aspect, some embodiments of this application provide a display device, including:

[0007] A display, configured to display a user interface; the user interface includes a floating window and a media playback window, the floating window is used to display the screen mirroring of a first terminal device; the media playback window is used to display the media playback screen of a second terminal device; the floating window is set on the upper layer of the media playback window;

[0008] A communicator, configured to establish communication connections with the first terminal device and the second terminal device respectively;

[0009] A controller, configured to:

[0010] In response to a key instruction input by the user based on the floating window, obtain the anchor point coordinates and key information of the key instruction; the floating window includes a first display area, and the first display area is used to display the screen mirroring; the screen mirroring display height of the screen mirroring is equal to the screen height of the display; the non-screen mirroring area outside the first display area in the floating window is a transparent view; the media playback window includes a second display area, and the second display area is used to display the media playback screen, and the second display area is calculated according to the screen size and the first display area;

[0011] Determine the operation area of the key instruction according to the anchor point coordinates;

[0012] Generate control information according to the operation area and the key information, and send the control information to the first terminal device or the second terminal device, and the control information is used to reversely control the first terminal device or the second terminal device.

[0013] In a second aspect, some embodiments of the present application provide a control method for a dual-channel window, which can be applied to the display device in the first aspect. The display device includes a display, a communicator and a controller. The control method for the dual-channel window includes:

[0014] In response to a key instruction input by the user based on the floating window, obtain the anchor point coordinates and key information of the key instruction; the floating window includes a first display area, and the first display area is used to display the screen mirroring; the screen mirroring display height of the screen mirroring is equal to the screen height of the display; the non-screen mirroring area outside the first display area in the floating window is a transparent view; the media playback window includes a second display area, and the second display area is used to display the media playback screen, and the second display area is calculated according to the screen size and the first display area;

[0015] Determine the operation area of the key instruction according to the anchor point coordinates;

[0016] Generate control information based on the operation area and key information, and send the control information to the first terminal device or the second terminal device, where the control information is used to reversely control the first terminal device or the second terminal device.

[0017] As can be seen from the above technical solutions, some embodiments of the present application provide a display device and a control method for a dual-channel window. In response to a key instruction input by a user based on a floating window, the display device obtains the anchor point coordinates and key information of the key instruction; determines the operation area of the key instruction according to the anchor point coordinates; generates control information based on the operation area and the key information, and executes reverse control of the first terminal device or the second terminal device according to the control information. The display device can respond to a key instruction input by a user based on a floating window, determine the device for reverse control by judging the operation area of the key instruction, and send the control information to different terminal devices through different applications, so as to realize reverse control of the first terminal device or the second terminal device during the dual-channel control process, and solve the problem that the media playback screen and the screen mirroring screen of the terminal device cannot be controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in some embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a schematic diagram of an operation scenario between a display device and a control device provided by some embodiments of the present application;

[0020] Figure 2 It is a block diagram of the hardware configuration of a display device provided by some embodiments of the present application;

[0021] Figure 3 It is a block diagram of the hardware configuration of a control device provided by some embodiments of the present application;

[0022] Figure 4 It is a schematic diagram of the software configuration in a display device provided by some embodiments of the present application;

[0023] Figure 5 It is a schematic diagram of the process of a display device executing a control method for a dual-channel window provided by some embodiments of the present application;

[0024] Figure 6 It is a schematic diagram of the layer stacking scenario between a floating window and a media playback window provided by some embodiments of the present application;

[0025] Figure 7Schematic diagram of the distribution of floating windows provided by some embodiments of the present application;

[0026] Figure 8 Schematic diagram of the timing of the display device executing the dual-channel window control method provided by some embodiments of the present application;

[0027] Figure 9 Schematic diagram of the effect of the display device executing the dual-channel window control mode provided by some embodiments of the present application;

[0028] Figure 10 Schematic diagram of the range of floating windows provided by some embodiments of the present application;

[0029] Figure 11 Schematic diagram of the range of the second display area provided by some embodiments of the present application;

[0030] Figure 12 Schematic diagram of the coordinate range of the second display area provided by some embodiments of the present application;

[0031] Figure 13 Schematic diagram of the anchor point adjustment interface scenario provided by some embodiments of the present application;

[0032] Figure 14 Schematic diagram of the process of a display device performing reverse control on a first terminal device or a second terminal device according to control information provided by some embodiments of the present application;

[0033] Figure 15 Schematic diagram of the process of a display device determining the target anchor point coordinates according to the direction key information provided by some embodiments of the present application;

[0034] Figure 16 Schematic diagram of another process of a display device performing reverse control on a first terminal device or a second terminal device according to control information provided by some embodiments of the present application. Detailed implementation manners

[0035] The embodiments will be described in detail below, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following embodiments do not represent all the implementation manners consistent with the present application. They are only examples of the systems and methods consistent with some aspects of the present application detailed in the claims.

[0036] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described implementation manners, and is not intended to limit the implementation manners of the present application. Unless otherwise specified, these terms should be understood in their ordinary and common meanings.

[0037] In this application, terms such as "first", "second", "third", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar or like objects or entities, and do not necessarily imply a limitation on a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0038] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components does not necessarily have to be limited to all the components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0039] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or a combination of hardware or / and software code that can perform functions related to that element.

[0040] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided for some embodiments of this application. As Figure 1 shown, the user can operate the display device 200 through touch operations, a mobile terminal 300 and a control device 100. Among them, the control device 100 can be a remote control, a stylus, etc.

[0041] In some embodiments, the mobile terminal 300 and the display device 200 can install software applications, and achieve connection and communication through network communication protocols to achieve the purpose of one-to-one control operations and data communication. It is also possible to transmit the audio and video content displayed on the mobile terminal 300 to the display device 200 to achieve a synchronous display function.

[0042] As Figure 1 also shown, the display device 200 also performs data communication with the server 400 through various communication methods. The display device 200 is allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0043] In addition to providing a broadcast receiving television function, the display device 200 can also additionally provide an intelligent network television function with computer support functions, including but not limited to, network television, smart television, Internet Protocol Television (IPTV), etc.

[0044] Figure 2 Provided for some embodiments of this application Figure 1 is a hardware configuration block diagram of the display device 200 in

[0045] In some embodiments, the display device 200 includes at least one of a tuner demodulator 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0046] In some embodiments, the detector 230 is configured to collect signals of the external environment or for external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures. Alternatively, the detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0047] In some embodiments, the display 260 includes a display screen component for presenting a picture, and a driving component for driving image display, which is configured to receive an image signal output from the controller and display video content, image content, a menu control interface, and a user control UI interface, etc.

[0048] In some embodiments, the communicator 220 is a component for communicating with an external device or a server 400 according to various communication protocol types. The display device 200 may be provided with multiple communicators 220 according to different supported communication methods. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communicator 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communicator 220 including a Bluetooth function.

[0049] The communicator 220 can enable the display device 200 to communicate with an external device or a server 400 in a wireless or wired connection manner. Among them, the wired connection can connect the display device 200 with an external device through components such as a data cable and an interface. The wireless connection can connect the display device 200 with an external device through a wireless signal or a wireless network. The display device 200 can directly establish a connection relationship with an external device, or indirectly establish a connection relationship through a gateway, a router, a connecting device, etc.

[0050] In some embodiments, the display device 200 can also support establishing communication connection relationships with multiple external devices simultaneously. For example, the external devices can be terminal devices 500 such as mobile phones and tablets. The first terminal device 510 and the second terminal device 520 can establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices can also connect to the display device 200 through different types of connection methods. For example, the first terminal device 510 connects to the display device 200 through a wireless local area network; the second terminal device 520 connects to the display device 200 through Bluetooth.

[0051] To achieve a communication connection between the display device 200 and the terminal device 500, communicators supporting the same communication type are respectively provided on the display device 200 and the terminal device 500. For example, when the display device 200 is provided with a communicator 220 including a WiFi module, the terminal device 500 should also be provided with a communicator including a WiFi module. Moreover, while the same type of communicators are provided, a specific communication transmission protocol is also required between the display device 200 and the terminal device 500 for data transfer. For example, WiFi protocol, Bluetooth connection protocol, ZigBee protocol, NFC protocol, etc.

[0052] In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0053] In some embodiments, the controller 250 and the tuner demodulator 210 may be located in different separate devices, that is, the tuner demodulator 210 may also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0054] In some embodiments, the user can input a user command on the graphical user interface (GUI) displayed on the display 260, and then the user input interface receives the user input command through the graphical user interface (GUI).

[0055] In some embodiments, the user interface 280 is an interface that can be used to receive control inputs.

[0056] Figure 3 For some embodiments provided by this application Figure 1 The hardware configuration block diagram of the control device. As Figure 3 shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0057] The control device 100 is configured to control the display device 200, and can receive user input operation instructions, and convert the operation instructions into instructions recognizable and responsive by the display device 200, playing an intermediary role in the interaction between the user and the display device 200.

[0058] In some embodiments, the control device 100 can be an intelligent device. For example: The control device 100 can install various applications for controlling the display device 200 according to user needs.

[0059] In some embodiments, as Figure 1 shown, after installing the application of the control and display device 200, the mobile terminal 300 or other intelligent electronic devices can perform functions similar to those of controlling the device 100.

[0060] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between internal components and the data processing functions between the external and internal.

[0061] Under the control of the controller 110, the communication interface 130 realizes the communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, and other near - field communication modules.

[0062] The user input / output interface 140, where the input interface includes at least one of a microphone 141, a touchpad 142, a sensor 143, a button 144, and other input interfaces.

[0063] In some embodiments, the control device 100 includes at least one of the communication interface 130 and the input / output interface 140. The communication interface 130 configured in the control device 100, such as modules like WiFi, Bluetooth, NFC, etc., can encode the user input instructions through the Wi - Fi protocol, or the Bluetooth protocol, or the NFC protocol and send them to the display device 200.

[0064] The memory 190 is used to store various operation programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0065] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0066] As Figure 4 , in some embodiments, the system is divided into four layers, from top to bottom are the application layer (abbreviated as "application layer"), the application framework layer (abbreviated as "framework layer"), the Android runtime and system library layer (abbreviated as "system runtime library layer"), and the kernel layer.

[0067] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system setting programs, or clock programs that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0068] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions of the applications in the application layer. Through the API interface, applications can access the resources in the system and obtain system services during execution.

[0069] As Figure 4 shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc. Among them, the view system can design and implement the interfaces and interactions of applications. The view system includes lists, grids, text boxes, buttons, etc. The managers include at least one of the following modules: The activity manager is used to interact with all the activities running in the system; the location manager is used to provide access to the system location service for system services or applications; the package manager is used to retrieve various information related to the application packages currently installed on the device; the notification manager is used to control the display and clearing of notification messages; the window manager is used to manage the icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0070] In some embodiments, the activity manager is used to manage the life cycles of various applications and the usual navigation back functions, such as controlling the exit, opening, and backward movement of applications. The window manager is used to manage all window programs, such as obtaining the size of the display screen, determining whether there is a status bar, locking the screen, taking screenshots, and controlling the changes in the display window (such as shrinking the display window, jittering display, distorting display, etc.).

[0071] In some embodiments, the system runtime layer provides support for the upper layer, i.e., the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries included in the system runtime layer to implement the functions that the framework layer needs to achieve.

[0072] In some embodiments, the kernel layer is the layer between hardware and software. As Figure 4 shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0073] The above embodiments illustrate the hardware / software architecture and function implementation of a display device 200, etc. The display device 200 is an intelligent device that can present a user interface and support user interaction. Taking a smart TV as an example, a smart TV is a display device based on Internet application technology, with an open operating system and a chip, and has an open application platform, enabling two-way human-computer interaction functions. A smart TV can integrate multiple functions such as audio-visual, entertainment, and data to meet the diverse and personalized needs of users. For example, users can watch various movies, TV dramas, variety shows, etc. on a smart TV, and can also connect terminal devices such as mobile phones and tablets to the smart TV to realize functions such as screen mirroring and screen projection.

[0074] To meet the user's demand for multi-device synchronous control, with the improvement of device chips, performance, etc., some display devices already have the function of dual-channel synchronous control. The emergence of the dual-channel synchronous control function can more conveniently realize the collaborative work of multiple devices, improving the device working efficiency and user experience.

[0075] In dual-channel synchronous control, one channel can be connected to a High Definition Multimedia Interface (HDMI) device for screen mirroring and reverse control. Among them, HDMI is a fully digital audio-visual interface that can transmit uncompressed video data and multi-channel audio signals. In dual-channel synchronous control, one HDMI interface is used to connect to high-definition multimedia devices such as computer devices to realize the screen mirroring and reverse control function. For example, users can project the content on the computer screen onto the display device and operate on the display device. For example, handwriting, adding annotations, etc., and these operations will be real-time feedback to the computer screen, thus realizing two-way interaction. That is to say, in the process of dual-channel control, users can project the media playback screen of terminal devices 500 such as computers, mobile phones, and tablets or the screen projection screen of mobile terminals onto the screen of the display device 200. In this way, users can view the content of multiple devices on one screen, which is convenient for comparison and analysis. At the same time, when users perform operations on the screen, such as handwriting, adding annotations, panning, etc., these operations can also be real-time feedback to the terminal device or mobile terminal to achieve the effect of two-way interaction.

[0076] The implementation of the dual-channel synchronous control function not only improves the user's work efficiency but also brings a richer entertainment experience to the user. For example, in an educational scenario, a teacher can project the courseware content on the computer screen onto the screen of the display device 200 through the dual-channel synchronous control function, and at the same time control the direction and size of the projected screen through the mobile terminal 300, which is convenient for classroom teaching. In a home entertainment scenario, a user can project the screen content of devices such as a TV box and a game console onto the screen of the display device 200 through the dual-channel synchronous control function, and at the same time control the direction and size of the projected screen through a mobile terminal 300 such as a mobile phone to achieve a more immersive viewing and gaming experience.

[0077] However, when implementing the dual-channel control function, it can only be executed on the touch-supported display device 200 or by connecting an external mouse to the display device 200, and it is impossible to control the media playback screen in the terminal device 500 or the projected screen in the mobile terminal 300. In this way, when a user operates the screen interface of the display device 200 using a remote control, only the content in the interface can be viewed, and it is impossible to control the media playback screen in the terminal device 500 and the projected screen in the mobile terminal 300 in the dual-channel control.

[0078] To solve the problem that it is impossible to control the media playback screen and the projected screen of the terminal device during the dual-channel control process, some embodiments of the present application provide a display device 200, and the display device 200 includes a display 260, a communicator 220, and a controller 250. Among them, the display 260 is configured to display a user interface, and the user interface includes a floating window and a media playback window. Among them, the floating window is used to display the projected screen of the first terminal device 510, and the media playback window is used to display the media playback screen of the second terminal device 520, and the floating window is set on the upper layer of the media playback window. The communicator 220 is configured to establish communication connections with the first terminal device 510 and the second terminal device 520 respectively, and the controller 250 is configured to execute the control method of the dual-channel window. The display device 200 can respond to a key instruction input by the user based on the floating window, determine the device for reverse control by judging the operation area of the key instruction, and send the control information to different terminal devices through different applications, so as to realize the reverse control of the first terminal device or the second terminal device during the dual-channel control process and solve the problem that it is impossible to control the media playback screen and the projected screen of the terminal device.

[0079] For the convenience of understanding the technical solutions in some embodiments of the present application, the following will be described in detail with reference to some specific embodiments and drawings for each step. Figure 5 It is a schematic flowchart of the control method for the dual-channel window executed by the display device provided in some embodiments of the present application, as Figure 5As shown in the figure, when the display device 200 executes the control of the dual-window, it may include the following steps S1 - S3, and the specific content is as follows:

[0080] Step S1: In response to a key instruction input by the user based on the floating window, the display device 200 acquires the anchor point coordinates and key information of the key instruction.

[0081] Figure 6 Schematic diagram of the layer stacking scenario of the floating window and the media playback window provided by some embodiments of the present application. As Figure 6 shown, two windows can be set in the display device 200, namely the floating window and the media playback window. Among them, the floating window is set on the upper layer of the media playback window. The floating window and the media playback window can be used for different application displays respectively.

[0082] Figure 7 Schematic diagram of the distribution of the floating window provided by some embodiments of the present application. As Figure 7 shown, in some embodiments, the floating window includes a first display area, i.e., the screen mirroring area, and the first display area is used to display the screen mirroring image, and the screen mirroring display height of the screen mirroring image is equal to the screen height of the display 260. Combining Figure 8 , the non-screen mirroring area outside the first display area in the floating window is a transparent view. For example, this transparent view can be set through the screen mirroring application. The media playback window includes a second display area, and the second display area is used to display the media playback image. The media playback image can include video images, game images, web page images, etc. Among them, the second display area is calculated based on the screen size and the first display area, and the determination method of the second display area will be introduced in detail later.

[0083] In order to implement the function of the display device 200 to execute the dual-window control, in some embodiments, multiple functional modules or applications can be deployed in the display device 200. Figure 8 Schematic diagram of the timing of the method for the display device to execute the dual-window control provided by some embodiments of the present application. As Figure 8 shown, a screen mirroring application, a live TV application, a platform module, and a middleware module can be deployed in the display device 200. At the same time, the display device 200 can also establish a communication connection relationship with external devices such as the second terminal device 520 and the screen mirroring device such as the first terminal device 510. Among them, the screen mirroring application can be used to display the screen mirroring image and send touch events, relevant control information, etc. to the first terminal device 510. The live TV application can be used to display the media playback image and send touch events, relevant control information, etc. to the second terminal device 520. The platform module can be used to receive events or messages sent by the screen mirroring application, the live TV application, etc., and the middleware module is used to execute media image playback, set the size and position of the media image, etc.

[0084] In some embodiments, the floating window can serve the screen mirroring application, and in combination with Figure 7 , that is, the screen mirroring area and the non-screen mirroring area can serve the screen mirroring application. The screen mirroring application can display the screen mirroring image of the first terminal device 510 in the form of a window and in the form of a floating window. When the screen mirroring application is started, the screen mirroring image can be located on the left or right side of the display device 200 such as a TV. Since it is located on the upper layer, the floating window will obtain all user key operations. At the same time, the non-screen mirroring area is set to a transparent view, so that the media playback image played in the media playback window located below it will not be affected, that is, it will not be blocked.

[0085] It should be noted that in the embodiments of the present application, since the floating window is on the top layer, it can obtain all input operation instructions and operation events. And the screen mirroring application serves the floating window. Therefore, the screen mirroring application can forward relevant instructions and operation events to other applications or modules. For example, it can be forwarded to the live TV application, the platform module, and the middleware module. These applications or modules will execute their specific contents after receiving the instructions and events.

[0086] In some embodiments, the display device 200 further includes a device interface 240, and the device interface 240 is configured to connect an external device. In response to the plug and unplug signal of the external device accessing the device interface 240, the display device 200 can switch the playback channel to the data channel corresponding to the device interface 240 and obtain the media playback image through the data channel. In response to the screen mirroring instruction for displaying the screen mirroring image, the display device 200 can generate a window switching instruction, and according to the window switching instruction, control the display 260 to display the screen mirroring image in the first display area of the floating window and display the media playback image in the second display area of the media playback window.

[0087] Exemplarily, the external device can be a second terminal device 520. When the user connects the second terminal device 520 to the TV, in response to the plug and unplug signal accessing the device interface 240, the display device 200 can switch to the data channel corresponding to the device interface 240 through the live TV application. In combination with Figure 8 , taking the user switching to the HDMI channel as an example, the switching instruction can be sent to the live TV application, and the live TV application can send the event of switching to the HDMI channel to the platform module, and the platform module then sends it to the middleware module. After receiving it, the middleware module can obtain the media playback image. When detecting the screen mirroring instruction of the first terminal device 510, the TV will generate a window switching instruction and switch to the dual-window control mode. Figure 9 For the effect schematic diagram of the display device executing the dual-window control mode provided by some embodiments of the present application, in combination with Figure 8 and Figure 9, in the dual-window control mode, the projection area of the floating window displays the projection screen, and the second display area of the media playback window displays the media playback screen. Among them, the range of the second display area is smaller than the non-projection area. Below, the setting methods of the first display area and the second display area will be introduced in detail.

[0088] When the display device 200 sets the first display area, it can first obtain the screen height and screen width of the display 260, then set the projection display height of the projection screen to the screen height, and then obtain the original projection screen height and original projection screen width of the projection screen, and calculate the projection scaling ratio according to the ratio of the original projection screen height to the screen height. Then, calculate the projection display width of the projection screen in the first display area according to the ratio of the original projection screen width to the projection scaling ratio. Finally, set the first display area according to the projection display height and the projection display width.

[0089] Exemplarily, when the user watches the media playback screen in the live TV application, in response to the projection instruction, the screen height and screen width of the display 260 can be obtained first. After the projection is started, after the display device 200 receives the projection protocol of the first terminal device 510, the projection display height of the projection screen is set to the screen height of the TV. Assume that W1 is the screen width of the TV, H1 is the screen height of the TV, W2 is the original projection screen width of the projection screen, and H2 is the original projection screen height of the projection screen. Then the projection scaling ratio Z1 = H2 / H1, that is, the ratio of the original projection screen height to the screen height. Then the projection display width of the projection screen in the first display area, that is, the projection display width W3 displayed on the TV side, is W3 = W2 / Z1, and the projection display height H3 = H1. In this way, after the projection display height H3 and the projection display width W3 are calculated, the first display area can be set.

[0090] In order to obtain the operation instructions executed by the user, the projection application can be displayed on the TV screen in the form of a floating window, and the window can be set to obtain the focus of the system for receiving the user's key operations. Figure 10 The following is a schematic diagram of the range of the floating window provided by some embodiments of the present application. As Figure 10 shown, in some embodiments, the width of the floating window is the screen width W1 of the TV, and the height of the floating window is the screen height H1 of the TV. On the right side of the TV is the projection screen with a projection display width of W3 and a projection display height of H1, that is, the first display area. On the left side of the projection screen is the non-projection area. The width of the non-projection area W4 = W1 - W3, and the height of the non-projection area is still H1. The non-projection area is the range of the transparent view.

[0091] When setting the second display area, the display device 200 can first determine the non-cast screen area based on the screen size and the first display area. Here, the screen size includes the screen height and the screen width. Then, set the non-cast screen area as a transparent view, and align the media playback window with the non-cast screen area. After alignment, set the playback screen width of the media playback screen to the width of the non-cast screen area, and obtain the original media screen height and the original media screen width of the media playback screen. Then, calculate the display scaling ratio according to the ratio of the original media screen width to the playback screen width, calculate the playback screen height of the media playback screen according to the ratio of the original media screen height to the display scaling ratio, and finally determine the second display area based on the playback screen height and the playback screen width.

[0092] Exemplarily, after displaying the cast screen in the first display area, the non-cast screen area can be determined according to the screen size and the first display area. Refer to Figure 8 , the cast application can send the display state of the cast screen, the dimensions W4 and H1 of the non-cast screen area to the live TV application in the form of a notification. After receiving the notification, the live TV application sets the non-cast screen area as a transparent view, and scales the media playback screen by aligning the media playback window with the non-cast screen area in terms of width, that is, calculates the second display area. Set the playback screen width HW1 of the media playback screen to the width W4 of the non-cast screen area. Assume that the original media screen width of the media playback screen is HOW1, the original media screen height is HOH1, and the playback screen width of the media playback screen is HW1, HW1 = W4. Then, calculate the display scaling ratio HZ1 = HOW1 / W4 of the media playback screen according to the ratio of the original media screen width to the playback screen width. When the cast screen is set on the right side of the screen, the media playback screen can be set on the left side of the screen, and its position can be set as needed. After determining the display scaling ratio, calculate the playback screen height HH1 of the media playback screen according to the ratio of the original media screen height HOH1 to the display scaling ratio HZ1, that is, HH1 = H0H1 / HZ1. After calculating the above values, the live TV application can send the data to the platform module, and the platform module can then send it to the middleware module, and the second display area can be set according to the values in the middleware module. The second display area can also be set by the live TV application, and the functions of each application can be set according to actual requirements.

[0093] Figure 11 The figure shows a schematic diagram of the range of the second display area provided by some embodiments of the present application, as Figure 11As shown, after the height HH1 and width HW1 of the playback screen are determined, the second display area can be determined. It can be seen that since the non-screen mirroring area is a transparent view, the content displayed in the second display area will not be blocked. In this way, when the screen mirroring screen is displayed in the first display area and the media playback screen is displayed in the second display area, the effect of two windows being displayed simultaneously can be presented.

[0094] To improve the display effect of the user interface, when the width HW1 of the playback screen in the second display area is equal to the width W4 of the non-screen mirroring area, the media playback screen can be centered and displayed in the non-screen mirroring area. Figure 12 Schematic diagram of the coordinate range of the second display area provided by some embodiments of the present application, as Figure 12 shown. In some embodiments, taking the upper left corner coordinates of the screen as (0, 0) as an example, the floating window range from the upper left corner to the lower right corner is the screen range, which is a rectangular area of (0, 0) to (W1, H1). The coordinate range of the screen mirroring screen display from the upper left corner to the lower right corner is a rectangular area of (W4, 0) to (W1, H1). The setting method of the second display area can be that the distance from the top of the second display area to the top of the screen is the same as the distance from the bottom of the second display area to the bottom of the screen, that is, there are black borders of the same width at the top and bottom of the second display area. This black border is the black border formed after the media playback screen is scaled, so as to ensure the centered display of the second display area. Correspondingly, when the height of the playback screen is HH1, its coordinate range from the upper left corner to the lower right corner is a rectangular area of (0, (H1 - HH1) / 2) to (W4, (H1 / 2 + HH1 / 2)). In this way, the setting of the second display area in the media playback window is completed.

[0095] Continue to refer to Figure 8 , after the first display area of the floating window and the second display area of the media playback window are set, when the user inputs a key command based on the floating window, for example, when inputting a key command through a remote control, the upper-layer floating window can obtain the anchor point coordinates and key information of the key command through the screen mirroring application, and perform subsequent operations based on the anchor point coordinates and key information. After step S1 is executed, the following step S2 can be executed.

[0096] Step S2: The display device 200 determines the operation area of the key command according to the anchor point coordinates.

[0097] When the user inputs a key command, since the floating window and the media playback window serve different applications respectively. For example, the floating window serves the screen mirroring application, and the media playback window serves the live TV application. Then the upper-layer floating window can obtain the input key command and can detect the operation behavior and operation path of the key command. For example, whether the key command operates on the screen mirroring area or the non-screen mirroring area, and whether the operation event is a press event, a slide event, a long press event, etc., and can distribute to different applications according to different operation areas. For example, for the operation on the screen mirroring area, it can be distributed to the screen mirroring application. For the operation on the non-screen mirroring area, it can be first distributed from the screen mirroring application to the live TV application, and then the live TV application sends it to the corresponding terminal device, and then through different applications, it is distributed to different terminal devices, achieving the effect of reverse control of the terminal device.

[0098] Exemplarily, the screen mirroring area belongs to the call range of the screen mirroring application. The screen mirroring application is responsible for displaying the screen mirroring picture of the first terminal device 510 and can distribute the touch operation corresponding to the key command to the first terminal device 510, so as to realize the reverse control of the first terminal device 510. And the non-screen mirroring area belongs to the call range of the live TV application. The live TV application is responsible for displaying the media playback picture of the second terminal device 520 and can distribute the touch operation corresponding to the key command to the second terminal device 520, so as to realize the reverse control of the second terminal device 520.

[0099] In some embodiments, the operation area includes a first operation area and a second operation area. When the display device 200 determines the operation area of the key command according to the anchor point coordinates, it can divide the first display area into the first operation area and divide the non-screen mirroring area into the second operation area. Then it can detect the falling range of the anchor point coordinates. If the anchor point coordinates fall within the first display area, it can determine that the operation area of the key command is the first operation area. If the anchor point coordinates fall within the non-screen mirroring area, it can determine that the operation area of the key command is the second operation area.

[0100] Exemplarily, the anchor point coordinates can be regarded as a virtual reference point, which can represent the position of the user's finger or other input device on the screen. By detecting the falling range of the anchor point coordinates, the display device 200 can determine in which operation area the user intends to execute the key instruction. If the anchor point coordinates fall within the screen mirroring area, then the display device 200 can determine that the operation area of the key instruction is the first operation area, which means that the user intends to execute the key instruction within the first display area. If the anchor point coordinates fall within the non-screen mirroring area, then the display device 200 can determine that the operation area of the key instruction is the second operation area. This means that the user intends to execute the key instruction within the non-screen mirroring area. In this way, the operation area of the key instruction can be determined according to the anchor point coordinates. After step S2 is completed, the following step S3 can be executed.

[0101] Step S3: The display device 200 generates control information according to the operation area and the key information, and sends the control information to the first terminal device 510 or the second terminal device 520. The control information is used to reversely control the first terminal device 510 or the second terminal device 520.

[0102] In some embodiments, the anchor point coordinates are adjustable. When entering the user interface, the anchor point coordinates can have a default initial anchor point position. When the anchor point coordinates are within the screen mirroring area, that is, within the rectangular area of (W4, 0) to (W1, H1), if the user adjusts the anchor point coordinates through the menu key, the adjustable range of the anchor point is (W4, 0) to (W1, H1). That is to say, without switching windows, the adjustable range of its anchor point is always (W4, 0) to (W1, H1). When switching windows, for example, switching from the first display area of the floating window to the second display area of the media playback window, the adjustable range of the anchor point is the rectangular area of (0, (H1 - HH1) / 2) to (W4, (H1 / 2 + HH1 / 2)).

[0103] To adjust the anchor point coordinates, the display device 200 can obtain the anchor point positioning instruction, and determine the anchor point coordinates of the target anchor point position according to the obtained anchor point positioning instruction and the key instruction input by the user. First, the display device 200 can obtain the anchor point positioning instruction, extract the anchor point positioning information in the anchor point positioning instruction, and determine the initial anchor point position in the user interface according to the anchor point positioning information. Then obtain the key instruction input by the user, extract the key information in the key instruction, then obtain the movement information of the anchor point according to the key information, and determine the target anchor point position according to the movement information, and determine the anchor point coordinates of the target anchor point position.

[0104] Exemplarily, the anchor positioning instruction can be an instruction input by the user through the remote control. The TV will parse this instruction and extract the anchor positioning information from it. The anchor positioning information may include the initial position, moving direction, and distance of the target anchor, etc. According to this information, the display device 200 can determine an initial anchor position on the user interface. Next, the display device 200 needs to obtain the key instruction input by the user. This instruction can be an instruction input by the user pressing a certain key on the remote control. The display device 200 will parse the instruction and extract the key information from it. Among them, the key information can include the moving direction, distance, etc. of the user instruction. According to this information, the display device 200 can calculate the moving information of the anchor, and then determine the target anchor position according to the moving information.

[0105] Figure 13 Schematic diagram of the anchor adjustment interface scenario provided by some embodiments of the present application, as Figure 13 shown, in some embodiments, when entering the anchor adjustment mode, an explicit circular mark will be displayed at the anchor coordinate position on the screen. This circular mark can move following the direction keys of the remote control. For example, the circular mark in the solid line part is the initial anchor position. After adjustment, when the remote control moves downward, the target anchor position will become the circular mark in the dashed line part, that is, the anchor will move to the target anchor position at the dashed line position.

[0106] In order to accelerate the adjustment speed of the anchor, in some embodiments, the display device 200 can adopt an acceleration strategy in the following manner. In response to the operation of long pressing the direction key, the display device 200 can gradually increase the moving speed of the anchor. For example, after recognizing that the user has long pressed the direction key of the remote control for 3 seconds, the moving speed of the anchor can gradually increase from 10 pixels per second to 100 pixels per second. The specific increasing gradient can be set according to actual needs, and the present application does not make specific limitations.

[0107] It can be understood that after adjusting the anchor, the target anchor position will become the starting coordinate position of subsequent events. In some embodiments, the subsequent event can be the key information of the key instruction. Then the display device 200 can generate control information according to the operation area and the key information, and send the control information to the first terminal device 510 or the second terminal device 520. This control information is used to reversely control the first terminal device 510 or the second terminal device 520.

[0108] Figure 14 Schematic diagram of the process for a display device to perform reverse control of a first terminal device or a second terminal device according to control information provided by some embodiments of the present application, as Figure 14As shown, in some embodiments, the key information of the key command may include the confirmation key information. For example, the key information may be the information corresponding to the confirmation key in the remote control menu key. When the key information is the confirmation key information, the following may be included when the display device 200 performs reverse control of the first terminal device 510 or the second terminal device 520.

[0109] In response to an operation instruction input by the user based on the operation area, the display device 200 obtains the operation information of the operation instruction, where the operation information at least includes the anchor coordinate information. Then, according to the anchor coordinate information, the operation area of the operation instruction is determined. If the operation area is the first operation area and the key information is the confirmation key information, the anchor coordinates are obtained, and the screen mirroring relative coordinates of the anchor relative to the preset corner point of the first display area are calculated based on the anchor coordinates and the first display area. Then, the first terminal coordinates of the anchor relative to the first terminal device 510 are calculated based on the screen mirroring relative coordinates and the screen mirroring zoom ratio. Finally, control information is generated according to the first operation area and the confirmation key information, and the control information and the first terminal coordinates are sent to the first terminal device 510 so that the first terminal device 510 executes the control information. If the operation area is the second operation area and the key information is the confirmation key information, the display device 200 may obtain the anchor coordinates, and calculate the screen relative coordinates of the anchor relative to the preset corner point of the second display area based on the anchor coordinates and the second display area. Then, the second terminal coordinates of the anchor relative to the second terminal device 520 are calculated based on the screen relative coordinates and the display zoom ratio. Then, control information may be generated according to the second operation area and the confirmation key information, and the control information and the second terminal coordinates are sent to the second terminal device 520 so that the second terminal device 520 executes the control information.

[0110] Exemplarily, when the key information is the confirmation key input by the user through the remote control, there is no need to adjust the position of the anchor at this time. The screen mirroring application may obtain the anchor coordinate information included in the operation information. For example, the current anchor coordinate information is (MX1, MH1). After determination, if the anchor coordinates are located in the screen mirroring area and the key information is the confirmation key information, combined with Figure 8, the screen mirroring application in the upper layer can convert the confirmation key information into control information for the first terminal device 510. The specific conversion process can be as follows: taking the preset corner point as the upper left corner as an example, first calculate the screen mirroring relative coordinates ((MX1 - W4), MH1) of the anchor point relative to the upper left corner of the screen mirroring area according to the anchor point coordinates (MX1, MH1) and the first display area, that is, the relative coordinates of the operation button position relative to the upper left corner of the screen mirroring area, and then calculate the first terminal coordinates ((MX1 - W4) * Z1, MH1 * Z1) of the anchor point relative to the first terminal device 510 according to the screen mirroring relative coordinates ((MX1 - W4), MH1) and the screen mirroring zoom ratio Z1. After the first terminal coordinates are determined, the display device 200 can generate control information according to the first operation area and the confirmation key information, and send the control information and the first terminal coordinates ((MX1 - W4) * Z1, MH1 * Z1) to the first terminal device 510 through the screen mirroring application, so that the first terminal device 510 executes the control information, and thus the reverse control of the first terminal device 510 can be realized.

[0111] Continue to refer to Figure 8 , if the anchor point coordinates are located in the second operation area and the button information is confirmation key information, the screen mirroring application in the upper layer can convert the confirmation key information into control information for the second terminal device 520. Then the screen mirroring application can send the control information to the live TV application. After receiving the control information, the live TV application can directly send it to the second terminal device 520, or send it to the second terminal device 520 through the breaking platform module and the middleware module in sequence. In some embodiments, the specific conversion process of the control information can be as follows: still taking the preset corner point as the upper left corner as an example, first calculate the screen relative coordinates (MX1, MH1 - (H1 - HH1) / 2) of the anchor point relative to the preset corner point of the second display area according to the anchor point coordinates (MX1, MH1) and the second display area, and then calculate the second terminal coordinates ((MX1 * HZ1, (MH1 - (H1 - HH1) / 2)) * HZ1) of the anchor point relative to the second terminal device 520 according to the screen relative coordinates (MX1, MH1 - (H1 - HH1) / 2) and the display zoom ratio HZ1. After the second terminal coordinates are determined, the display device 200 can generate control information according to the second operation area and the confirmation key information, and send the control information and the second terminal coordinates ((MX1 * HZ1, (MH1 - (H1 - HH1) / 2)) * HZ1) to the second terminal device 520, so that the second terminal device 520 executes the control information, and thus the reverse control of the second terminal device 520 can be realized.

[0112] In some embodiments, when the display device 200 performs reverse control of the first terminal device 510 or the second terminal device 520 in the background, the above process may be executed in the form of sending events in the background. Taking the display device 200 performing direction control on the first terminal device 510 as an example, in response to the user pressing the confirmation key through the remote control, an event of KEY_DOWN can be generated through the remote control. After the display device 200 receives the KEY_DOWN event sent by the remote control, an ACTION_DOWN event of touch can be generated, and the coordinates are the anchor point coordinates ((MX1 - W4) * Z1, MH1 * Z1). Then, it can be sent to the first terminal device 510 through a screen mirroring protocol such as the Miracast protocol. Miracast is a wireless display standard that can wirelessly project the screen content of a terminal device onto a TV or other display device. Through the Miracast protocol, the first terminal device 510 can receive this event and perform corresponding processing according to the specific content of the event.

[0113] In some embodiments, when the touch event is a single press, the above process can be executed. After the press ends, the display device 200 can send an ACTION_UP event to the first terminal device 510 to notify the end of the single press. When the touch event is a long press event, since the key press frequency of the remote control is slower than the actual touch frequency of the first terminal device 510, in order to prevent the signal of the first terminal device 510 from being interrupted, during the process of not receiving the key lift, the display device 200 can send an ACTION_DOWN of touch every certain period, such as every 25 ms, and at the same time, the number of times of sending this event can be accumulated, such as adding 1 to the repetition count each time, while the coordinates of the event remain unchanged. This process will continue until the event of receiving the key lift is received. After receiving the event of the key lift, the display device 200 can send an ACTION_UP event of touch to the first terminal device 510 to indicate the end of the long press event. At this time, the repetition count can be cleared. This process is to ensure that the first terminal device 510 can accurately know the time point of the key lift and thus perform corresponding processing.

[0114] In some embodiments, the key information further includes direction key information, and the display device 200 can also determine the target anchor point coordinates of the first terminal device 510 and the second terminal device 520 according to the direction key information. Figure 15 Schematic diagram of the process for the display device provided in some embodiments of the present application to determine the target anchor point coordinates according to the direction key information, as Figure 15As shown, in response to the key information being direction key information, for example, when the user presses the up, down, left, or right direction keys on the remote control, the display device 200 can obtain the sliding pixel displacement, and determine the first target anchor point coordinates after the anchor point slides the sliding pixel displacement on the first terminal device 510 according to the first terminal coordinates ((MX1 - W4) * Z1, MH1 * Z1) and the sliding pixel displacement, and determine the second target anchor point coordinates after the anchor point slides the sliding pixel displacement on the second terminal device 520 according to the second terminal coordinates ((MX1 * HZ1, (MH1 - (H1 - HH1) / 2)) * HZ1) and the sliding pixel displacement.

[0115] When the key information is direction key information, when the display device 200 performs reverse control on the first terminal device 510 or the second terminal device 520, it may include the following contents. Figure 16 It is a schematic flowchart of another display device provided by some embodiments of the present application for performing reverse control on the first terminal device or the second terminal device according to control information, as Figure 16 As shown, in response to the operation instruction input by the user based on the operation area, obtain the operation information of the operation instruction, where the operation information at least includes anchor point coordinate information, and determine the operation area of the operation instruction according to the anchor point coordinate information. If the operation area is the first operation area, the display device 200 can send the first terminal coordinates, the sliding pixel displacement, and the first target anchor point coordinates to the first terminal device 510, so that the first terminal device 510 slides from the first terminal coordinates as the starting point to the first target anchor point coordinates with the sliding pixel displacement as the sliding distance; if the operation area is the second operation area, the display device 200 can send the second terminal coordinates, the sliding pixel displacement, and the second target anchor point coordinates to the second terminal device 520, so that the second terminal device 520 slides from the second terminal coordinates as the starting point to the second target anchor point coordinates with the sliding pixel displacement as the sliding distance.

[0116] In some embodiments, in response to the operation instruction input by the user based on the operation area, the display device 200 first needs to obtain the operation information of the operation instruction, and the operation information at least includes anchor point coordinate information, such as (MX1, MH1). In addition, the operation information may also include other operation-related data, such as the sliding direction, sliding speed, sliding distance, etc. After obtaining the operation information, the display device 200 can determine the operation area of the operation instruction according to the anchor point coordinate information, and judge whether the operation area is the first operation area or the second operation area. The specific judgment process has been introduced above and will not be repeated here. When the operation area is determined, the display device 200 can perform corresponding operations according to different operation areas. Specifically:

[0117] If the operation area is the first operation area, the display device 200 may send the first terminal coordinates, the sliding pixel displacement, and the first target anchor point coordinates to the first terminal device 510. In this way, the first terminal device 510 can, based on the received data, start from the first terminal coordinates and slide to the first target anchor point coordinates with the sliding pixel displacement as the sliding distance. This process can synchronize the touch screen operation on the first terminal device 510 with the content on the display device 200, realizing the reverse control of the first terminal device 510. If the operation area is the second operation area, the display device 200 may send the second terminal coordinates, the sliding pixel displacement, and the second target anchor point coordinates to the second terminal device 520. In this way, the second terminal device 520 can, based on the received data, start from the second terminal coordinates and slide to the second target anchor point coordinates with the sliding pixel displacement as the sliding distance. This process can synchronize the touch screen operation on the second terminal device 520 with the content on the display device 200, realizing the reverse control of the second terminal device 520.

[0118] Exemplarily, when the user presses the direction keys, if the anchor point coordinates are within the screen mirroring area, the device for reverse control is the first terminal device 510. The first terminal coordinates ((MX1 - W4) * Z1, MH1 * Z1) of the first terminal device 510 are the starting anchor point coordinates. The first terminal coordinates, the sliding pixel displacement, and the first target anchor point coordinates can be sent to the first terminal device 510 through the screen mirroring application. Since the operation corresponding to the direction key information is actually a sliding operation, when calculating the target anchor point coordinates of the first terminal device 510, addition or subtraction of the horizontal or vertical coordinates can be performed according to its direction keys. For example, taking the sliding pixel displacement as 100 pixels, the calculation method for addition or subtraction of its target anchor point coordinates is as follows:

[0119] When the direction key information is the left key, the target anchor point coordinates are: ((MX1 - W4) * Z1 - 100, MH1 * Z1);

[0120] When the direction key information is the right key, the target anchor point coordinates are: ((MX1 - W4) * Z1 + 100, MH1 * Z1);

[0121] When the direction key information is the up key, the target anchor point coordinates are: ((MX1 - W4) * Z1, MH1 * Z1 - 100);

[0122] When the direction key information is the down key, the target anchor point coordinates are: ((MX1 - W4) * Z1, MH1 * Z1 + 100).

[0123] When the user presses the direction key, if the anchor point coordinates are in the non-screen mirroring area, the device for reverse control is the second terminal device 520. The second terminal coordinates ((MX1*HZ1, (MH1 - (H1 - HH1) / 2))*HZ1) of the second terminal device 520 are the starting anchor point coordinates. In some embodiments, the second terminal coordinates, the sliding pixel displacement, and the second target anchor point coordinates can be sent to the second terminal device 520 through the live TV application. Since the operation corresponding to the direction key information is actually a sliding operation, when calculating the target anchor point coordinates of the second terminal device 520, addition or subtraction of the horizontal or vertical coordinates can be performed according to its direction key. For example, still taking the sliding pixel displacement as 100 pixels, the addition or subtraction calculation method for its target anchor point coordinates is as follows:

[0124] When the direction key information is the left key, the target anchor point coordinates are:

[0125] ((MX1*HZ1 - 100, (MH1 - (H1 - HH1) / 2))*HZ1);

[0126] When the direction key information is the right key, the target anchor point coordinates are:

[0127] ((MX1*HZ1 + 100, (MH1 - (H1 - HH1) / 2))*HZ1);

[0128] When the direction key information is the up key, the target anchor point coordinates are:

[0129] ((MX1*HZ1, (MH1 - (H1 - HH1) / 2))*HZ1 - 100);

[0130] When the direction key information is the down key, the target anchor point coordinates are:

[0131] ((MX1*HZ1, (MH1 - (H1 - HH1) / 2))*HZ1 + 100).

[0132] In some embodiments, when the background performs reverse control on the first terminal device 510 or the second terminal device 520, it can still be achieved in the form of sending events. Still taking the display device 200 performing direction control on the first terminal device 510 as an example, when the user presses the direction key, the display device 200 can generate a touch ACTION_DOWN event, with the coordinates being the anchor point coordinates ((MX1 - W4) * Z1, MH1 * Z1). After that, it can immediately send a touch ACTION_MOVE, that is, a sliding event, to the first terminal device 510, slide according to the sliding pixel displacement. After the sliding is completed, it then sends an ACTION_UP event to the first terminal device 510. During the process when the user has not lifted the button, the time interval for sending events can also be set, which can be comprehensively determined according to specific events and actual usage scenarios, and this application does not make specific limitations.

[0133] As can be seen from the above technical solutions, the above embodiments provide a display device 200 that, in response to a key instruction input by a user based on a floating window, obtains the anchor point coordinates and key information of the key instruction; determines the operation area of the key instruction according to the anchor point coordinates; generates control information according to the operation area and the key information, and performs reverse control on the first terminal device or the second terminal device according to the control information. Among them, the floating window includes a first display area for displaying a screen mirroring image; the screen mirroring display height of the screen mirroring image is equal to the screen height of the display 260; the non-screen mirroring area outside the first display area in the floating window is a transparent view; the media playback window includes a second display area for displaying a media playback image, and the second display area is calculated according to the screen size and the first display area. The display device 200 can, in response to a key instruction input by a user based on a floating window, determine the device for reverse control by judging the operation area of the key instruction, and send the control information to different terminal devices through different applications, thereby realizing reverse control of the first terminal device or the second terminal device during the dual-channel control process, and solving the problem that the media playback image and the screen mirroring image of the terminal device cannot be controlled.

[0134] Based on the above display device 200, some embodiments of the present application further provide a control method for a dual-channel window, which can be applied to the display device 200 in the above embodiments. The display device 200 includes a display 260, a communicator 220, and a controller 250. In some embodiments, the control method for the dual-channel window may include the following content:

[0135] In response to a key instruction input by a user based on a floating window, obtain the anchor coordinates and key information of the key instruction; the floating window includes a first display area for displaying a screen mirroring image; the screen display height of the screen mirroring image is equal to the screen height of the display 260; the non-screen mirroring area outside the first display area in the floating window is a transparent view; the media playback window includes a second display area for displaying a media playback image, and the second display area is calculated based on the screen size and the first display area;

[0136] Determine the operation area of the key instruction according to the anchor coordinates;

[0137] Generate control information according to the operation area and the key information, and perform reverse control on the first terminal device or the second terminal device according to the control information.

[0138] As can be seen from the above technical solutions, the above embodiments provide a control method for a dual-channel window. The method can respond to a key instruction input by a user based on a floating window, determine the device for reverse control by judging the operation area of the key instruction, and send the control information to different terminal devices through different applications, so as to realize the reverse control of the first terminal device or the second terminal device in the dual-channel control process, and solve the problem that the media playback image and the screen mirroring image of the terminal device cannot be controlled.

[0139] For the same and similar parts between the various embodiments in this specification, reference can be made to each other, and details will not be repeated here.

[0140] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods in the various embodiments or some parts of the embodiments of the present invention.

[0141] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the various embodiments of the present application.

[0142] For the sake of convenience in explanation, the above description has been made in connection with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and actual applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, Including: A display configured to display a user interface; the user interface includes a floating window and a media playback window, and the floating window is used to display the screen mirroring image of the first terminal device; The media playback window is used to display the media playback image of the second terminal device; The floating window is arranged on the upper layer of the media playback window; A communicator configured to establish communication connections with the first terminal device and the second terminal device respectively; A controller configured to: In response to a key instruction input by the user based on the floating window, obtain the anchor point coordinates and key information of the key instruction; the floating window includes a first display area for displaying the screen mirroring image; the screen mirroring display height of the screen mirroring image is equal to the screen height of the display; the non-screen-mirroring area outside the first display area in the floating window is a transparent view; the media playback window includes a second display area for displaying the media playback image, and the second display area is calculated according to the screen size and the first display area; Determine the operation area of the key instruction according to the anchor point coordinates; Generate control information according to the operation area and key information, and send the control information to the first terminal device or the second terminal device, and the control information is used to inversely control the first terminal device or the second terminal device.

2. The display device according to claim 1, characterized in that, It further includes a device interface configured to connect an external device, and the controller is further configured to: In response to the plugging and unplugging signal of the external device accessing the device interface, switch the playback channel to the data channel corresponding to the device interface; Obtain the media playback image through the data channel; In response to a screen mirroring instruction for displaying the screen mirroring image, generate a window switching instruction; According to the window switching instruction, control the display to display the screen mirroring image in the first display area of the floating window and display the media playback image in the second display area of the media playback window.

3. The display device according to claim 2, characterized in that, The controller is further configured to: Obtain the screen height and screen width of the display; Set the screen mirroring display height of the screen mirroring image to the screen height; Obtain the original screen mirroring image height and original screen mirroring image width of the screen mirroring image; Calculate the screen mirroring zoom ratio according to the ratio of the original screen mirroring image height to the screen height; Calculate the screen mirroring display width of the screen mirroring image in the first display area according to the ratio of the original screen mirroring image width to the screen mirroring zoom ratio; Set the first display area according to the screen mirroring display height and the screen mirroring display width.

4. The display device according to claim 3, wherein The controller is further configured to: Determine the non-screen-mirroring area according to the screen size and the first display area; the screen size includes the screen height and the screen width; Set the non-screen-mirroring area as a transparent view, and align the media playback window and the non-screen-mirroring area; Set the playback image width of the media playback image to the width of the non-screen-mirroring area; Obtain the original media image height and original media image width of the media playback image; Calculate the display zoom ratio based on the ratio of the width of the original media asset screen to the width of the playback screen; Calculate the playback screen height of the media asset playback screen based on the ratio of the height of the original media asset screen to the display zoom ratio; Determine the second display area based on the playback screen height and the playback screen width.

5. The display device according to claim 4, characterized in that, The operation area includes a first operation area and a second operation area. The step of the controller performing the operation of determining the operation area of the key command according to the anchor coordinates is further configured to: Divide the first display area into the first operation area; Divide the non-screen mirroring area into the second operation area; If the anchor coordinates fall within the first display area, determine that the operation area of the key command is the first operation area; If the anchor coordinates fall within the non-screen mirroring area, determine that the operation area of the key command is the second operation area.

6. The display device according to claim 5, wherein The controller is further configured to: Obtain an anchor positioning command; Extract the anchor positioning information in the anchor positioning command; Determine the initial anchor position on the user interface according to the anchor positioning information; Obtain a key command input by the user; Extract the key information in the key command; Obtain the movement information of the anchor according to the key information; Determine the target anchor position according to the movement information, and determine the anchor coordinates of the target anchor position.

7. The display device according to claim 6, characterized in that The key information includes confirmation key information. The step of the controller performing reverse control of the first terminal device or the second terminal device according to the control information is further configured to: In response to an operation instruction input by the user based on the operation area, obtain the operation information of the operation instruction; the operation information at least includes anchor coordinate information; Determine the operation area of the operation instruction according to the anchor coordinate information; If the operation area is the first operation area and the key information is confirmation key information, obtain the anchor coordinates; Calculate the screen mirroring relative coordinates of the anchor relative to the preset corner point of the first display area according to the anchor coordinates and the first display area; Calculate the first terminal coordinates of the anchor relative to the first terminal device according to the screen mirroring relative coordinates and the screen mirroring zoom ratio; Generate the control information according to the first operation area and the confirmation key information; Send the control information and the first terminal coordinates to the first terminal device so that the first terminal device executes the control information; If the operation area is the second operation area and the key information is confirmation key information, obtain the anchor coordinates; Calculate the screen relative coordinates of the anchor relative to the preset corner point of the second display area according to the anchor coordinates and the second display area; Calculate the second terminal coordinates of the anchor relative to the second terminal device according to the screen relative coordinates and the display zoom ratio; Generate the control information according to the second operation area and the confirmation key information; Send the control information and the second terminal coordinates to the second terminal device so that the second terminal device executes the control information.

8. The display device according to claim 7, wherein The key information includes direction key information, and the controller is further configured to: In response to the key information being direction key information, obtain a sliding pixel displacement; Determine a first target anchor point coordinate after the anchor point slides the sliding pixel displacement on the first terminal device according to the first terminal coordinate and the sliding pixel displacement, and determine a second target anchor point coordinate after the anchor point slides the sliding pixel displacement on the second terminal device according to the second terminal coordinate and the sliding pixel displacement.

9. The display device according to claim 8, characterized in that, When the controller executes the step of performing reverse control on the first terminal device or the second terminal device according to the control information, it is further configured to: In response to an operation instruction input by the user based on the operation area, obtain operation information of the operation instruction; the operation information at least includes anchor point coordinate information; Determine the operation area of the operation instruction according to the anchor point coordinate information; If the operation area is the first operation area, send the first terminal coordinate, the sliding pixel displacement, and the first target anchor point coordinate to the first terminal device, so that the first terminal device slides from the first terminal coordinate as a starting point to the first target anchor point coordinate with the sliding pixel displacement as a sliding distance; If the operation area is the second operation area, send the second terminal coordinate, the sliding pixel displacement, and the second target anchor point coordinate to the second terminal device, so that the second terminal device slides from the second terminal coordinate as a starting point to the second target anchor point coordinate with the sliding pixel displacement as a sliding distance.

10. A control method for a dual-channel window, characterized in that, Applied to the display device according to any one of claims 1-9, the display device includes a display, a communicator, and a controller, and the control method for the dual-channel window includes: In response to a key instruction input by the user based on the floating window, obtain the anchor point coordinate and key information of the key instruction; the floating window includes a first display area for displaying the screen mirroring image; the screen mirroring display height of the screen mirroring image is equal to the screen height of the display; the non-screen mirroring area other than the first display area in the floating window is a transparent view; the media playback window includes a second display area for displaying the media playback image, and the second display area is calculated according to the screen size and the first display area; Determine the operation area of the key instruction according to the anchor point coordinate; Generate control information according to the operation area and the key information, and send the control information to the first terminal device or the second terminal device, and the control information is used to reversely control the first terminal device or the second terminal device.

Images