A display device and a method for switching between multiple projection windows.

Abstract

This application provides a display device and a method for switching multiple projection windows. The method responds to a projection connection command sent by a third terminal device and detects a projection replacement mode. If the projection replacement mode is a first mode, the focus position of the focus frame is detected, and a first replacement projection window at the focus position is removed. A target projection window is then displayed based on the orientation of the first replacement projection window. If the projection replacement mode is a second mode, the locking attribute of the projection window is detected, and a second replacement projection window is removed according to the locking attribute. A target projection window is then displayed based on the orientation of the second replacement projection window. The second replacement projection window is a projection window whose locking attribute is unlocked. This method can replace projection windows based on window locking attributes or focus positions according to different projection replacement modes, allowing the display device to quickly switch the target projection window to the user's desired position.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a method for switching display of multiple projection windows. Background Technology

[0002] Display devices are intelligent devices capable of presenting a user interface and supporting user interaction. Taking smart TVs as an example, smart TVs are television products based on Internet application technology, possessing an open operating system and chip, an open application platform, and enabling two-way human-computer interaction. They integrate multiple functions such as audio-visual entertainment and data to meet diverse and personalized user needs. Display devices can establish communication connections with terminal devices such as mobile phones and, based on specific screen mirroring protocols, obtain screen mirroring data from the terminal devices to display the mirrored screen content.

[0003] Some display devices can establish screen mirroring connections with multiple terminal devices simultaneously and receive screen mirroring data from multiple terminal devices, forming a multi-screen mirroring scenario. However, in a multi-screen mirroring scenario, the display device can only display the screen mirroring image of a single terminal device, and cannot display two or more screen mirroring images at the same time, which increases the limitations of multi-screen mirroring and reduces the user experience. Summary of the Invention

[0004] This application provides a display device and a method for switching between multiple projection windows to solve the problem of high limitations in multi-projection in display devices.

[0005] In a first aspect, this application provides a display device, including a display, a communicator, and a controller. The display is configured to display a multi-projection interface, the multi-projection interface including projection windows, a first projection window and a second projection window. The first projection window displays a first projection screen from a first terminal device; the second projection window displays a second projection screen from a second terminal device. The first projection window and the second projection window share the same root node as the multi-projection interface. The communicator is configured to establish communication connections with both the first and second terminal devices. The controller is configured to execute the following program steps:

[0006] In response to the screen mirroring connection command sent by the third terminal device, detect the screen mirroring replacement mode;

[0007] If the screen mirroring replacement mode is the first mode, the focus position of the focus box is detected, and the first replacement screen mirroring window at the focus position is removed. The target screen mirroring window is displayed according to the orientation mark of the first replacement screen mirroring window. The orientation mark is used to represent the display orientation of the window. The target screen mirroring window is used to display the screen mirroring screen of the third terminal device. The target screen mirroring window is the same as the root node of the multi-channel screen mirroring interface.

[0008] If the screen mirroring replacement mode is the second mode, the locking attribute of the screen mirroring window is detected, and the second replacement screen mirroring window is removed according to the locking attribute, and the target screen mirroring window is displayed according to the orientation mark of the second replacement screen mirroring window; the second replacement screen mirroring window is a screen mirroring window whose locking attribute is in an unlocked state.

[0009] Secondly, some embodiments of this application also provide a method for switching and displaying multiple projection windows, applied to the display device provided in the first aspect. The display device includes a display, a communicator, and a controller. The display is configured to display a multiple projection interface, which includes projection windows. The projection windows include a first projection window and a second projection window. The first projection window is used to display a first projection screen from a first terminal device; the second projection window is used to display a second projection screen from a second terminal device. The first projection window and the second projection window share the same root node as the multiple projection interface. The communicator is configured to establish communication connections with the first terminal device and the second terminal device, respectively. The method includes:

[0010] In response to the screen mirroring connection command sent by the third terminal device, detect the screen mirroring replacement mode;

[0011] If the screen mirroring replacement mode is the first mode, the focus position of the focus box is detected, and the first replacement screen mirroring window at the focus position is removed. The target screen mirroring window is displayed according to the orientation mark of the first replacement screen mirroring window. The orientation mark is used to represent the display orientation of the window. The target screen mirroring window is used to display the screen mirroring screen of the third terminal device. The target screen mirroring window is the same as the root node of the multi-channel screen mirroring interface.

[0012] If the screen mirroring replacement mode is the second mode, the locking attribute of the screen mirroring window is detected, and the second replacement screen mirroring window is removed according to the locking attribute, and the target screen mirroring window is displayed according to the orientation mark of the second replacement screen mirroring window; the second replacement screen mirroring window is a screen mirroring window whose locking attribute is in an unlocked state.

[0013] As can be seen from the above technical solutions, some embodiments of this application provide a display device and a method for switching multiple projection windows. The method can respond to a projection connection command sent by a third terminal device and detect the projection replacement mode. If the projection replacement mode is a first mode, the focus position of the focus frame is detected, and the first replacement projection window at the focus position is removed, and the target projection window is displayed according to the orientation marker of the first replacement projection window. If the projection replacement mode is a second mode, the locking attribute of the projection window is detected, and the second replacement projection window is removed according to the locking attribute, and the target projection window is displayed according to the orientation marker of the second replacement projection window. The second replacement projection window is a projection window whose locking attribute is unlocked. The method can replace projection windows according to different projection replacement modes by window locking attributes or focus positions, enabling the display device to quickly switch the target projection window to the user's desired position, thus improving the user experience. Attached Figure Description

[0014] Figure 1 This is a schematic diagram illustrating the usage scenarios of the display device in some embodiments of this application;

[0015] Figure 2 Hardware configuration block diagrams of display devices provided in some embodiments of this application;

[0016] Figure 3 Hardware configuration block diagrams of control devices provided in some embodiments of this application;

[0017] Figure 4 This is a schematic diagram of the software configuration in a display device provided in some embodiments of this application;

[0018] Figure 5 These are schematic diagrams illustrating screen mirroring provided for some embodiments of this application;

[0019] Figure 6 These are schematic diagrams illustrating push-to-screen projection provided in some embodiments of this application;

[0020] Figure 7 This is a schematic diagram of a multi-channel projection interface provided in some embodiments of this application;

[0021] Figure 8 This is a schematic diagram of the functional modes of a display device provided in some embodiments of this application;

[0022] Figure 9 A schematic diagram of a guide page provided for some embodiments of this application;

[0023] Figure 10 This application provides schematic diagrams illustrating the interface display during one-channel screen mirroring in some embodiments.

[0024] Figure 11This is a schematic diagram illustrating the effect of dynamically adjusting the display area provided in some embodiments of this application;

[0025] Figure 12 This application provides schematic diagrams of the interface display during two-way screen projection in some embodiments.

[0026] Figure 13 A flowchart illustrating a method for switching display between multiple projection windows provided in some embodiments of this application;

[0027] Figure 14 A flowchart illustrating the setting of replacement mode parameters provided in some embodiments of this application;

[0028] Figure 15 This is a schematic diagram illustrating the display effect of the locking control provided in some embodiments of this application;

[0029] Figure 16 This is a schematic diagram illustrating the display effect of the locking control in an unlocked state according to some embodiments of this application;

[0030] Figure 17 A schematic diagram illustrating the display effect of a target projection window replacing a second projection window, provided in some embodiments of this application;

[0031] Figure 18 This is a schematic diagram illustrating the interface display effect in the second mode provided in some embodiments of this application;

[0032] Figure 19 This is a schematic diagram illustrating the display effect of a target projection window replacing a first projection window, as provided in some embodiments of this application. Detailed Implementation

[0033] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0034] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0035] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0036] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0037] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0038] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. The control device 100 can be a remote control, stylus, etc.

[0039] In some embodiments, the mobile terminal 300 may install software applications with the display device 200 to establish a connection and communication via a network communication protocol, thereby achieving one-to-one control operation and data communication. Alternatively, audio and video content displayed on the mobile terminal 300 can be transmitted to the display device 200 to achieve synchronous display.

[0040] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0041] In addition to providing broadcast television reception functions, the display device 200 can also be equipped with intelligent network television functions that provide computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0042] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0043] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0044] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0045] In some embodiments, the display 260 includes a display screen component for presenting an image, a driving component for driving image display, a component for receiving image signals output from a controller, and a user control UI interface, etc.

[0046] In some embodiments, the communicator 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communicators 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communicator 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communicator 220 with Bluetooth functionality.

[0047] The communicator 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0048] In some embodiments, the display device 200 can also support establishing communication connections with multiple external devices simultaneously. These multiple external devices can be the same, for example, terminal devices 500 such as mobile phones or tablets. The first terminal device 510 and the second terminal device 520 can establish communication connections with the display device 200 by accessing the wireless local area network (WLAN) where the display device 200 is located. The multiple external devices can also connect to the display device 200 using different types of connection methods. For example, the first terminal device 510 connects to the display device 200 via a WLAN; the second terminal device 520 connects to the display device 200 via Bluetooth.

[0049] To establish a communication connection between the display device 200 and the terminal device 500, both devices are equipped with communicators supporting the same communication type. For example, if the display device 200 has a communicator 220 containing a WiFi module, the terminal device 500 should also have a communicator containing a WiFi module. Furthermore, in addition to having the same type of communicator, the display device 200 and the terminal device 500 also require a specific communication transmission protocol for data transfer. Examples include WiFi, Bluetooth, ZigBee, and NFC protocols.

[0050] In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, a ROM, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0051] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0052] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0053] In some embodiments, user interface 280 is an interface that can be used to receive control input.

[0054] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the central control device. (Example) Figure 3 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0055] The control device 100 is configured to control the display device 200, and to receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0056] In some embodiments, the control device 100 may be an intelligent device. For example, the control device 100 may be equipped with various applications for controlling the display device 200 according to user needs.

[0057] In some embodiments, such as Figure 1 As shown, the mobile terminal 300 or other smart electronic devices can perform similar functions to the control device 100 after installing the application of the control display device 200.

[0058] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation of the control device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0059] Under the control of the controller 110, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC module 133.

[0060] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, and button 144.

[0061] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via the WiFi, Bluetooth, or NFC protocol and send them to the display device 200.

[0062] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can also store various control signal instructions input by the user.

[0063] The power supply 180 is used to provide operating power support for the various components of the control device 100 under the control of the controller.

[0064] like Figure 4 In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer.

[0065] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0066] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0067] like Figure 4 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0068] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0069] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0070] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0071] After establishing a communication connection with the terminal device 500, the aforementioned display device 200 can establish a projection connection channel and transmit projection data based on the projection connection, so that the display device 200 can display the projection screen of the terminal device 500. In some embodiments, in order to establish a projection connection, the user can send a projection connection request through the terminal device 500 and send the projection connection request to the display device 200 via a WiFi network. The display device 200 then completes the transmission protocol configuration according to the projection connection request, thereby establishing a transmission channel for projection data with the terminal device 500.

[0072] The display device 200 can also establish a projection connection with the terminal device 500 through different connection methods. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, a projection connection can be established based on the WiFi network. As another example, when both the display device 200 and the terminal device 500 are equipped with NFC (Near Field Communication) components, a projection connection can be established through the NFC components. Obviously, the display device 200 and the terminal device 500 can also use other wired or wireless connection methods to establish a projection connection, such as RF radio frequency connection, infrared connection, cellular network, and other connection methods that those skilled in the art can conceive of based on the connection methods provided in the above embodiments.

[0073] After establishing a screen mirroring connection, the display device 200 can receive screen mirroring-related data from the terminal device 500 through the screen mirroring data channel, thereby playing the screen mirroring-related content on the display device 200. Under different screen mirroring data transmission protocols, the form of screen mirroring data obtained by the display device 200 varies. For example, the display device 200 and the terminal device 500 can establish a screen mirroring connection through protocols such as the Digital Living Network Alliance (DLNA) protocol, Miracast protocol, AirPlay push protocol, WiDi screen mirroring, NFC mirroring, and LeBao screen mirroring.

[0074] DLNA is a protocol used to enable media resource sharing between devices. This protocol utilizes network technology (wired or wireless) to interconnect various devices, exchanging data through standard protocols to allow media resources to be shared between them. Based on the UPnP protocol, DLNA enables devices to discover and control each other, achieving seamless transmission and sharing of media resources. DLNA supports various devices, including televisions, audio equipment, mobile phones, and computers, allowing users to seamlessly switch and share media content across different devices.

[0075] The Miracast protocol is a wireless display standard developed by the Wi-Fi Alliance, based on Wi-Fi Direct. The Miracast screen mirroring protocol is compatible with Android devices, meaning Android devices can share video feeds via Miracast. Android smart devices running Android 5.0 and above have the Miracast protocol built-in, allowing for direct wireless screen mirroring between the mirroring device and the receiving device's Wi-Fi network card without installing any software. Furthermore, Miracast screen mirroring supports simultaneous audio and video transmission.

[0076] AirPlay is a wireless playback technology developed by Apple, suitable for wireless screen mirroring on iOS and Mac systems. Apple devices have built-in AirPlay mirroring functionality. When used with a screen mirroring device that supports AirPlay server service, the effect is clear and stable, and the protocol includes a built-in screen mirroring code. However, the Apple device and the screen mirroring device must be on the same local area network (same network segment) and cannot be used across network segments or VLANs.

[0077] WiDi screen mirroring is a wireless playback technology spearheaded by Intel, based on WiFi Direct technology. Windows 8 / 10 / 11 laptops natively support WiDi screen mirroring, allowing direct screen mirroring without installing any software, and also support USB reverse control functionality.

[0078] In some embodiments, after the display device 200 establishes a screen projection connection with the terminal device 500, screen projection data can be transmitted through mirroring mode and push mode. Mirroring mode refers to the display device 200 synchronously displaying the screen content displayed by the terminal device 500. For example, ... Figure 5 As shown, display device 200 displays the main interface of terminal device 500, and when the user performs interactive operations on terminal device 500, causing changes in the content of the main interface, the content displayed on display device 200 also changes accordingly. Therefore, terminal device 500 can generate a video projection data stream based on the displayed content and send the projection data stream to display device 200 for display.

[0079] Push mode refers to the display device 200 accessing a network link pushed by the terminal device 500 to obtain the corresponding projected screen. For example, ... Figure 6 As shown, when the terminal device 500 is playing video media, a screen mirroring control is provided in the upper right corner of the playback interface. After the user clicks the screen mirroring control and selects the display device 200 as the device to display the screen mirrored image, the terminal device 500 can send the URL address of the currently playing video media to the display device 200. After receiving the URL address, the display device 200 accesses the URL address to obtain the video media data and plays the video media.

[0080] Among the screen mirroring protocols supported by display device 200 and terminal device 500, some protocols are based on mirroring mode, while others are based on push mode. For example, under the DLNA protocol, display device 200 can obtain the link address of a resource in terminal device 500 through the screen mirroring data channel, and then obtain media asset content data by accessing that link address, thereby playing the media asset data. Simultaneously, display device 200 can also obtain control commands sent by terminal device 500 through the screen mirroring data channel while playing media asset data, such as fast forward, pause, and stop screen mirroring. For example, after establishing a connection through the communication protocol, terminal device 500 (such as a mobile phone) and display device 200 will negotiate device functions and network conditions to select a suitable audio-visual transmission format. After establishing the audio-visual streaming connection protocol (Session), a series of Real-Time Streaming Protocol (RTSP) control commands are then used to control the playback and termination of the audio-visual streaming.

[0081] Under other screen mirroring protocols such as Miracast, AirPlay, NFC mirroring, and LeBao screen mirroring, display device 200 can directly obtain video data streams from terminal device 500. This video data stream can be the video data stream corresponding to the final display screen of the terminal device 500's display unit; that is, the video data stream includes media content, UI interface, and other video content that can be displayed on the display unit of the terminal device 500. Alternatively, the video data stream can simply be the video data that can be obtained from the terminal device 500, such as currently playing media video data. In this case, terminal device 500 can act as a signal source for display device 200. Display device 200 can continuously obtain video data through the screen mirroring data channel and display it in real time on monitor 260, achieving a synchronized display effect with terminal device 500.

[0082] It should be noted that the above-mentioned screen mirroring protocol can not only obtain the video data stream from the terminal device 500, but also the audio data stream from the terminal device 500, so that the video data and audio data can be sent to the display device 200 simultaneously, enabling the display device 200 to play the video and audio synchronously.

[0083] Terminal device 500 can obtain different screen mirroring data based on different screen mirroring protocols. For example, for system applications or screen mirroring directly at the system level, terminal device 500 can establish a screen mirroring data channel using Miracast, AirPlay, or WiDi screen mirroring protocols. Since the Miracast and AirPlay screen mirroring protocols require system-level application permissions, they are only applicable to system applications, or for display devices 200 and terminal device 500 from the same manufacturer to establish a screen mirroring connection directly through built-in rules in the operating system.

[0084] The display device 200 can establish a projection connection with the terminal device 500 based on any of the projection methods provided in the above embodiments. In some embodiments, the display device 200 can also establish projection connections with multiple communication terminals through one or more of the above projection methods. For example, the display device 200 can simultaneously establish projection connections with the first terminal device 510 and the second terminal device 520, and divide the display screen into display areas for displaying the projection screen shared by the first terminal device 510 and the second terminal device 520 respectively.

[0085] To enable the display device 200 to display a shared projection screen from multiple terminal devices 500, in some embodiments, when the display device 200 establishes a projection connection with multiple terminal devices 500, the display device 200 can display multiple projection interfaces, which may include multiple projection windows. Each projection window displays the projection screen from a corresponding terminal device 500 within a single projection connection channel. For example, ... Figure 7 As shown, the multi-channel projection interface can include two projection windows: a first projection window and a second projection window. The first projection window displays the projection screen of the first terminal device 510; the second projection window displays the projection screen of the second terminal device 520.

[0086] It should be noted that the casting window in the multi-channel casting interface does not necessarily have a strong binding relationship with the terminal device 500. That is, in some embodiments, the content displayed in the casting window can be determined according to the order in which the terminal device 500 and the display device 200 establish casting connections. For example, when the first terminal device 510 establishes a casting connection with the display device 200 before the second terminal device 520, the first casting window on the left side of the multi-channel casting interface displays the casting screen of the first terminal device 510. Conversely, when the first terminal device 510 establishes a casting connection with the display device 200 after the second terminal device 520, the first casting window on the left side of the multi-channel casting interface displays the casting screen of the second terminal device 520.

[0087] To achieve better image display and improve the coordination of multi-screen projection interfaces, in some embodiments, the display device 200 can also configure the layout of the projection windows. For this purpose, such as... Figure 8 As shown, the display device 200 can integrate multiple functional modules within the same application, such as a multi-channel projection application. The multi-channel projection application includes: a data management module, a message processing module, a service management module, an algorithm processing module, a view management module, an application interaction processing module, a location management module, a substitution management module, a broadcast control management module, a time management module, a focus management module, and a projection management module, etc.

[0088] The data management module primarily manages all message data involved in the current template and synchronizes and updates the data information. The message processing module mainly handles the transmission of relevant messages between different modules, such as transmitting data changes to the View layer for refresh display, or synchronizing View changes to the data management module for broadcast control data processing. The service management module is responsible for starting and stopping services throughout the application and handling customized processes, such as starting specific services in specific scenarios. The algorithm processing module is responsible for all layout and position calculations throughout the application. The View management module is primarily responsible for all UI displays of the entire template application, receiving data and refreshing the View display.

[0089] The application interaction processing module is responsible for handling interactions with external applications. For example, any entry point to launch the current template application needs to interact with this module, and different customized management is performed based on the different entry points. The position management module is responsible for the rules of displaying the projection position in the left or right side windows, such as whether it's on the left or right. The substitution management module is responsible for the substitution rules when the number of projection channels exceeds the maximum display limit, such as how to substitute when there is focus or not. The playback control management module is responsible for the playback control processing of different media asset types in the projection, such as fast forward, rewind, play / pause, etc. The time management module is responsible for the focus display timing processing in the projection, whether it's the large frame focus or the focus at the playback control position. The focus management module is responsible for the focus display and button processing throughout the application. The projection management module is responsible for projection protocol interaction, client list management, and handling the status information of the window displayed content.

[0090] It should be noted that the multi-channel projection interface can be based on a multi-channel projection viewing application to display content. This application, capable of displaying multi-protocol, multi-channel mirroring and push content, is a higher-level application within the display device 200. The framework layer of the display device 200 provides Java interfaces for hardware interfaces. The multi-channel projection viewing application uses these Java interfaces to interface with lower layers and obtain information. Other layers such as Nativie, Runtime, HAL, and the Linux kernel are standard Android architecture layers, and these layers interact according to business processes.

[0091] The display device 200 first starts the application through the service management module and performs some initialization preparations, such as starting the listening image or pushing the corresponding protocol service; the view management module initializes the view displayed in the application layer; the data management module creates some data for the data processing module; the algorithm processing module performs algorithm layout calculations on the projected content; and the position management module is responsible for locating the left and right positions of the projected content.

[0092] The service management module can launch the screen mirroring application and register a Session during the screen mirroring connection establishment process; it can also listen for Session Connection Listener, Session Prepare Listener, Session Display Listener, and Session Event Listener. A Session represents a single conversation between the client and the server and can store information from the client. The Session Connection Listener listens for Session connection and disconnection events. The Session Prepare Listener interacts with the client during screen mirroring preparation. The Session Display Listener monitors playback information, status changes, and client-side status changes. The Session Event Listener handles client-side control information.

[0093] After the View is initialized, the display device 200 will display a default onboarding window when entering the application. This onboarding window displays a setup page, which shows device information such as the device name and corresponding network information of the current display device 200. The setup page can include multiple standard system TextViews and ImageViews. The display device 200 can control the TextViews and ImageViews and their corresponding layouts to display the setup page in a symmetrical horizontal layout. Figure 9 As shown. The guide page includes a reference resolution; when the display device 200 displays the guide page through the guide window, it needs to scale the guide page according to the reference resolution.

[0094] In some embodiments, the guide window and the multi-screen projection interface share a root node, meaning they share the same root node. For example, after a user runs the multi-screen projection application on the display device 200, the display device 200 controls the monitor 260 to display the multi-screen projection interface in full screen and displays the guide window in the corresponding position on the multi-screen projection interface. The multi-screen projection interface is the main interface of the multi-screen projection application, and the guide window uses the same root node as the main interface of the multi-screen projection application.

[0095] After the guidance window is displayed, the display device 200 can listen for and receive the screen mirroring connection command sent by the terminal device 500, and establish a screen mirroring connection channel according to the screen mirroring connection command, thereby displaying it in the screen mirroring window. For example, the screen mirroring management module is always ready for the access of a new session. When Client1 is ready to start the connection, the screen mirroring management module listens for the onConnect connection of the new session through the UseConnectionList stener, and then creates the screen mirroring window, that is, the playback control SurfaceView with TextureView.

[0096] To facilitate the management of the display orientation of the projection window, in some embodiments, the display device 200 sets an orientation identifier for the projection window through a location management module when creating the projection window. The orientation identifier represents the display orientation of the window, and the display device 200 can display the projection window according to the orientation identifier. When setting the orientation identifier, the display device 200 can set the orientation identifier of the projection window according to the default priority order and the projection order of the terminal device 500. The orientation identifier can be different numbers or English characters.

[0097] For example, the display orientation of the projection window includes the left and right sides, with the default priority order being left > right. The orientation identifier of the left side is 1, and the orientation identifier of the right side is 2. After receiving the projection connection instruction from the first terminal device 510, the location management module sets the orientation identifier of the first projection window to 1 according to the default priority order, that is, sets mViewPosition:1.

[0098] To facilitate the management of View and Session, in some embodiments, a management object MultiCastPlayView can be set up. The management object MultiCastPlayView can be associated with information such as Session, SurfaceView / TextureView, and mViewPosition. For example, for the first terminal device 510, the data corresponding to its Client1 can be represented as: Session:SessionClient1, View:SurfaceView / TextureView, MViewPosition:1.

[0099] Once the display device 200, acting as the server, has prepared the View, it can notify the terminal device 500, acting as the mobile device, to record or send the video stream via the UseSessionPrepareListener. When the terminal device 500 is ready to send video information, the display device 200 learns through the onFirstFrame() method of the UseSessionDisplayListener that it needs to cast the video. Therefore, it can obtain the current video width and height of Client1 from the Session, determine the current video stream direction based on the width and height, and then calculate the position and size to be displayed using an algorithm.

[0100] In some embodiments, to distinguish the display range of the projection area, a certain blank area can be left between the first projection window and the second projection window, the projection window and the guide area. For example, the blank area is 12 pixels, and the resolution of the display device 200 is 1920×1080. Figure 10 As shown, when the first terminal device 510 (Client1) is displayed in landscape mode, the size of the projected screen in the corresponding video stream is 1920×1080. Then the size of the guide page is 954×538. At this time, the ratio of the video stream and the ratio of the guide video stream on the right are close to 16:9 and both are landscape. Therefore, it can be calculated by the algorithm that the left and right sides can be displayed in equal proportions, that is, they can be displayed with equal width and height. That is, the width of the video stream and the guide page is 954 and the height is approximately 538.

[0101] Therefore, the display device 200 can set the parameters of the first projection window, namely Client 1: width1 (width), height1 (height); Guide: width_guide, height_guide. The height of the video stream from the top edge of the screen is set to top, because height1 and height_guide will eventually be the same height, so the same variable height can be used instead. The video stream ratio ratio1 and the guide page ratio ratio_guide are both known constants. Then, width1, top, height, and width_guide can be calculated by the following combination of calculations: width1 + width_guide = 1920; top * 2 + height = 1080; width1 / height = ratio1; width_guide / height = ratio_guide.

[0102] Based on the aforementioned algorithm, the algorithm processing module of display device 200 can derive information such as the width, height, and position of the content displayed on the left and right sides. Specifically, the calculated width and height of Client1 are 848 and 400, respectively, with corresponding coordinates (0, 116) from the top left corner, and the coordinates of the guiding page being (412, 116).

[0103] After the above position calculation is completed, the data management module of the display device 200 can sort and store the data, and then transmit the data, including: Session:SessionClient1, View:SurfaceView / TextureView, MViewPosition:1, Position:(0, 116), Width:400, Height:848; Session:View:layout, MViewPosition:Position:(412, 116), Width:1508, Height:848. After receiving the data information, the View management module of the display device 200 sets the display parameters through LayoutParams, including the position (i.e., coordinates) from the top and left boundaries, as well as the corresponding display width and height, and then displays the position through the setLayoutParams method. That is:

[0104] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0);lp1.width=400;lp1.height=848;lp1.leftMargin=0;lp1.topMargin=116;mPlayerView.setLayoutPa rams(lp1);

[0105] FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0);lp1.width=1508;lp1.height=848;lp1.leftMargin=412;lp1.topMargin=116;mPlayerView.setLayoutPa rams(lp1).

[0106] By setting the display position of the projected SurfaceView / TextureView and the display position of the guided layout respectively, you can obtain the following: Figure 11 The display effect of the multi-channel projection interface is shown.

[0107] That is, the first and second projection windows can also present a shape that adapts to the size of the projected image. For example, when both the first and second projection images are landscape images of 1920×1080, the display device 200 can present a first and second projection window with a size of 960×540 in the multi-projection interface.

[0108] Display device 200 displays, for example Figure 11Following the multi-screen projection interface shown, in response to the projection connection command sent by the second terminal device 520, a second projection window of the second terminal device 520 is displayed in the multi-screen projection interface. That is, in some embodiments, when the display device 200 hears the projection connection command from the second terminal device Client2, it synchronously creates a second projection window View corresponding to the second terminal device 520. After the View is successfully created, the position management module is activated to set and manage the display position of SessionClient2, and the position is processed and obtained through the position management module. For example, since the first projection window is already displayed on the left side of the display device 200, the orientation of the second projection window is marked as the orientation on the right, that is, the mViewPosition of Client2 is marked as 2. The View management data of the managed object includes: Session: SessionClient2; View: SurfactView / TextureView; MViewPosition: 2.

[0109] Because the terminal device 500 can be placed in different states depending on its usage, such as smartphones, it may be in portrait or landscape mode. Portrait mode refers to a display where the width of the screen is less than its height, while landscape mode refers to a display where the width is greater than its height. In these different placement states, the terminal device 500 can present different screen layouts.

[0110] Based on this, in some embodiments, when the display device 200 displays the projection screens of multiple terminal devices 500 through a region division method, each display area will present a different screen ratio depending on the placement state of the terminal devices 500. For example, if the resolution of the display device 200 is 1920×1080, the first terminal device 510 is in portrait mode with a first projection screen resolution of 512×1088, and the second terminal device 520 is in landscape mode with a second projection screen resolution of 512×1088, then the first projection window can be 473×1006, and the second projection window can be 1435×1006.

[0111] Therefore, the display device 200 can set the parameters of the first and second projection windows, namely Client1 (first terminal device): width1 (width), height1 (height); Client2 (second terminal device): width2 (width), height2 (height). The height of the video stream from the top edge of the screen is set to top, because height1 and height2 will eventually be the same height, so the same variable height can be used instead. The video stream ratio ratio1 of the first terminal device 510 and the video stream ratio ratio2 of the second terminal device 520 are both known constants. Then, width1, top, height, and width2 can be calculated by the following combination of calculations: width1 + width2 = 1920; top * 2 + height = 1080; width1 / height = ratio1; width2 / height = ratio2.

[0112] Based on the above algorithm, the algorithm processing module of the display device 200 can obtain information such as the width, height, and position of the content displayed on the left and right sides. That is, the width and height of Client1 can be calculated to be 473 and 1006, respectively, with the corresponding position coordinates (0, 37) from the top left corner; the width and height of Client2 are 1435 and 1006, respectively, with the corresponding position coordinates (485, 37) from the top left corner.

[0113] After the above position calculation is completed, the data management module of the display device 200 can sort and store the data, and then transmit the data, including: Session: SessionClient1; View: SurfaceView / TextureView; MViewPosition:1; Position:(0, 37); Width:473; Height:1006; Session: SessionClient2; View: SurfaceView / TextureView; MViewPosition:2; Position:(485, 37); Width:1435; Height:1006. After receiving the data information, the View management module of the display device 200 sets the display parameters through LayoutParams, including the position (i.e., coordinates) from the top and left boundaries, as well as the corresponding display width and height, and then displays the position through the setLayoutParams method. That is:

[0114] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.width=473; lp1.height=1006; lp1.leftMargin=0; lp1.topMargin=37; mPlayerView.set LayoutParams(lp1);

[0115] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.width=1435; lp1.height=1006; lp1.leftMargin=485; lp1.topMargin=37; mPlayerView.setLayoutParams(lp1).

[0116] By setting the display position of the SurfaceView / TextureView for screen mirroring respectively, you can obtain results such as... Figure 12 The display effect of the multi-channel projection interface is shown.

[0117] In order to make full use of the display space of the multi-screen projection interface and reduce large black screen areas, in some embodiments, the display device 200 can set the width or height of the first projection window and the second projection window to be the same when the sizes of the first projection screen and the second projection screen are different, and scale the projection window according to the width or height.

[0118] Some embodiments of this application provide a display device 200, including a display 260, a communicator 220, and a controller 250. The display 260 is configured to display a multi-channel projection interface, which includes projection windows. The projection windows include a first projection window and a second projection window. The first projection window displays a first projection image from a first terminal device; the second projection window displays a second projection image from a second terminal device. The first and second projection windows share the same root node as the multi-channel projection interface. The communicator 220 is configured to establish communication connections with both a first terminal device 510 and a second terminal device 520. Figure 13 As shown, the controller is configured to perform the following program steps:

[0119] S100: In response to the screen mirroring connection command sent by the third terminal device, detect the screen mirroring replacement mode.

[0120] Due to screen resolution limitations, the number of projection windows that the display device 200 can display simultaneously is limited. In some embodiments, the display device 200 can display a maximum of a first projection window and a second projection window simultaneously. The total number of the first and second projection windows is the target number. Therefore, if the display device 200 has already displayed the first and second projection windows, and then receives a projection connection command from the third terminal device 530, requiring the display device 200 to replace either the already displayed first or second projection window, the current projection replacement mode of the display device 200 is detected.

[0121] For example, if the target number is 2, when the display device 200 receives the screen mirroring connection command sent by the terminal device 500, it detects the number of terminal devices currently connected to the display device 200. If the number of terminal devices currently connected to the display device 200 is greater than 2, it detects the screen mirroring replacement mode to replace the first or second screen mirroring window; if the number of terminal devices currently connected to the display device 200 is greater than 2, it sets the orientation identifier of the corresponding screen mirroring window of the terminal device according to the default priority order to display the screen mirroring window through the orientation identifier.

[0122] For ease of description, the target quantity is described as 2 in this embodiment, that is, the first projection window represents one projection window, the second projection window represents another projection window; the first terminal device 510 represents one terminal device, and the second terminal device 520 represents another terminal device.

[0123] Therefore, when the first and second projection windows are already displayed in the multi-projection interface of the display device 200, it indicates that the display device 200 has reached its maximum number of projection windows. At this time, if the display device 200 receives a projection connection command from the third terminal device 530, it will detect the current projection replacement mode and execute different interactive programs according to the detected projection replacement mode.

[0124] like Figure 14As shown, the screen mirroring replacement mode of the display device 200 can be set based on the locking attribute of the screen mirroring window. In some embodiments, the display device 200 also obtains the locking attribute of the screen mirroring window. The locking attribute includes a locked state and an unlocked state, with different parameters corresponding to the locked and unlocked states. For example, the locked state of the locking attribute is "lock," and the unlocked state is "unlock." If all the locking attributes of the screen mirroring window are in the unlocked state, the replacement mode parameter is set to the first parameter; if the locking attributes of the screen mirroring window include a locked state, the replacement mode parameter is set to the second parameter. The first parameter indicates that the screen mirroring replacement mode is a first mode, and the second parameter indicates that the screen mirroring replacement mode is a second mode. The first mode queries the screen mirroring window to be replaced based on the focus position of the focus box, while the second mode queries the screen mirroring window to be replaced based on the state of the locking attribute.

[0125] In some embodiments, when displaying a multi-channel projection interface, the display device 200 can also display playback control function controls within the multi-channel projection interface. Users can use the playback control function to perform interactive operations, input control commands, and control the display device 200 to execute interactive responses. For example, since the projection content of the display device 200 and the terminal device 500 is diverse, including mirrored video streams, pushed URL addresses, and different media asset types such as images and music, the playback control function controls are also different for different media asset types. For example, mirrored video projection does not support fast forward / rewind or play / pause operations; pushed URL addresses can perform fast forward, rewind, play, and pause operations; music can perform fast forward, rewind, play, and pause operations; and images do not support fast forward, rewind, play, and pause operations.

[0126] Since the locking attribute is a user-customizable attribute, the broadcast control function control can also include a locking control. Therefore, in some embodiments, when the display device 200 displays the projection window, it also controls the display 260 to display a locking control in the projection window. The locking control is associated with the projection window. The locking control can float above the projection window; or, as... Figure 15 As shown, the lock control can also be displayed in the top or bottom edge areas of the projection window in the multi-screen projection interface.

[0127] In some embodiments, when a user selects a lock control, the display device 200 automatically generates a selection command for the lock control and refreshes the current value of the lock control and the lock attribute of the projection window in response to the selection command. After the display device 200 refreshes the current value of the lock control, the displayed content of the lock control also changes. For example, the display device 200 displays as follows: Figure 15When the user selects the lock control shown on the right, the display device 200 refreshes the lock attribute of the projection window to unlock and updates the current value of the lock control, making the lock control visible. Figure 16 The content shown is as follows.

[0128] S200: If the screen casting replacement mode is the first mode, detect the focus position of the focus box, remove the first replacement screen casting window at the focus position, and display the target screen casting window according to the orientation mark of the first replacement screen casting window.

[0129] When the display device 200 detects that the screen mirroring replacement mode is in the first mode, it means that the locking attributes of both the first and second screen mirroring windows are unlocked. At this time, the display device 200 detects the focus position of the focus frame and removes the first replacement screen mirroring window at the focus frame's position. For example, if the focus position of the focus frame is on the first screen mirroring window, the first screen mirroring window is removed; if the focus position of the focus frame is on the second screen mirroring window, the second screen mirroring window is removed.

[0130] While removing the first replacement projection window, the display device 200 also creates a target projection window. This target projection window displays the projection screen from the third terminal device 530, and its root node is the same as the multi-channel projection interface. The target projection window is then displayed according to the orientation of the first replacement projection window.

[0131] Therefore, in some embodiments, if the screen casting replacement mode is the first mode, when the display device 200 displays the target screen casting window according to the orientation identifier of the first replacement window, it records the first orientation identifier of the first replacement window and creates the target screen casting window. Then, it sets the orientation identifier of the target screen casting window to the first orientation identifier and draws the target screen casting window according to the first orientation identifier.

[0132] When the screen mirroring management module detects the onConnect connection of the Session corresponding to the third terminal device Client3 through UseConnectionListener, it calls the Session.finish() interface to disconnect the Session corresponding to the screen mirroring window selected by the focus box. For example, when the screen mirroring window selected by the focus box is the second screen mirroring window, the screen mirroring management module notifies the View management module to remove the View of Client2 by rootview.removeView(). At the same time, the View management module also stores the information of Client3 and creates the SurfaceView / TextureView of the target screen mirroring window. After the SurfaceView / TextureView is successfully created, the display device 200 then sets the orientation identifier of Client3 to the orientation identifier of Client2 through the starting position management module, thereby replacing the position of the second screen mirroring window. Therefore, for the third terminal device, the data corresponding to its Client3 can be represented as: Session:SessionClient3; View:SurfaceView / TextureView; MViewPosition:2.

[0133] For example, when display device 200 establishes a screen mirroring connection with first terminal device 510 and second terminal device 520, the display shows, for example... Figure 16 The multi-channel projection interface is shown. The left side is the first projection window, and the right side is the second projection window. Both the first and second projection windows are unlocked. At this time, a projection connection command is sent from the third terminal device 530 to the display device 200, and the display device 200 detects the focus position of the focus frame according to the first mode. For example... Figure 16 The focus is on the second projection window, meaning the second projection window is the first replacement projection window. The display device 200 then removes the second projection window and sets the orientation marker of the target projection window to the orientation marker of the second projection window, causing the target projection window to replace the second projection window, resulting in the following display. Figure 17 The multi-channel projection interface shown.

[0134] It is understandable that when the display device 200 replaces the first replacement projection window with the target projection window, it can calculate the display area of ​​the target projection window based on the algorithm provided in the above embodiments, which will not be elaborated here.

[0135] Obviously, to facilitate user modification of the lock attribute of the target projection window, the display device 200 also generates a lock control associated with the target projection window after displaying it. Therefore, in some embodiments, the display device 200 also controls the display 260 to display the target lock control, and in response to a selection command input by the user based on the target lock control, refreshes the current value of the lock control and the lock attribute of the target projection window. The target lock control is associated with the target projection window.

[0136] S300: If the screen casting replacement mode is the second mode, detect the locking attribute of the screen casting window, remove the second replacement screen casting window according to the locking attribute, and display the target screen casting window according to the orientation mark of the second replacement screen casting window.

[0137] When the display device 200 detects that the screen mirroring replacement mode is the first mode, it indicates that the locking attribute of the first screen mirroring window and / or the second screen mirroring window is locked. At this time, the display device 200 detects the locking attribute of each screen mirroring window and determines the second replacement screen mirroring window based on the locking attribute of each window. The second replacement screen mirroring window is a screen mirroring window whose locking attribute is unlocked.

[0138] Since the locking attribute is user-customizable, both the first and second projection windows may be locked. In this case, the display device 200 cannot replace any already displayed projection window with the target projection window. Therefore, in some embodiments, if the projection replacement mode is the second mode, the display device 200 iterates through the locking attributes of the projection windows. If the locking attribute of the projection window does not include an unlocked state, the control communicator 220 sends a projection failure notification to the third terminal device, so that the third terminal device can notify the user of the projection failure information.

[0139] For example, display device 200 displays such as Figure 18 The multi-channel screen mirroring interface is shown. The left side is the first screen mirroring window, and the right side is the second screen mirroring window. Both the first and second screen mirroring windows are locked. At this time, after the screen mirroring management module detects the onConnection event of the session corresponding to the third terminal device Client3 through UseConnectionListener, it will not disconnect the first terminal device Client1 or the second terminal device Client2. The screen mirroring management module will not send any notification to the View management module, and the display device 200 will not create a view for Client3. Instead, it will notify Client3 that screen mirroring is no longer possible in the current mode. After receiving this message, Client3 will display a screen mirroring failure message to the user on the mobile device.

[0140] If the locking attribute of the projection window includes an unlocked state, in some embodiments, when the display device 200 displays the target projection window according to the orientation identifier of the second replacement window, it records the orientation identifier of the second replacement projection window. The second replacement projection window is the projection window with the locking attribute in an unlocked state. A target projection window is created, and its orientation identifier is set to the second orientation identifier. The target projection window is then drawn according to the second orientation identifier.

[0141] When the screen mirroring management module detects the onConnect connection of the Session corresponding to the third terminal device Client3 through UseConnectionListener, it uses index detection to find the unlocked screen mirroring window and calls the Session.finish() interface to disconnect the Session corresponding to the unlocked screen mirroring window. For example, if the unlocked screen mirroring window is the first screen mirroring window, the screen mirroring management module notifies the View management module to remove the View of Client1 using rootview.removeView(). Simultaneously, the View management module stores the information of Client3 and creates the SurfaceView / TextureView for the target screen mirroring window. After the SurfaceView / TextureView is successfully created, the display device 200 then sets the orientation identifier of Client3 to the orientation identifier of Client1 through the starting position management module, thereby replacing the position of the first screen mirroring window. Therefore, for the third terminal device, the data corresponding to its Client3 can be represented as: Session:SessionClient3; View:Surface View / TextureView; MViewPosition:1.

[0142] For example, when display device 200 establishes a screen mirroring connection with first terminal device 510 and second terminal device 520, the display shows, for example... Figure 15 The multi-screen projection interface is shown. The left side is the first projection window, and the right side is the second projection window. The first projection window is unlocked, while the second projection window is locked. At this time, a projection connection command is sent to the display device 200 via the third terminal device 530. The display device 200 checks the locking attribute of the projection windows according to the second mode. The first projection window, which is unlocked, becomes the second replacement projection window. The display device 200 then removes the first projection window and sets the location identifier of the target projection window to the location identifier of the first projection window, thus replacing the first projection window with the target projection window, resulting in the following interface. Figure 19 The multi-channel projection interface shown.

[0143] It is understood that the embodiments of this application are described with a target number as the standard. In some application scenarios, if the target number set by the display device 200 is greater than 2, the locking attribute of the projection window is locked, and the number of projection windows in the unlocked state is greater than 1, the display device 200 can mark the projection window in the unlocked state with the highest priority as the second replacement window according to the preset priority order, so as to replace the second replacement window with the target projection window.

[0144] Furthermore, when display device 200 is displaying the first and second projection windows, if the user disconnects the projection connection of one of the terminal devices, the projection window of the remaining terminal device may move. For example, when the projection connection between the first terminal device 510 and display device 200 is disconnected, display device 200 will remove the first projection window and display a guide window and the second projection window in the multi-projection interface. Since display device 200 has a default priority order, when recalculating the display area of ​​each window, display device 200 may set the orientation marker of the second projection window to a higher priority orientation marker, causing the second projection window to suddenly switch to the position of the first projection window. Thus, when display device 200 displays multiple projection windows simultaneously, if the projection connection of one of the terminal devices 500 is disconnected, the position of the projection window will switch back and forth, and the user's gaze will also need to follow the position of the projection window, affecting the user's viewing experience.

[0145] To improve the issue of switching back and forth between projection windows, in some embodiments, after displaying the target projection window, the display device 200 also deletes the target projection session in response to a termination command. The target projection session is a communication session generated based on a third-party terminal device. The third-party identifier of the target projection window is recorded, the target projection window is removed, and a guide window is invoked. This guide window displays a guide page, and its root node is the same as the root node of the multi-channel projection interface. The display 260 is then controlled to display the guide window according to the third-party identifier.

[0146] For example, the display device has already connected to two clients: a first terminal device Client1 and a third terminal device Client3. Figure 17The multi-projection interface shown depicts Client1's first projection window on the left and Client3's target projection window on the right. At this point, Client3's projection connection with the display device is established. The projection management service listens for Client3's disconnection request via UseConnectionListener and notifies the data management module to remove Client3's session. After the data management module successfully removes Client3's information, it synchronously notifies the View management module, causing the View management module to execute rootview.removeView() to remove the corresponding View. This preserves Client1's data information, including: Session: SessionClient1; View: SurfaceView / TextureView; MViewPosition: 1. Furthermore, the guide window on the right is re-invoked according to Client3's location, ensuring the multi-projection interface presents as shown. Figure 10 The display effect shown.

[0147] In some embodiments, the guide window and the multi-screen projection interface share the same root, meaning the root node of the guide window and the multi-screen projection interface is the same. When the display device 200 invokes the guide window, it can either set the guide window to a displayed state or re-invoke the relevant resources of the window object corresponding to the guide window.

[0148] To ensure the standardization of the layout of each window, the display device 200 calculates the display area of ​​the guide window based on the resolution of the guide page when displaying the guide window. Specifically, in some embodiments, the display device 200 detects the screen resolution, the reference resolution, and the projection screen resolution. The reference resolution is the resolution of the guide page, and the projection screen resolution is the resolution of the projection screen in the multi-channel projection interface, i.e., the projection screen corresponding to the terminal device 500 currently connected to the display device 200. Then, the layout parameters of the guide window are calculated based on the screen resolution, the reference resolution, and the projection screen resolution. These layout parameters include the layout width and the layout height.

[0149] After calculating the layout parameters of the guide window, the display device 200 can display the guide window according to the layout parameters. Therefore, in some embodiments, when the display device 200 displays the guide window according to a third-party bit identifier, it generates the layout position coordinates of the guide window based on the layout parameters and the third-party bit identifier, and records the layout parameters and layout position coordinates. Then, it draws the guide window according to the layout parameters and layout position coordinates.

[0150] For example, display device 200 displays such as Figure 17The multi-projection interface shown has Client1's first projection window on the left and Client3's target projection window on the right. At this point, the projection connection between Client3 and the display device is disconnected, the display device 200 deletes Client3's session, and removes the corresponding View. Based on the algorithm provided in the above embodiment, the width and height of each window, as well as the coordinates of the top-left corner, are calculated according to the display device 200's screen resolution, the resolution of the first projection screen, and the baseline resolution of the guide window. This data is then stored in the data management module. This data is then sent to the View management module, causing it to use the setLayoutParams method to display the projection SurfaceView / TextureView and the guide Layout, thus obtaining the desired display position. Figure 10 The display effect of the multi-channel projection interface is shown.

[0151] Similarly, in some embodiments, when the multi-channel projection interface of the display device 200 includes only one projection window, that is, when the display device 200 only establishes a projection connection with one terminal device 500, the projection connection between the terminal device 500 and the display device 200 is disconnected. The display device 200 then deletes the projection session of the terminal device and removes the projection window to display the default guide window, such as... Figure 9 As shown.

[0152] For example, the screen resolution is 1920*1080, and the base resolution of the guide page is 954*538. When displaying the default guide window, the display device 200 calculates the layout parameters and layout position coordinates based on the algorithm provided in the above embodiment. The data transmitted by the data transmission module includes: Session; View:layout; Position:(0, 271); Width:954; Height:538; Session; View; Position:(966, 271); Width:954; Height:538.

[0153] The View management module uses the `setLayoutParams` method to display the position of the guide layout, thus obtaining the desired position. Figure 9 The effect shown is as shown.

[0154] Based on the aforementioned display device 200, some embodiments of this application also provide a method for switching and displaying multiple projection windows, applied to the display device 200 provided in the above embodiments. The display device 200 includes a display 260, a communicator 220, and a controller 250. The display 260 is configured to display a multiple projection interface, which includes projection windows. Each projection window includes a first projection window and a second projection window. The first projection window displays a first projection image from a first terminal device; the second projection window displays a second projection image from a second terminal device. The first and second projection windows share the same root node as the multiple projection interface. The communicator 220 is configured to establish communication connections with the first terminal device 510 and the second terminal device 520, respectively. Figure 13 As shown, the method includes the following procedural steps:

[0155] S100: In response to the screen mirroring connection command sent by the third terminal device, detect the screen mirroring replacement mode;

[0156] S200: If the screen casting replacement mode is the first mode, detect the focus position of the focus box, remove the first replacement screen casting window at the focus position, and display the target screen casting window according to the orientation mark of the first replacement screen casting window. The orientation mark is used to characterize the display orientation of the window. The target screen casting window is used to display the screen casting screen of the third terminal device. The target screen casting window is the same as the root node of the multi-channel screen casting interface.

[0157] S300: If the screen casting replacement mode is the second mode, detect the locking attribute of the screen casting window, remove the second replacement screen casting window according to the locking attribute, and display the target screen casting window according to the orientation mark of the second replacement screen casting window; the second replacement screen casting window is a screen casting window whose locking attribute is in an unlocked state.

[0158] As can be seen from the above technical solutions, some embodiments of this application provide a display device and a method for switching multiple projection windows. The method can respond to a projection connection command sent by a third terminal device and detect the projection replacement mode. If the projection replacement mode is a first mode, the focus position of the focus frame is detected, and the first replacement projection window at the focus position is removed, and the target projection window is displayed according to the orientation marker of the first replacement projection window. If the projection replacement mode is a second mode, the locking attribute of the projection window is detected, and the second replacement projection window is removed according to the locking attribute, and the target projection window is displayed according to the orientation marker of the second replacement projection window. The second replacement projection window is a projection window whose locking attribute is unlocked. The method can replace projection windows according to different projection replacement modes by window locking attributes or focus positions, enabling the display device to quickly switch the target projection window to the user's desired position, thus improving the user experience.

[0159] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0160] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0162] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized by comprising: include: The display is configured to present the user interface; A communicator is configured to establish a communication connection with an external device or server. The controller is configured as follows: After establishing screen mirroring connections with the first terminal device and the second terminal device respectively, a multi-screen mirroring interface is displayed. The multi-screen mirroring interface includes a screen mirroring window, which includes a first screen mirroring window and a second screen mirroring window. The first screen mirroring window is used to display the first screen mirroring screen of the first terminal device. The second projection window is used to display the second projection screen of the second terminal device; the first projection window, the second projection window and the root node of the multi-channel projection interface are the same; In response to the screen mirroring connection command sent by the third terminal device, detect the screen mirroring replacement mode; If the screen mirroring replacement mode is the first mode, the focus position of the focus box is detected, and the first replacement screen mirroring window at the focus position is removed. The target screen mirroring window is displayed according to the orientation mark of the first replacement screen mirroring window. The orientation mark is used to represent the display orientation of the window. The target screen mirroring window is used to display the target screen mirroring screen of the third terminal device. The target screen mirroring window is the same as the root node of the multi-channel screen mirroring interface. If the screen casting replacement mode is the second mode, detect the locking attribute of the screen casting window, remove the second replacement screen casting window according to the locking attribute, and display the target screen casting window according to the orientation mark of the second replacement screen casting window; The second replacement projection window is the projection window whose locking attribute is in an unlocked state.

2. The display device of claim 1, wherein, The controller is also configured to: Obtain the lock attribute of the screen projection window; If all the lock attributes of the screen projection window are in an unlocked state, set the replacement mode parameter to the first parameter, which is used to indicate that the screen projection replacement mode is the first mode; If the screen mirroring window's lock attribute includes a locked state, the replacement mode parameter is set to a second parameter, which is used to characterize the screen mirroring replacement mode as a second mode.

3. The display device of claim 1, wherein, If the screen mirroring replacement mode is the first mode, the controller is configured to display the target screen mirroring window according to the orientation identifier of the first replacement screen mirroring window. Record the first orientation identifier of the first replacement projection window; Create the target projection window; Set the orientation identifier of the target projection window to the first orientation identifier; The target projection window is drawn according to the first orientation marker.

4. The display device according to claim 1, characterized in that, If the screen mirroring replacement mode is the second mode, the controller is configured as follows: Iterate through the locking attributes of the projection window; If the locking attribute of the screen mirroring window does not include an unlocked state, the communicator is controlled to send a screen mirroring failure notification to the third terminal device.

5. The display device according to claim 4, characterized in that, If the locking attribute of the projection window includes an unlocked state, the controller executes the function of displaying the target projection window according to the orientation identifier of the second replacement projection window, as configured to: Record the second orientation identifier of the second replacement projection window; Create the target projection window; Set the orientation identifier of the target projection window to the second orientation identifier; The target projection window is drawn according to the second orientation marker.

6. The display device according to claim 1, characterized in that, The controller is also configured to: The display is controlled to show a target locking control, which is associated with the target projection window; In response to a selection command input by the user based on the target locking control, refresh the current value of the locking control and the locking attribute of the target projection window.

7. The display device according to claim 1, characterized in that, The controller is also configured to: In response to the termination command of the target screen projection session, the target screen projection session is deleted. The target screen projection session is a communication session generated based on the third terminal device. Record the third-party identifier of the target projection window; Remove the target projection window and invoke the guide window, which is used to display the guide page. The root node of the guide window is the same as the root node of the multi-channel projection interface. Control the display to show the guide window according to the third-party bit identifier.

8. The display device according to claim 7, characterized in that, The controller is also configured to: The screen resolution, the baseline resolution, and the projection screen resolution are detected. The baseline resolution is the resolution of the guide page, and the projection screen resolution is the resolution of the projection screen in the multi-channel projection interface. The layout parameters of the guide window are calculated based on the screen resolution, the reference resolution, and the projection screen resolution. The layout parameters include the layout width and the layout height.

9. The display device according to claim 8, characterized in that, The controller is configured to control the display to show the guide window according to the third bit identifier. The layout position coordinates of the guide window are generated based on the layout parameters and the third-party bit identifier; Record the layout parameters and the layout position coordinates; The guide window is drawn according to the layout parameters and the layout position coordinates.

10. A method for switching and displaying multiple projection windows, characterized in that, Applied to a display device, the display device including a display, a communicator and a controller, the display being configured to present a user interface; The communicator is configured to establish a communication connection with an external device or server; the method includes: After establishing screen mirroring connections with the first terminal device and the second terminal device respectively, a multi-screen mirroring interface is displayed. The multi-screen mirroring interface includes a screen mirroring window, which includes a first screen mirroring window and a second screen mirroring window. The first screen mirroring window is used to display the first screen mirroring screen of the first terminal device; the second screen mirroring window is used to display the second screen mirroring screen of the second terminal device; the first screen mirroring window and the second screen mirroring window have the same root node as the multi-screen mirroring interface. In response to the screen mirroring connection command sent by the third terminal device, detect the screen mirroring replacement mode; If the screen mirroring replacement mode is the first mode, the focus position of the focus box is detected, and the first replacement screen mirroring window at the focus position is removed. The target screen mirroring window is displayed according to the orientation mark of the first replacement screen mirroring window. The orientation mark is used to represent the display orientation of the window. The target screen mirroring window is used to display the screen mirroring screen of the third terminal device. The target screen mirroring window is the same as the root node of the multi-channel screen mirroring interface. If the screen mirroring replacement mode is the second mode, the locking attribute of the screen mirroring window is detected, and the second replacement screen mirroring window is removed according to the locking attribute, and the target screen mirroring window is displayed according to the orientation mark of the second replacement screen mirroring window; the second replacement screen mirroring window is a screen mirroring window whose locking attribute is in an unlocked state.

Images