Display equipment and switching display method of multi-path screen projection window

Abstract

The embodiment of the invention provides a display device and a switching display method for multiple projection screen windows, and the method can detect a projection screen replacement mode in response to a projection screen connection instruction sent by a third terminal device. And if the screen projection replacement mode is a first mode, detecting a focus position of the focus frame, removing a first replacement screen projection window at the focus position, and displaying a target screen projection window according to a direction identifier of the first replacement screen projection window. And if the screen projection replacement mode is a second mode, detecting a locking attribute of the screen projection window, removing a second replacement screen projection window according to the locking attribute, and displaying the target screen projection window according to the orientation identifier of the second replacement screen projection window. Wherein the second replacement screen projection window is a screen projection window of which the locking attribute is an unlocked state. According to the method, the projection screen window can be replaced through the window locking attribute or the focus position according to different projection screen replacement modes, so that the display equipment can quickly switch the target projection screen window to the position expected by a user.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a method for switching display of multiple projection windows. Background Art

[0002] Display devices are intelligent devices that can present user interfaces and support user interaction. Taking smart TVs as an example, smart TVs are based on Internet application technology, have open operating systems and chips, have open application platforms, can realize two-way human-computer interaction functions, and are TV products that integrate multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users. Display devices can establish communication connections with terminal devices such as mobile phones, and based on specific screen projection protocols, obtain the screen projection data of terminal devices, so as to display the screen projection images of terminal devices through display devices.

[0003] Some display devices can establish screen projection connections with multiple terminal devices at the same time and receive projection data from multiple terminal devices, forming a multi-channel projection scene. However, in a multi-channel projection scene, the display device can only display the projection screen of a single terminal device, and cannot display two or more projection screens at the same time, which increases the limitations of multi-channel projection and reduces the user experience. Summary of the invention

[0004] The present application provides a display device and a method for switching display of multiple screen projection windows to solve the problem of high limitations of multiple screen projection in display devices.

[0005] In a first aspect, the present application provides a display device, including a display, a communicator, and a controller. The display is configured to display a multi-way screen projection interface, the multi-way screen projection interface includes a screen projection window, the screen projection window includes a first screen projection window and a second screen projection window, the first screen projection window is used to display a first screen projection screen of a first terminal device; the second screen projection window is used to display a second screen projection screen of a second terminal device; the first screen projection window and the second screen projection window are the same as the root node of the multi-way screen projection interface; the communicator is configured to establish a communication connection with the first terminal device and the second terminal device respectively; the controller is configured to execute the following program steps:

[0006] In response to a screen projection connection instruction sent by a third terminal device, detecting a screen projection replacement mode;

[0007] If the screen projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement screen projection window at the focus position is removed, and the target screen projection window is displayed according to the orientation mark of the first replacement screen projection window, the orientation mark is used to characterize the display orientation of the window, and the target screen projection window is used to display the projection screen of the third terminal device, and the target screen projection window is the same as the root node of the multi-channel screen projection interface;

[0008] If the projection replacement mode is the second mode, detect the locking attribute of the projection window, remove the second replacement projection window according to the locking attribute, and display the target projection window according to the orientation mark of the second replacement projection window; the second replacement projection window is a projection window whose locking attribute is in an unlocked state.

[0009] In a second aspect, some embodiments of the present application further provide a method for switching and displaying multiple screen projection windows, which is applied to the display device provided in the first aspect, the display device comprising a display, a communicator and a controller, the display being configured to display a multiple screen projection interface, the multiple screen projection interface comprising projection windows, the projection windows comprising a first projection window and a second projection window, the first projection window being used to display a first projection screen of a first terminal device; the second projection window being used to display a second projection screen of a second terminal device; the first projection window and the second projection window being the same as the root node of the multiple screen projection interface; the communicator being configured to establish communication connections with the first terminal device and the second terminal device, respectively; the method comprising:

[0010] In response to a screen projection connection instruction sent by a third terminal device, detecting a screen projection replacement mode;

[0011] If the screen projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement screen projection window at the focus position is removed, and the target screen projection window is displayed according to the orientation mark of the first replacement screen projection window, the orientation mark is used to characterize the display orientation of the window, and the target screen projection window is used to display the projection screen of the third terminal device, and the target screen projection window is the same as the root node of the multi-channel screen projection interface;

[0012] If the projection replacement mode is the second mode, detect the locking attribute of the projection window, remove the second replacement projection window according to the locking attribute, and display the target projection window according to the orientation mark of the second replacement projection window; the second replacement projection window is a projection window whose locking attribute is in an unlocked state.

[0013] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a method for switching display of multiple projection windows, and the method can detect the projection replacement mode in response to the projection connection instruction sent by the third terminal device. If the projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement projection window at the focus position is removed, and the target projection window is displayed according to the orientation mark of the first replacement projection window. If the projection replacement mode is the second mode, the locking attribute of the projection window is detected, and the second replacement projection window is removed according to the locking attribute, and the target projection window is displayed according to the orientation mark of the second replacement projection window. Among them, the second replacement projection window is a projection window whose locking attribute is in an unlocked state. The method can replace the projection window by the window locking attribute or the focus position according to different projection replacement modes, so that the display device can quickly switch the target projection window to the position expected by the user, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram showing usage scenarios of a display device in some embodiments of the present application;

[0015] Figure 2 A hardware configuration block diagram of a display device provided in some embodiments of the present application;

[0016] Figure 3 A hardware configuration block diagram of a control device provided in some embodiments of the present application;

[0017] Figure 4 A schematic diagram of software configuration in a display device provided in some embodiments of the present application;

[0018] Figure 5 A schematic diagram of mirror screen projection provided in some embodiments of the present application;

[0019] Figure 6 A schematic diagram of push screen projection provided in some embodiments of the present application;

[0020] Figure 7 A schematic diagram of a multi-screen projection interface provided in some embodiments of the present application;

[0021] Figure 8 A schematic diagram of a functional mode of a display device provided in some embodiments of the present application;

[0022] Fig. 9 A schematic diagram of a guide page provided for some embodiments of the present application;

[0023] Fig.10 A schematic diagram of the interface display when casting a screen in some embodiments of the present application;

[0024] Fig.11A schematic diagram of the effect of dynamically adjusting the display area provided in some embodiments of the present application;

[0025] Fig.12 A schematic diagram of the interface display when two-way screen projection is provided in some embodiments of the present application;

[0026] Fig.13 A schematic diagram of a flow chart of a method for switching display of multiple projection windows provided in some embodiments of the present application;

[0027] Fig.14 A schematic diagram of a process for setting replacement mode parameters provided in some embodiments of the present application;

[0028] Fig.15 A schematic diagram of the display effect of a locking control provided in some embodiments of the present application;

[0029] Fig.16 A schematic diagram of the display effect of a locked control in an unlocked state provided in some embodiments of the present application;

[0030] Fig.17 A schematic diagram of the display effect of replacing the second projection window with the target projection window provided in some embodiments of the present application;

[0031] Fig.18 A schematic diagram of the interface display effect in the second mode provided in some embodiments of the present application;

[0032] Fig.19 Schematic diagram of the display effect of replacing the first projection window with the target projection window provided in some embodiments of the present application. DETAILED DESCRIPTION

[0033] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0034] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0035] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0036] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0037] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0038] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 may be a remote controller, a stylus pen, or the like.

[0039] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0040] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0041] In addition to providing the broadcast receiving television function, the display device 200 may also provide an intelligent network television function with computer support functions, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0042] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0043] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0044] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0045] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from a controller, display video content, image content, and a menu control interface component and a user control UI interface, etc.

[0046] In some embodiments, the communicator 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communicators 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communicator 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communicator 220 including a Bluetooth function.

[0047] The communicator 220 can connect the display device 200 to the external device or the server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 to the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 to the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0048] In some embodiments, the display device 200 may also support establishing a communication connection relationship with multiple external devices at the same time. Multiple external devices may be connected through the same. For example, the external device is a terminal device 500 such as a mobile phone or a tablet computer. The first terminal device 510 and the second terminal device 520 may establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices may also be connected to the display device 200 through different types of connection methods. For example, the first terminal device 510 is connected to the display device 200 via a wireless local area network; the second terminal device 520 is connected to the display device 200 via Bluetooth.

[0049] In order to realize the communication connection between the display device 200 and the terminal device 500, the display device 200 and the terminal device 500 are respectively provided with communicators supporting the same communication type. For example, when the display device 200 is provided with a communicator 220 including a WiFi module, the terminal device 500 should also be provided with a communicator including a WiFi module. In addition, when the same type of communicators are provided, a specific communication transmission protocol is also required between the display device 200 and the terminal device 500 for data transmission. For example, WiFi protocol, Bluetooth connection protocol, ZigBee protocol, NFC protocol, etc.

[0050] In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0051] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0052] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0053] In some embodiments, the user interface 280 is an interface that can be used to receive control input.

[0054] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0055] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0056] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0057] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0058] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0059] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0060] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0061] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, which can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0062] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0063] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0064] like Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0065] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0066] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0067] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0068] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0069] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0070] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0071] After establishing a communication connection relationship with the terminal device 500, the display device 200 can establish a screen projection connection channel, and transmit the projection data based on the screen projection connection, so that the display device 200 can display the projection screen of the terminal device 500. In some embodiments, in order to establish a screen projection connection relationship, the user can send a screen projection connection request through the terminal device 500, and send the screen projection connection request to the display device 200 through the WiFi network. The display device 200 then completes the transmission protocol configuration according to the screen projection connection request, thereby establishing a transmission channel for the projection data with the terminal device 500.

[0072] The display device 200 can also achieve a screen projection connection with the terminal device 500 through different connection methods. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, a screen projection connection can be established based on a WiFi network. For another example, when both the display device 200 and the terminal device 500 are provided with NFC (Near Field Communication) components, a screen projection connection relationship can be established through the NFC component. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the terminal device 500 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and other connection methods that can be associated with those skilled in the art based on the connection methods provided in the above embodiments.

[0073] After the screen projection connection is established, the display device 200 can receive the screen projection related data of the terminal device 500 through the screen projection data channel, so as to play the screen projection related content in the display device 200. Under different screen projection data transmission protocols, the screen projection data obtained by the display device 200 is in different forms. For example, the display device 200 and the terminal device 500 can establish a screen projection connection through the Digital Living Network Alliance (DLNA) protocol, Miracast protocol, AirPlay push protocol, WiDi screen projection, NFC mirror screen projection, LeTV screen projection and other protocols.

[0074] Among them, the DLNA protocol is a protocol for sharing media resources between devices. The protocol uses network technology (wired or wireless) to interconnect various devices and exchange data through standard protocols, so that media resources can be shared between devices. The DLNA protocol is based on the UPnP protocol, which enables devices to discover and control each other and achieve seamless transmission and sharing of media resources. The DLNA protocol supports a variety of devices, including TVs, speakers, mobile phones, computers, etc., allowing users to seamlessly switch and share media content between different devices.

[0075] Miracast protocol is a wireless display standard based on Wi-Fi direct connection and developed by the Wi-Fi Alliance. Miracast screen projection protocol is applicable to Android devices, that is, Android devices can share video images through Miracast screen projection. Smart devices with Android 5.0 or above have built-in Miracast protocol, so you can directly wirelessly project the screen through the WiFi P2P function between the projection device and the WiFi network card of the receiving device without installing any software. In addition, Miracast screen projection also supports synchronous transmission of audio and video.

[0076] Airplay protocol is a wireless playback technology launched by Apple, which is applicable to the wireless screen projection protocol of iOS and Mac systems. Apple devices have built-in Airplay screen projection function. When used with a screen projection device that supports AirPlay server service, the effect is clear and stable. The protocol comes with a screen projection code function. However, it is necessary to ensure that the Apple device and the screen projection device are used in the same LAN (same network segment), and cannot be used across network segments or VLANs.

[0077] WiDi screen projection is a wireless playback technology led by Intel, and is also based on WiFi Direct technology. Windows 8 / 10 / 11 laptops natively support WiDi screen projection, which can be directly projected without installing any software, and supports USB reverse control function.

[0078] In some embodiments, after the display device 200 establishes a screen projection connection with the terminal device 500, the screen projection data can be transmitted through the mirror mode and the push mode. Among them, the mirror mode means that the display device 200 synchronously displays the screen content displayed by the terminal device 500. For example, Figure 5 As shown, the display device 200 displays the main interface of the terminal device 500, and when the user performs an interactive operation on the terminal device 500, resulting in a change in the content of the main interface, the content displayed by the display device 200 also changes. To this end, the terminal device 500 can generate a screen projection data stream in the form of a video based on the displayed content, and send the screen projection data stream to the display device 200 for display.

[0079] The push mode means that the display device 200 accesses the network link according to the network link pushed by the terminal device 500 to obtain the corresponding projection screen. Figure 6 As shown, when the terminal device 500 plays the video media, a screen projection control is provided in the upper right corner of the play interface. After the user clicks the screen projection control and selects the display device 200 as the device for displaying the screen projection image, the terminal device 500 can send the URL address of the currently played video media to the display device 200. After receiving the URL address, the display device 200 accesses the URL address to obtain the video media data and plays the video media.

[0080] Among the screen projection protocols supported by the display device 200 and the terminal device 500, some protocols are based on the mirror mode, and some protocols are based on the push mode. For example, under the DLNA protocol, the display device 200 can obtain the link address of a resource in the terminal device 500 through the screen projection data channel, and obtain the media content data by accessing the link address, thereby playing the media data. At the same time, the display device 200 can also obtain control instructions sent by the terminal device 500 through the screen projection data channel when playing media data, such as fast forward, pause, stop screen projection, etc. For example, after establishing a connection through the communication protocol, the terminal device 500 such as a mobile phone and the display device 200 will negotiate the device functions and network conditions to select a suitable audio and video transmission format. After establishing the online protocol (Session) of the audio and video streaming, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the audio and video streaming.

[0081] Under other screen projection connection protocols such as Miracast, Airplay, NFC mirroring, and LeTV, the display device 200 can directly obtain the video data stream from the terminal device 500. The video data stream can be the video data stream corresponding to the final display screen of the display unit of the terminal device 500, that is, the video data stream includes the media content screen, UI interface, and other video screen content that can be displayed in the display unit of the terminal device 500. The video data stream can also be only the video data that can be obtained in the terminal device 500, such as the media video data being played. At this time, the terminal device 500 can be used as a signal source for the display device 200, and the display device 200 can continuously obtain video data through the screen projection data channel, and display it in real time on the display 260, so as to achieve the effect of synchronous display with the terminal device 500.

[0082] It should be noted that the above-mentioned screen projection protocol can not only obtain the video data stream in the terminal device 500, but also obtain the audio data stream in the terminal device 500, so that the video data and audio data can be sent to the display device 200 synchronously, so that the display device 200 can play the video and audio synchronously.

[0083] The terminal device 500 can obtain different projection data according to different projection protocols. For example, for system applications or direct projection at the system level, the terminal device 500 can use Miracast projection, Airplay projection or WiDi projection protocol to establish a projection data channel. Since the above-mentioned Miracast projection protocol and Airplay projection protocol require system-level application permissions, the Miracast projection protocol and Airplay projection protocol are only applicable to system applications, or the display device 200 and the terminal device 500 of the same manufacturer directly establish a projection connection through the built-in rules of the operating system.

[0084] The display device 200 can establish a screen projection connection with the terminal device 500 based on any of the screen projection methods provided in the above embodiments. In some embodiments, the display device 200 can also establish a screen projection connection relationship with multiple communication terminals through one or more of the above screen projection methods. For example, the display device 200 can simultaneously establish a screen projection connection with the first terminal device 510 and the second terminal device 520, and divide the display area in the display screen to display the projection screen shared by the first terminal device 510 and the second terminal device 520 respectively.

[0085] In order to display the projection screen shared by multiple terminal devices 500 on the display device 200, in some embodiments, when the display device 200 establishes a projection connection with multiple terminal devices 500, the display device 200 can display a multi-channel projection interface, which can include multiple projection windows. The projection window is used to display the projection screen of the corresponding terminal device 500 under a projection connection channel. For example, Figure 7 As shown, the multi-channel screen projection interface may include two screen projection windows, namely a first screen projection window and a second screen projection window. The first screen projection window is used to display the projection screen of the first terminal device 510; the second screen projection window is used to display the projection screen of the second terminal device 520.

[0086] It should be noted that the projection window in the multi-channel projection interface may not be strongly bound to the terminal device 500, that is, in some embodiments, the display content of the projection window may be determined according to the order in which the terminal device 500 establishes a projection connection with the display device 200. For example, when the first terminal device 510 establishes a projection connection with the display device 200 before the second terminal device 520, the first projection window on the left side of the multi-channel projection interface displays the projection screen of the first terminal device 510. When the first terminal device 510 establishes a projection connection with the display device 200 later than the second terminal device 520, the first projection window on the left side of the multi-channel projection interface displays the projection screen of the second terminal device 520.

[0087] In order to obtain a better picture display effect and improve the coordination of the multi-channel projection interface, in some embodiments, the display device 200 can also set the layout of the projection window. Figure 8 As shown, the display device 200 can have multiple functional modules built into the same application, such as a multi-channel screen projection application. The multi-channel screen projection application includes: a data management module, a message processing module, a service management module, an algorithm processing module, a View management module, an application interaction processing module, a location management module, a replacement management module, a broadcast control management module, a time management module, a focus management module, and a screen projection management module.

[0088] Among them, the data management module mainly manages all message data involved in the current template and updates the data information synchronously. The message processing module mainly transmits related messages between different modules, such as transmitting data changes to the View layer for refresh display, or synchronizing View changes to the data management module for change processing of broadcast control data, etc. The service management module is responsible for opening and closing the services of the entire application and some customized processing, such as starting specific services in specific scenarios. The algorithm processing module is responsible for all layout position calculations of the entire application. The View management module is mainly responsible for all UI displays of the entire template application, and receives data to refresh the View display.

[0089] The application interaction processing module is responsible for the processing of interactions with external applications. For example, any entry to start the current template application needs to interact with this module, and different customized management is performed according to the differences in the startup entries. The position management module is responsible for the rules for displaying the projection position in the left and right windows, such as on the left or on the right. The replacement management module is responsible for the replacement rules when the number of projection paths exceeds the maximum display number, such as how to replace when there is focus or no focus. The broadcast control management module is responsible for the broadcast control processing of different media types in the projection, such as fast forward, rewind, play pause, etc. The time management module is responsible for the focus display timing processing in the projection, whether it is the large frame focus or the broadcast control position focus. The focus management module is responsible for the focus display and button processing in the entire application. The projection management module is responsible for the projection protocol interaction and the client list management and window display content status information processing.

[0090] It should be noted that the multi-channel screen projection interface can realize screen display based on the multi-channel screen projection viewing application. The multi-channel screen projection viewing application is an application that can display multi-protocol multi-channel mirror projection and push content, and is an upper-level application inside the display device 200. The framework layer of the display device 200 can provide a Java interface for docking with hardware, etc. The multi-channel screen projection viewing application realizes docking with the lower layer and obtains information through the Java interface of the Framework layer. Other National, Runtime, HAL, and Linux kernel layers are all standard layers of the Android architecture, and each layer interacts according to the business process.

[0091] The display device 200 first starts the application through the service management module and performs some initial preparations, such as starting the monitoring mirror or pushing the corresponding protocol service; the View management module initializes the View displayed by the application layer; the data management module creates some data for the data processing module; the algorithm processing module performs algorithmic layout calculations on the projection content, and the position management module is responsible for positioning the left and right positions of the projection content.

[0092] The service management module can start the screen projection application and register the Session during the process of establishing the screen projection connection; Usession Connection Listener, Usession Prepare Listener, Usession DisplayListener and Usession Event Listener listen. Among them, Session represents a conversation between the client and the server, which can save the information of the Client. Uses ession Connection Listener is used to monitor the connection and disconnection events of the Session. Usession Prepare Listener interacts with the screen projection preparation of the Client. UsessionDisplay Listener monitors the changes in playback information and status and the status changes of the Client. Usession EventListener processes information such as Client control.

[0093] After the initialization of View is completed, the display device 200 will display the default guide window when entering the application. The guide window is used to display the guide page. The guide page will display some device information, such as the device name of the current display device 200 and the corresponding network information, that is, the guide page includes TextView and ImageView that can include multiple system standards. The display device 200 can display the guide page in equal proportions on the left and right by controlling the TextView and ImageView controls and the corresponding Layout, such as Fig. 9 The guide page includes a reference resolution, and when the display device 200 displays the guide page through the guide window, it is necessary to scale the guide page according to the reference resolution.

[0094] In some embodiments, the guide window and the multi-channel projection interface share a root, that is, the root node of the guide window and the multi-channel projection interface is the same. For example, after the user runs the multi-channel projection application in the display device 200, the display device 200 controls the display 260 to display the multi-channel projection interface in full screen, and displays the guide window in the corresponding position of the multi-channel projection interface. Among them, the multi-channel projection interface is the homepage interface of the multi-channel projection application, and the guide window and the homepage interface of the multi-channel projection application use the same root.

[0095] After the guide window is displayed, the display device 200 can monitor and receive the screen projection connection instruction sent by the terminal device 500, and establish a screen projection connection channel according to the screen projection connection instruction, so as to display it in the screen projection window. For example, the screen projection management module is always ready for the access of a new Session. When Client1 is ready to start the connection, the screen projection management module monitors the onConnect connection of the new Session through UsessessionConnectionListener, and then creates the screen projection window, that is, the playback control SurfaceVie w / TextureView.

[0096] In order to facilitate the management of the display orientation of the projection window, in some embodiments, when creating the projection window, the display device 200 also sets the orientation identifier of the projection window through the position management module. The orientation identifier is used to characterize the display orientation of the window, and the display device 200 can display the projection window according to the orientation identifier. When setting the orientation identifier, the display device 200 can set the orientation identifier of the projection window according to the projection order of the terminal device 500 according to the default priority order. The orientation identifier can be different numbers or English characters.

[0097] For example, the display orientation of the projection window includes the left side and the right side, and the default priority order is left side > right side. The orientation identifier of the left side is 1, and the orientation identifier of the right side is 2. After receiving the projection connection instruction from the first terminal device 510, the position management module sets the orientation identifier of the first projection window to 1 according to the default priority order, that is, sets mViewPosition:1.

[0098] To facilitate the management of View and Session, in some embodiments, a management object MultiCastPlayView can be set, wherein the management object MultiCastPlayView can be associated with information such as Session, SurfaceView / TextureView, and mViewPosition. For example, for the first terminal device 510, the data corresponding to its Client1 can be represented as: Session:SessionClient1, View:SurfaceView / TextureView, MViewPosition:1.

[0099] When the display device 200 prepares the View as the server, it can notify the terminal device 500 as the mobile terminal to record or send the video stream through UsesessionPrepareListener. When the terminal device 500 is ready to send video information, the display device 200 knows that it will be cast through onFirstFrame() of UsesessionDisplayListener, so it can get the current width and height of Client1's video from the Session, determine the direction of the current video stream through the width and height, and then calculate the current position and size to be displayed through the algorithm.

[0100] In some embodiments, in order to distinguish the display range of the projection area, a certain blank area may be left in the first projection window and the second projection window, the projection window and the guide area. For example, the blank area is 12 pixels, and the resolution of the display device 200 is 1920×1080. Fig.10 As shown, when the first terminal device 510 (Client 1) is displayed in horizontal format, the size of the projection screen in the corresponding projection video stream is 1920×1080, and the guide page display size is 954×538. At this time, the video stream ratio and the right guide video stream ratio are close to 16:9 and are both horizontal. It can be calculated through the algorithm that the left and right sides can be displayed in proportion, that is, they can be displayed with equal width and height, that is, the width of the video stream and the guide page is 954, and the height is about 538.

[0101] To this end, the display device 200 can set the parameters of the first projection window, namely Client 1: width1 (width), height1 (height); Guide (guide page): width_guide, height_guide. The height of the projection screen video stream from the upper edge of the screen is set to top, because height1 and height_guide are ultimately the same height, and can be replaced by the same variable height. The video stream ratio ratio1 and the guide page ratio ratio_guide are both known constants. Through the following combined calculations, width1, top, height, and width_guide can be calculated; that is, width1+width_guide=1920; top*2+height=1080; width1 / height=ratio1; width_guide / height=ratio_guide.

[0102] Based on the above algorithm, the algorithm processing module of the display device 200 can obtain the corresponding width and height position of the content displayed on the left and right sides. That is, the width and height of Client1 can be calculated to be 848 and 400, and the corresponding position coordinates are (0, 116) from the upper left corner, and the guide page coordinates are (412, 116).

[0103] After the above position calculation is completed, the data management module of the display device 200 can sort and store, and then transmit the data, including: Session: SessionClient1, View: SurfaceView / TextureView, MViewPosition: 1, Position: (0, 116), Width: 400, Height: 848; Session: View: layout, MViewPosition: Position: (412, 116), Width: 1508, Height: 848. After receiving the data information, the View management module of the display device 200 sets the display parameters through La youtParams, including the position from the upper boundary and the left boundary (i.e., coordinates), as well as the corresponding display width and height, and then displays the position through the setLayoutParams method. That is:

[0104] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0);lp1.width=400;lp1.height=848;lp1.leftMargin=0;lp1.topMargin=116;mPlayerView.setLayoutPa rams(lp1);

[0105] FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0);lp1.width=1508;lp1.height=848;lp1.leftMargin=412;lp1.topMargin=116;mPlayerView.setLayoutPa rams(lp1).

[0106] By setting the display position of the projection SurfaceView / TextureView and the display position of the guide Layout respectively, you can get the following Fig.11 The display effect of the multi-channel screen projection interface is shown.

[0107] That is, the first projection window and the second projection window can also present corresponding shapes according to the size of the displayed projection screen. For example, when the first projection screen and the second projection screen are both 1920×1080 horizontal screens, the display device 200 can present the first projection window and the second projection window with a size of 960×540 in the multi-channel projection interface.

[0108] The display device 200 displays the Fig.11After the multi-way screen projection interface shown, the second screen projection window of the second terminal device 520 is displayed in the multi-way screen projection interface in response to the screen projection connection instruction sent by the second terminal device 520. That is, in some embodiments, when the display device 200 listens to the screen projection connection instruction of the second terminal device Client2, the second screen projection window View corresponding to the second terminal device 520 is synchronously created. After the View is successfully created, the position management module is started to set and manage the display position of SessionClient2, and it is acquired through the position management module. For example, since the first screen projection window has been displayed on the left side of the display device 200, the orientation of the second screen projection window is identified as the orientation identification on the right, that is, the mViewPosition of Client2 is marked as 2. The View management data of the management object includes: Session:SessionClient2; View:Surfac eView / TextureView; MViewPosition:2.

[0109] Since the terminal device 500 may be placed in different states according to different usage modes, for example, a terminal device 500 such as a smart phone may be in a portrait state and a landscape state during use. The portrait state refers to a state in which the width of the displayed image is less than the height, and the landscape state refers to a state in which the width of the displayed image is greater than the height. In different placement states, the terminal device 500 may present different screen layouts.

[0110] Based on this, in some embodiments, when the display device 200 displays the projection screens of multiple terminal devices 500 by area division, each display area will present different screen ratios due to the different placement states of the terminal devices 500. For example, the resolution of the display device 200 is 1920×1080, the first terminal device 510 is in a vertical screen state, the resolution of the first projection screen is 512×1088, the second terminal device 520 is in a horizontal screen state, and the resolution of the second projection screen is 512×1088. Then the first projection window can be 473×1006, and the second projection window can be 1435×1006.

[0111] To this end, the display device 200 can set the parameters of the first projection window and the second projection window, namely, Client1 (first terminal device): width1 (width), height1 (height); Client2 (second terminal device): width2 (width), height2 (height). The height of the projection screen video stream from the upper edge of the screen is set to top, because height1 and height2 are ultimately the same height, and can be replaced by the same variable height. The video stream ratio ratio1 of the first terminal device 510 and the video stream ratio ratio2 of the second terminal device 520 are both known constants. Then, through the following combined calculations, width1, top, height, and width2 can be calculated; namely, width1+width2=1920; top*2+height=1080; width1 / height=ratio1; widt h2 / height=ratio2.

[0112] Based on the above algorithm, the algorithm processing module of the display device 200 can obtain the corresponding width, height, and position information of the displayed content on the left and right sides. That is, it can be calculated that the width and height of Client1 are 473 and 1006, and the corresponding position coordinates are (0, 37) away from the upper left corner; the width and height of Client2 are 1435 and 1006, and the corresponding position coordinates are (485, 37) away from the upper left corner.

[0113] After the above position calculation is completed, the data management module of the display device 200 can sort and store, and then transmit the data, including: Session: SessionClient1; View: SurfaceView / TextureView; MViewPosition: 1; Position: (0, 37); Width: 473; Height: 1006; Session: SessionClient2; View: SurfaceView / TextureView; MViewPosition: 2; Position: (485, 37); Width: 1435; Height: 1006. After receiving the data information, the View management module of the display device 200 sets the display parameters through LayoutParams, including the position (i.e., coordinates) from the upper and left boundaries, and the corresponding display width and height, and then displays the position through the setLayoutParams method. That is:

[0114] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.width=473; lp1.height=1006; lp1.leftMargin=0; lp1.topMargin=37; mPlayerView.set LayoutParams(lp1);

[0115] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.width=1435; lp1.height=1006; lp1.leftMargin=485; lp1.topMargin=37; mPlayerView.setLayoutParams(lp1).

[0116] By setting the display position of the projection SurfaceView / TextureView respectively, you can get the following Fig.12 The display effect of the multi-channel screen projection interface is shown.

[0117] In order to make full use of the display space of the multi-way projection interface and reduce large black screen areas, in some embodiments, when the sizes of the first projection screen and the second projection screen are inconsistent, the display device 200 can set the width or height of the first projection window and the second projection window to be consistent, and scale the projection windows according to the width or height.

[0118] Some embodiments of the present application provide a display device 200, including a display 260, a communicator 220 and a controller 250. The display 260 is configured to display a multi-way screen projection interface, the multi-way screen projection interface includes a screen projection window, the screen projection window includes a first screen projection window and a second screen projection window, the first screen projection window is used to display a first screen projection screen of a first terminal device; the second screen projection window is used to display a second screen projection screen of a second terminal device; the first screen projection window and the second screen projection window are the same as the root node of the multi-way screen projection interface; the communicator 220 is configured to establish a communication connection with the first terminal device 510 and the second terminal device 520 respectively; Fig.13 As shown, the controller is configured to perform the following program steps:

[0119] S100: In response to the screen projection connection instruction sent by the third terminal device, detect the screen projection replacement mode.

[0120] Due to the limitation of screen resolution, the number of projection windows that the display device 200 can display at the same time is limited. In some embodiments, the display device 200 can display at most the first projection window and the second projection window at the same time. Among them, the sum of the number of the first projection window and the second projection window is the target number. Therefore, after the display device 200 has displayed the first projection window and the second projection window, if it receives the projection connection instruction sent by the third terminal device 530, the display device 200 needs to replace the displayed first projection window or the second projection window, then the current projection replacement mode of the display device 200 is detected.

[0121] For example, when the target number is 2, the display device 200 detects the number of terminal devices currently connected to the display device 200 when receiving the projection connection instruction sent by the terminal device 500. If the number of terminal devices currently connected to the display device 200 is greater than 2, the projection replacement mode is detected to replace the first projection window or the second projection window through the projection replacement mode; if the number of terminal devices currently connected to the display device 200 is greater than 2, the orientation mark of the projection window corresponding to the terminal device is set according to the default priority order to display the projection window through the orientation mark.

[0122] For ease of description, the target number is 2 in the embodiment of the present application, that is, the first projection window represents one projection window, and the second projection window represents another projection window; the first terminal device 510 represents one terminal device, and the second terminal device 520 represents another terminal device.

[0123] Therefore, when the first projection window and the second projection window are displayed in the multi-channel projection interface of the display device 200, it means that the maximum number of projection windows that can be displayed by the current display device 200 has been reached. At this time, when the display device 200 receives the projection connection instruction sent by the third terminal device 530, it detects the current projection replacement mode to execute different interactive programs according to the detected projection replacement mode.

[0124] like Fig.14As shown, the screen projection replacement mode of the display device 200 can be set based on the locking attribute of the screen projection window, that is, in some embodiments, the display device 200 also obtains the locking attribute of the screen projection window. The locking attribute includes a locked state and an unlocked state, and the locked state and the unlocked state correspond to different parameters. For example, the locking state of the locking attribute is lock, and the unlocked state is unlock. If the locking attributes of the screen projection window are all unlocked, set the replacement mode parameter to the first parameter; if the locking attributes of the screen projection window include a locked state, set the replacement mode parameter to the second parameter. Among them, the first parameter is used to characterize the screen projection replacement mode as the first mode, and the second parameter is used to characterize the screen projection replacement mode as the second mode. The first mode queries the screen projection window that needs to be replaced according to the focus position of the focus frame, and the second mode queries the screen projection window that needs to be replaced according to the state of the locking attribute.

[0125] In some embodiments, when the display device 200 displays the multi-channel projection interface, it can also display the broadcast control function control in the multi-channel projection interface, and the user can perform interactive operations through the broadcast control function control to input control instructions to control the display device 200 to perform interactive responses. For example, since the projection contents of the display device 200 and the terminal device 500 are various, there are mirrored video streams, pushed URL addresses, and different media types such as pictures and music, the broadcast control function controls of different media types are also different. For example, the mirrored video projection does not support fast forward and rewind operations nor does it support play and pause operations; the pushed URL address can fast forward, rewind, play, pause, etc.; the music can fast forward, rewind, play, pause, etc.; the picture does not support fast forward, rewind, play, pause, etc.

[0126] Since the lock attribute is a user-defined attribute, the playback control function control may also include a lock control. Therefore, in some embodiments, when the display device 200 displays the projection window, it also controls the display 260 to display the lock control in the projection window. The lock control is associated with the projection window. The lock control may be displayed floating above the projection window; or, as shown in FIG. Fig.15 As shown, the lock control can also be displayed in the edge area such as above or below the projection window in the multi-way projection interface.

[0127] In some embodiments, when the user performs a selection operation on the lock control, the display device 200 automatically generates a selection instruction for the lock control, and refreshes the current value of the lock control and the lock attribute of the projection window in response to the selection instruction. After the display device 200 refreshes the current value of the lock control, the display content presented by the lock control will also change. For example, the display device 200 displays the following Fig.15When the user selects the locked control on the right, the display device 200 refreshes the lock attribute of the projection window to unlock and refreshes the current value of the locked control, so that the locked control appears. Fig.16 The display content shown.

[0128] S200: If the screen projection replacement mode is the first mode, detect the focus position of the focus frame, remove the first replacement screen projection window at the focus position, and display the target screen projection window according to the orientation mark of the first replacement screen projection window.

[0129] When the display device 200 detects that the projection replacement mode is the first mode, it means that the lock attributes of the first projection window and the second projection window are both unlocked. At this time, the display device 200 detects the focus position of the focus frame and removes the first replacement projection window at the focus frame position. For example, if the focus position of the focus frame is on the first projection window, remove the first projection window; if the focus position of the focus frame is on the second projection window, remove the second projection window.

[0130] The display device 200 removes the first replacement projection window and creates a target projection window. The target projection window is used to display the projection screen of the third terminal device 530, and the target projection window is the same as the root node of the multi-channel projection interface. The target projection window is then displayed according to the orientation mark of the first replacement projection window.

[0131] Therefore, in some embodiments, if the projection replacement mode is the first mode, when the display device 200 displays the target projection window according to the orientation identifier of the first replacement window, the first orientation identifier of the first replacement window is recorded, and the target projection window is created. Then, the orientation identifier of the target projection window is set to the first orientation identifier, and the target projection window is drawn according to the first orientation identifier.

[0132] That is, when the projection management module listens to the onConnect connection of the Session corresponding to the third terminal device Client3 through UsesessionConnectionListener, the Session.finish() interface is called to perform a disconnection process on the Session corresponding to the projection window selected by the focus frame. For example, when the projection window selected by the focus frame is the second projection window, the projection management module notifies the View management module to remove the View of Client2 by rootview.removeView(). At the same time, the View management module also stores the information of Client3 and creates the target projection window playback control SurfaceView / TextureView. After the SurfaceView / TextureView is successfully created, the display device 200 sets the orientation identifier of Client3 to the orientation identifier of Client2 by starting the position management module, thereby replacing the position of the second projection window. Therefore, for the third terminal device, the data corresponding to its Client3 can be represented as: Session:SessionClient3; View:SurfaceView / TextureView; MViewPosition:2.

[0133] For example, when the display device 200 establishes a screen projection connection with the first terminal device 510 and the second terminal device 520, the following is displayed: Fig.16 The multi-channel screen projection interface shown in the figure. The left side is the first screen projection window, and the right side is the second screen projection window. The lock attributes of the first screen projection window and the second screen projection window are both unlocked. At this time, the third terminal device 530 sends a screen projection connection instruction to the display device 200, and the display device 200 detects the focus position of the focus frame according to the first mode. Fig.16 , the focus position is located at the second projection window, that is, the second projection window is the first replacement projection window. The display device 200 removes the second projection window and sets the position mark of the target projection window to the position mark of the second projection window, so that the target projection window replaces the position of the second projection window, and the display is as follows Fig.17 The multi-screen projection interface shown.

[0134] It is understandable that when the display device 200 replaces the first replacement projection window with the target projection window, the display area of ​​the target projection window can be calculated based on the algorithm provided in the above embodiment, which will not be elaborated here.

[0135] Obviously, in order to facilitate the user to modify the lock attribute of the target projection window, the display device 200 will also generate a lock control associated with the target projection window after displaying the target projection window. Therefore, in some embodiments, the display device 200 also controls the display 260 to display the target lock control, and in response to the user's selection instruction based on the target lock control input, refreshes the current value of the lock control and the lock attribute of the target projection window. Among them, the target lock control is associated with the target projection window.

[0136] S300: If the projection replacement mode is the second mode, detect the locking property of the projection window, remove the second replacement projection window according to the locking property, and display the target projection window according to the orientation mark of the second replacement projection window.

[0137] When the display device 200 detects that the projection replacement mode is the first mode, it indicates that the locking attribute of the first projection window and / or the second projection window is in a locked state. At this time, the display device 200 detects the locking attribute of each projection window and determines the second replacement projection window according to the locking attribute of each window. Among them, the second replacement projection window is a projection window whose locking attribute is in an unlocked state.

[0138] Since the lock attribute is user-definable, the lock attributes of the first projection window and the second projection window may both be in a locked state. At this time, the display device 200 cannot replace any displayed projection window with the target projection window. Therefore, in some embodiments, if the projection replacement mode is the second mode, the display device 200 traverses the lock attributes of the projection window. If the lock attribute of the projection window does not include an unlocked state, the control communicator 220 sends a projection failure notification to the third terminal device, so that the third terminal device prompts the user with information about the projection failure.

[0139] For example, the display device 200 displays Fig.18 The multi-channel screen projection interface shown. Among them, the left side is the first screen projection window, and the right side is the second screen projection window. The lock properties of the first screen projection window and the second screen projection window are both in the locked state. At this time, after the screen projection management module monitors the onConnect connection of the corresponding Session of the third terminal device Client3 through UsessionConnectionListener, it will not perform disconnection processing on the first terminal device Client1 and the second terminal device Client2, and the screen projection management module will not send any notification to the View management module, and the display device 200 will not create a view for Client3, but notify Client3 that the current mode can no longer be projected. After receiving this message, Client3 will prompt the user on the mobile terminal with information that the screen projection has failed.

[0140] If the lock attribute of the projection window includes an unlocked state, in some embodiments, when the display device 200 displays the target projection window according to the orientation identifier of the second replacement window, the orientation identifier of the second replacement projection window is recorded, and the second replacement projection window is the projection window whose lock attribute is unlocked. Create a target projection window, and set the orientation identifier of the target projection window to the second orientation identifier. Then draw the target projection window according to the second orientation identifier.

[0141] That is, when the projection management module listens to the onConnect connection of the Session corresponding to the third terminal device Client3 through UsessionConnectionListener, the index detection finds the unlocked projection window, and calls the Session.finish() interface to perform the disconnection process on the Session corresponding to the unlocked projection window. For example, when the unlocked projection window is the first projection window, the projection management module notifies the View management module to remove the View of Client1 by rootview.removeView(). At the same time, the View management module also stores the information of Client3 and creates the target projection window playback control SurfaceView / TextureView. After the SurfaceView / TextureView is successfully created, the display device 200 sets the orientation identifier of Client3 to the orientation identifier of Client1 by starting the position management module, thereby replacing the position of the first projection window. Therefore, for the third terminal device, the data corresponding to its Client3 can be represented as: Session:SessionClient3; View:Surface View / TextureView; MViewPosition:1.

[0142] For example, when the display device 200 establishes a screen projection connection with the first terminal device 510 and the second terminal device 520, the following is displayed: Fig.15 The multi-way screen projection interface shown. Among them, the left side is the first screen projection window, the right side is the second screen projection window, the first screen projection window is in an unlocked state, and the locking attribute of the second screen projection window is in a locked state. At this time, a screen projection connection instruction is sent to the display device 200 through the third terminal device 530, and the display device 200 detects the locking attribute of the screen projection window according to the second mode. The first screen projection window with the locking attribute in the unlocked state is the second replacement screen projection window. The display device 200 removes the first screen projection window, and sets the orientation mark of the target screen projection window to the orientation mark of the first screen projection window, so that the target screen projection window replaces the position of the first screen projection window, and it appears as follows. Fig.19 The multi-screen projection interface shown.

[0143] It can be understood that the embodiments of the present application are illustrated using the target number as an example. In some application scenarios, if the target number set by the display device 200 is greater than 2, the locking attribute of the projection window is in a locked state, and the number of projection windows in an unlocked state is greater than 1, the display device 200 can mark the projection window in an unlocked state with the highest priority as the second replacement window in a preset priority order, so as to replace the second replacement window with the target projection window.

[0144] In addition, when the display device 200 displays the first projection window and the second projection window, if the user disconnects the projection connection relationship of one of the terminal devices, the projection window of the remaining terminal device may move. For example, when the projection connection relationship between the first terminal device 510 and the display device 200 is disconnected, the display device 200 removes the first projection window and displays the guide window and the second projection window in the multi-projection interface. Since the display device 200 is provided with a default priority order, the display device 200 may set the orientation mark of the second projection window to an orientation mark with a higher priority when recalculating the display area of ​​each window, so that the second projection window suddenly switches to the position of the first projection window. In this way, when the display device 200 displays multiple projection windows at the same time, if the projection connection relationship of one of the terminal devices 500 is disconnected, the position of the projection window will switch back and forth, and the user's line of sight also needs to switch back and forth with the position of the projection window, affecting the user's viewing experience.

[0145] In order to improve the problem of switching back and forth of the above-mentioned projection windows, in some embodiments, after displaying the target projection window, the display device 200 also responds to the termination instruction of the target projection session and deletes the target projection session. The target projection session is a communication session generated based on a third terminal device. The third-party position identifier of the target projection window is recorded, and the target projection window is removed, and the guide window is called, wherein the guide window is used to display the guide page, and the root node of the guide window is the same as the root node of the multi-channel projection interface. Then control the display 260 to display the guide window according to the third-party position identifier.

[0146] For example, the display device has been connected to two clients, namely the first terminal device Client1 and the third terminal device Client3. Fig.17The multi-channel screen projection interface shown in the figure has the first screen projection window of Client1 on the left and the target screen projection window of Client3 on the right. At this time, Client3 is connected to the display device. The screen projection management service listens to the onDisConnect() request of Client3 to disconnect through UsessionConnectionListener, and notifies the data management module to remove the Session of Client3. After the data management module successfully removes the information of Client3, the data management module synchronously notifies the View management module to execute the rootview.removeView() removal operation on the corresponding View, and retains the data information of Client1, including: Session:SessionClient1; View:SurfaceView / TextureView; MViewPos ition: 1. In addition, the guide window on the right is re-called according to the position identifier of Client3, so that the multi-channel screen projection interface appears as follows. Fig.10 The display effect shown.

[0147] In some embodiments, the guide window and the multi-channel projection interface also share the same root, that is, the root node of the guide window and the multi-channel projection interface is also the same. When calling the guide window, the display device 200 can set the guide window to a display state; or, re-call the relevant resources of the window object corresponding to the guide window.

[0148] In order to ensure the standardization of the layout of each window, when displaying the guide window, the display device 200 also calculates the display area of ​​the guide window according to the resolution of the guide page, that is, in some embodiments, the display device 200 detects the screen resolution, the reference resolution and the projection screen resolution. Among them, the reference resolution is the resolution of the guide page, and the projection screen resolution is the resolution of the projection screen in the multi-channel projection interface, that is, the projection screen corresponding to the terminal device 500 currently connected to the display device 200. Then calculate the layout parameters of the guide window according to the screen resolution, the reference resolution and the projection screen resolution. Among them, the layout parameters include layout width and layout height.

[0149] After the display device 200 calculates the layout parameters of the guide window, it can display the guide window according to the layout parameters. Therefore, in some embodiments, when the display device 200 displays the guide window according to the third position identifier, the layout position coordinates of the guide window are generated according to the layout parameters and the third position identifier, and the layout parameters and the layout position coordinates are recorded. Then, the guide window is drawn according to the layout parameters and the layout position coordinates.

[0150] For example, the display device 200 displays Fig.17The multi-channel screen projection interface shown has the first screen projection window of Client1 on the left and the target screen projection window of Client3 on the right. At this time, the screen projection connection between Client3 and the display device is disconnected, the display device 200 deletes the Session of Client3, and removes the View corresponding to Client3. Based on the algorithm provided in the above embodiment, the width and height of each window, as well as the upper left corner coordinates of the window are calculated according to the screen resolution of the display device 200, the picture resolution of the first projection screen and the reference resolution of the guide window, and the above data information is stored through the data management module. The data information is then sent to the View management module, so that the View management module displays the display position of the projection Screen View / TextureView and the display position of the guide Layout through the setLayoutParams method, and the following can be obtained: Fig.10 The display effect of the multi-channel screen projection interface is shown.

[0151] Similarly, in some embodiments, when the multi-channel screen projection interface of the display device 200 includes only one screen projection window, that is, the display device 200 establishes a screen projection connection relationship with only one terminal device 500, the screen projection connection relationship between the terminal device 500 and the display device 200 is disconnected, and the display device 200 deletes the screen projection session of the terminal device and removes the screen projection window to display the default guide window, such as Fig. 9 shown.

[0152] For example, the screen resolution is 1920*1080 and the base resolution of the guide page is 954*538. When the display device 200 displays the default guide window, the layout parameters and layout position coordinates are calculated based on the algorithm provided in the above embodiment. The data transmitted by the data transmission module includes: Session; View: layout; Position: (0, 271); Width: 954; Height: 538; Session; View; Position: (966, 271); Width: 954; Height: 538.

[0153] The View management module uses the setLayoutParams method to display the display position of the guide Layout, which can be obtained as follows: Fig. 9 The display effect is shown.

[0154] Based on the above-mentioned display device 200, some embodiments of the present application also provide a method for switching display of multiple screen projection windows, which is applied to the display device 200 provided in the above-mentioned embodiment, and the display device 200 includes a display 260, a communicator 220 and a controller 250. Among them, the display 260 is configured to display a multiple screen projection interface, and the multiple screen projection interface includes projection windows, and the projection windows include a first projection window and a second projection window. The first projection window is used to display a first projection screen of a first terminal device; the second projection window is used to display a second projection screen of a second terminal device; the first projection window and the second projection window are the same as the root node of the multiple screen projection interface; the communicator 220 is configured to establish communication connections with the first terminal device 510 and the second terminal device 520, respectively; Fig.13 As shown, the method comprises the following procedural steps:

[0155] S100: In response to a screen projection connection instruction sent by a third terminal device, detecting a screen projection replacement mode;

[0156] S200: If the screen projection replacement mode is the first mode, detect the focus position of the focus frame, remove the first replacement screen projection window at the focus position, and display the target screen projection window according to the orientation mark of the first replacement screen projection window, the orientation mark is used to represent the display orientation of the window, the target screen projection window is used to display the projection screen of the third terminal device, and the target screen projection window is the same as the root node of the multi-channel screen projection interface;

[0157] S300: If the screen projection replacement mode is the second mode, detect the locking attribute of the screen projection window, remove the second replacement screen projection window according to the locking attribute, and display the target screen projection window according to the orientation mark of the second replacement screen projection window; the second replacement screen projection window is a screen projection window whose locking attribute is in an unlocked state.

[0158] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a method for switching display of multiple projection windows, and the method can detect the projection replacement mode in response to the projection connection instruction sent by the third terminal device. If the projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement projection window at the focus position is removed, and the target projection window is displayed according to the orientation mark of the first replacement projection window. If the projection replacement mode is the second mode, the locking attribute of the projection window is detected, and the second replacement projection window is removed according to the locking attribute, and the target projection window is displayed according to the orientation mark of the second replacement projection window. Among them, the second replacement projection window is a projection window whose locking attribute is in an unlocked state. The method can replace the projection window by the window locking attribute or the focus position according to different projection replacement modes, so that the display device can quickly switch the target projection window to the position expected by the user, thereby improving the user experience.

[0159] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0160] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0162] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: A display, configured to display a multi-screen projection interface, wherein the multi-screen projection interface includes a projection window, wherein the projection window includes a first projection window and a second projection window, wherein the first projection window is used to display a first projection screen of a first terminal device; The second screen projection window is used to display a second screen projection screen of the second terminal device; The first screen projection window, the second screen projection window and the root node of the multi-channel screen projection interface are the same; A communicator, configured to establish communication connections with the first terminal device and the second terminal device respectively; The controller is configured as: In response to a screen projection connection instruction sent by a third terminal device, detecting a screen projection replacement mode; If the screen projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement screen projection window at the focus position is removed, and the target screen projection window is displayed according to the orientation mark of the first replacement screen projection window, the orientation mark is used to characterize the display orientation of the window, and the target screen projection window is used to display the target screen projection screen of the third terminal device, and the target screen projection window is the same as the root node of the multi-channel screen projection interface; If the projection replacement mode is the second mode, detecting a locking attribute of the projection window, removing the second replacement projection window according to the locking attribute, and displaying the target projection window according to the orientation mark of the second replacement projection window; The second replacement projection window is a projection window whose locking attribute is in an unlocked state.

2. The display device according to claim 1, characterized in that The controller is also configured to: Obtain the lock property of the screen projection window; If the locking attributes of the projection window are all in the unlocked state, setting the replacement mode parameter to the first parameter, where the first parameter is used to characterize that the projection replacement mode is the first mode; If the locking attribute of the screen projection window includes a locking state, the replacement mode parameter is set to a second parameter, and the second parameter is used to characterize that the screen projection replacement mode is a second mode.

3. The display device according to claim 1, characterized in that If the screen projection replacement mode is the first mode, the controller displays the target screen projection window according to the position mark of the first replacement screen projection window, and is configured to: Recording a first position identifier of the first replacement screen projection window; Creating the target screen projection window; Setting the position identifier of the target projection window to the first position identifier; Draw the target screen projection window according to the first orientation identifier.

4. The display device according to claim 1, characterized in that If the screen projection replacement mode is the second mode, the controller is configured to: Traversing the lock properties of the projection window; If the locking attribute of the screen projection window does not include an unlocked state, control the communicator to send a screen projection failure notification to the third terminal device.

5. The display device according to claim 4, characterized in that If the locking attribute of the projection window includes an unlocked state, the controller performs displaying the target projection window according to the position identifier of the second replacement projection window, and is configured to: Recording a second position identifier of the second replacement screen projection window; Creating the target screen projection window; Setting the position identifier of the target projection window to the second position identifier; Draw the target screen projection window according to the second orientation identifier.

6. The display device according to claim 1, characterized in that The controller is also configured to: Controlling the display to display a target locking control, wherein the target locking control is associated with the target projection window; In response to a selection instruction input by the user based on the target lock control, the current value of the lock control and the lock attribute of the target projection window are refreshed.

7. The display device according to claim 1, characterized in that The controller is also configured to: In response to a termination screen projection instruction of a target screen projection session, deleting the target screen projection session, where the target screen projection session is a communication session generated based on the third terminal device; Recording the third position identifier of the target projection window; Remove the target screen projection window and call a guide window, where the guide window is used to display a guide page, and the root node of the guide window is the same as the root node of the multi-channel screen projection interface; The display is controlled to display the guide window according to the third position mark.

8. The display device according to claim 7, characterized in that The controller is also configured to: Detecting the screen resolution, the base resolution and the projection screen resolution, wherein the base resolution is the resolution of the guide page, and the projection screen resolution is the resolution of the projection screen in the multi-channel projection screen interface; The layout parameters of the guide window are calculated according to the screen resolution, the reference resolution and the projection screen resolution, and the layout parameters include a layout width and a layout height.

9. The display device according to claim 8, characterized in that The controller controls the display to display the guide window according to the third position mark, and is configured to: Generating the layout position coordinates of the guide window according to the layout parameters and the third position identifier; Recording the layout parameters and the layout position coordinates; The guide window is drawn according to the layout parameters and the layout position coordinates.

10. A method for switching display of multiple projection windows, characterized in that: Applied to a display device, the display device includes a display, a communicator and a controller, the display is configured to display a multi-way screen projection interface, the multi-way screen projection interface includes a screen projection window, the screen projection window includes a first screen projection window and a second screen projection window, the first screen projection window is used to display a first screen projection picture of a first terminal device; The second screen projection window is used to display a second screen projection screen of the second terminal device; The first screen projection window, the second screen projection window and the root node of the multi-channel screen projection interface are the same; The communicator is configured to establish communication connections with a first terminal device and a second terminal device respectively; the method comprises: In response to a screen projection connection instruction sent by a third terminal device, detecting a screen projection replacement mode; If the screen projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement screen projection window at the focus position is removed, and the target screen projection window is displayed according to the orientation mark of the first replacement screen projection window, the orientation mark is used to characterize the display orientation of the window, and the target screen projection window is used to display the projection screen of the third terminal device, and the target screen projection window is the same as the root node of the multi-channel screen projection interface; If the projection replacement mode is the second mode, detect the locking attribute of the projection window, remove the second replacement projection window according to the locking attribute, and display the target projection window according to the orientation mark of the second replacement projection window; the second replacement projection window is a projection window whose locking attribute is in an unlocked state.

Images