Display equipment and multi-window switching method

Abstract

The embodiment of the invention provides a display device and a multi-window switching method, and the method can obtain the screen resolution of a display in response to a full-screen instruction for full-screen display of a first screen projection window when the display device displays a multi-screen projection interface. Setting a first display area of the first screen projection window according to the screen resolution, and setting a second display area of which the number of pixel points is smaller than or equal to a preset pixel point threshold value; therefore, the first projection screen picture is displayed through the first display area, and the second projection screen picture is displayed through the second display area. According to the method, when the first projection screen window is displayed in a full screen, the second projection screen window is not directly closed, but the second projection screen picture is displayed by using a small number of pixel points which are difficult to perceive by a user in a mode of setting the second display area, so that the display equipment can freely and quickly switch multi-path projection screen interfaces of the full screen and the multi-window according to a use scene, and the user experience is improved. And user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a multi-window switching method. Background Art

[0002] Display devices are intelligent devices that can present user interfaces and support user interaction. Taking smart TVs as an example, smart TVs are based on Internet application technology, have open operating systems and chips, have open application platforms, can realize two-way human-computer interaction functions, and are TV products that integrate multiple functions such as audio and video, entertainment, and data to meet the diverse and personalized needs of users. Display devices can establish communication connections with terminal devices such as mobile phones, and based on specific screen projection protocols, obtain the screen projection data of terminal devices, so as to display the screen projection images of terminal devices through display devices.

[0003] Some display devices can establish communication connections with multiple terminal devices at the same time and receive projection data from multiple terminal devices, forming a multi-channel projection scenario. However, in a multi-channel projection scenario, the display device can only display the projection screen of a single terminal device. When the user wants to switch between multiple projection channels, the projection signal source needs to be frequently switched, and each switching process requires the projection connection to be re-established, resulting in a long waiting time for the switching process, which is not conducive to fast switching and reduces the user experience. Summary of the invention

[0004] The present application provides a display device and a multi-channel window switching method to solve the problem of long waiting time when switching multi-channel screen projection.

[0005] In a first aspect, some embodiments of the present application provide a display device, including: a display, a communicator, and a controller. The display is configured to display a multi-way screen projection interface, the multi-way screen projection interface includes a first screen projection window and a second screen projection window, the first screen projection window is used to display a first screen projection screen of a first terminal device; the second screen projection window is used to display a second screen projection screen of a second terminal device; the communicator is configured to establish a communication connection with the first terminal device and the second terminal device respectively; the controller is configured to execute the following program steps:

[0006] In response to a full screen instruction for displaying the first projection window in full screen, obtaining a screen resolution;

[0007] Setting a first display area of ​​the first projection window according to the screen resolution, and setting a second display area of ​​the second projection window, wherein the number of pixels included in the second display area is less than or equal to a preset pixel threshold;

[0008] Control the display to display the first projection screen in the first display area, and control the display to display the second projection screen in the second display area.

[0009] In a second aspect, some embodiments of the present application further provide a multi-window switching method, which is applied to the display device provided in the first aspect, wherein the display device includes a display, a communicator, and a controller, wherein the display is configured to display a multi-screen projection interface, wherein the multi-screen projection interface includes a first projection window and a second projection window, wherein the first projection window is used to display a first projection screen of a first terminal device; and the second projection window is used to display a second projection screen of a second terminal device; the communicator is configured to establish communication connections with the first terminal device and the second terminal device, respectively; and the method includes:

[0010] In response to a full screen instruction for displaying the first projection window in full screen, obtaining a screen resolution;

[0011] Setting a first display area of ​​the first projection window according to the screen resolution, and setting a second display area of ​​the second projection window, wherein the number of pixels included in the second display area is less than or equal to a preset pixel threshold;

[0012] Control the display to display the first projection screen in the first display area, and control the display to display the second projection screen in the second display area.

[0013] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and a multi-window switching method, which can obtain the screen resolution of the display in response to a full-screen instruction for full-screen display of the first projection window when the display device displays a multi-projection interface. Then, the first display area of ​​the first projection window is set according to the screen resolution, and the second display area of ​​the second projection window is set, wherein the number of pixels contained in the second display area is less than or equal to the preset pixel threshold. Thus, the first projection screen is displayed through the first display area, and the second projection screen is displayed through the second display area. When the first projection window is displayed in full screen, the method does not directly close the second projection window, but displays the second projection screen by setting the second display area using a small number of pixels that are difficult for the user to perceive, so that the display device can freely and quickly switch between the full-screen and multi-window projection interfaces according to the usage scenario, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0016] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0017] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0018] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0019] Figure 5 A schematic diagram of mirror screen projection provided in some embodiments of the present application;

[0020] Figure 6 A schematic diagram of push screen projection provided in some embodiments of the present application;

[0021] Figure 7 A schematic diagram of a multi-screen projection interface provided in some embodiments of the present application;

[0022] Figure 8 A schematic diagram of a functional mode of a display device provided in some embodiments of the present application;

[0023] Fig. 9 A schematic diagram of a guide page provided for some embodiments of the present application;

[0024] Fig.10 A schematic diagram of the interface display when casting a screen in some embodiments of the present application;

[0025] Fig.11 A schematic diagram of the effect of dynamically adjusting the display area provided in some embodiments of the present application;

[0026] Fig.12 A schematic diagram of a playback control button control provided in some embodiments of the present application;

[0027] Fig.13 A schematic diagram of the effect of dynamically adjusting the number of play control button controls provided in some embodiments of the present application;

[0028] Fig.14A schematic diagram of switching full screen effect provided in some embodiments of the present application;

[0029] Fig.15 A schematic diagram of a multi-window switching method flow chart provided in some embodiments of the present application;

[0030] Fig.16 A schematic diagram of the full-screen display effect provided in some embodiments of the present application. DETAILED DESCRIPTION

[0031] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0032] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0033] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0034] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0035] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0036] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 may be a remote controller, a stylus pen, or the like.

[0037] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0038] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0039] In addition to providing the broadcast receiving television function, the display device 200 may also provide an intelligent network television function with computer support functions, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0040] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0041] In some embodiments, the display device 200 includes at least one of a tuner 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0042] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0043] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from a controller, display video content, image content, and a menu control interface component and a user control UI interface, etc.

[0044] In some embodiments, the communicator 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communicators 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communicator 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communicator 220 including a Bluetooth function.

[0045] The communicator 220 can connect the display device 200 to the external device or the server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 to the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 to the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0046] In some embodiments, the display device 200 may also support establishing a communication connection relationship with multiple external devices at the same time. Multiple external devices may be connected through the same. For example, the external device is a terminal device 500 such as a mobile phone or a tablet computer. The first terminal device 510 and the second terminal device 520 may establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices may also be connected to the display device 200 through different types of connection methods. For example, the first terminal device 510 is connected to the display device 200 via a wireless local area network; the second terminal device 520 is connected to the display device 200 via Bluetooth.

[0047] In order to realize the communication connection between the display device 200 and the terminal device 500, the display device 200 and the terminal device 500 are respectively provided with communicators supporting the same communication type. For example, when the display device 200 is provided with a communicator 220 including a WiFi module, the terminal device 500 should also be provided with a communicator including a WiFi module. In addition, when the same type of communicators are provided, a specific communication transmission protocol is also required between the display device 200 and the terminal device 500 for data transmission. For example, WiFi protocol, Bluetooth connection protocol, ZigBee protocol, NFC protocol, etc.

[0048] In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0049] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0050] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0051] In some embodiments, the user interface 280 is an interface that can be used to receive control input.

[0052] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0053] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0054] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0055] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0056] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0057] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0058] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0059] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0060] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0061] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0062] like Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Androidruntime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0063] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0064] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0065] like Figure 4As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0066] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0067] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0068] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0069] After establishing a communication connection relationship with the terminal device 500, the display device 200 can establish a screen projection connection channel, and transmit the projection data based on the screen projection connection, so that the display device 200 can display the projection screen of the terminal device 500. In some embodiments, in order to establish a screen projection connection relationship, the user can send a screen projection connection request through the terminal device 500, and send the screen projection connection request to the display device 200 through the WiFi network. The display device 200 then completes the transmission protocol configuration according to the screen projection connection request, thereby establishing a transmission channel for the projection data with the terminal device 500.

[0070] The display device 200 can also achieve a screen projection connection with the terminal device 500 through different connection methods. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, a screen projection connection can be established based on a WiFi network. For another example, when both the display device 200 and the terminal device 500 are provided with NFC (Near Field Communication) components, a screen projection connection relationship can be established through the NFC component. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the terminal device 500 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network, and other connection methods that can be associated with those skilled in the art based on the connection methods provided in the above embodiments.

[0071] After the screen projection connection is established, the display device 200 can receive the screen projection related data of the terminal device 500 through the screen projection data channel, so as to play the screen projection related content in the display device 200. Under different screen projection data transmission protocols, the screen projection data obtained by the display device 200 is in different forms. For example, the display device 200 and the terminal device 500 can establish a screen projection connection through the Digital Living Network Alliance (DLNA) protocol, Miracast protocol, AirPlay push protocol, WiDi screen projection, NFC mirror screen projection, LeTV screen projection and other protocols.

[0072] Among them, the DLNA protocol is a protocol for sharing media resources between devices. The protocol uses network technology (wired or wireless) to interconnect various devices and exchange data through standard protocols, so that media resources can be shared between devices. The DLNA protocol is based on the UPnP protocol, which enables devices to discover and control each other and achieve seamless transmission and sharing of media resources. The DLNA protocol supports a variety of devices, including TVs, speakers, mobile phones, computers, etc., allowing users to seamlessly switch and share media content between different devices.

[0073] Miracast protocol is a wireless display standard developed by Wi-Fi Alliance based on Wi-Fi direct connection. Miracast screen projection protocol is applicable to Android devices, that is, Android devices can share video images through Miracast screen projection. Smart devices with Android 5.0 or above have built-in Miracast protocol, so you can directly wirelessly project the screen through the WiFi P2P function between the projection device and the WiFi network card of the receiving device without installing any software. In addition, Miracast screen projection also supports synchronous transmission of audio and video.

[0074] Airplay protocol is a wireless playback technology launched by Apple, which is applicable to the wireless screen projection protocol of iOS and Mac systems. Apple devices have built-in Airplay screen projection function. When used with a screen projection device that supports AirPlay server service, the effect is clear and stable. The protocol comes with a screen projection code function. However, it is necessary to ensure that the Apple device and the screen projection device are used in the same LAN (same network segment), and cannot be used across network segments or VLANs.

[0075] WiDi screen projection is a wireless playback technology led by Intel, and is also based on WiFi Direct technology. Windows 8 / 10 / 11 laptops natively support WiDi screen projection, which can be directly projected without installing any software, and supports USB reverse control function.

[0076] In some embodiments, after the display device 200 establishes a screen projection connection with the terminal device 500, the screen projection data can be transmitted through the mirror mode and the push mode. Among them, the mirror mode means that the display device 200 synchronously displays the screen content displayed by the terminal device 500. For example, Figure 5 As shown, the display device 200 displays the main interface of the terminal device 500, and when the user performs an interactive operation on the terminal device 500, resulting in a change in the content of the main interface, the content displayed by the display device 200 also changes. To this end, the terminal device 500 can generate a screen projection data stream in the form of a video based on the displayed content, and send the screen projection data stream to the display device 200 for display.

[0077] The push mode means that the display device 200 accesses the network link according to the network link pushed by the terminal device 500 to obtain the corresponding projection screen. Figure 6As shown, when the terminal device 500 plays the video media, a screen projection control is provided in the upper right corner of the play interface. After the user clicks the screen projection control and selects the display device 200 as the device for displaying the screen projection image, the terminal device 500 can send the URL address of the currently played video media to the display device 200. After receiving the URL address, the display device 200 accesses the URL address to obtain the video media data and plays the video media.

[0078] Among the screen projection protocols supported by the display device 200 and the terminal device 500, some protocols are based on the mirror mode, and some protocols are based on the push mode. For example, under the DLNA protocol, the display device 200 can obtain the link address of a resource in the terminal device 500 through the screen projection data channel, and obtain the media content data by accessing the link address, thereby playing the media data. At the same time, the display device 200 can also obtain control instructions sent by the terminal device 500 through the screen projection data channel when playing media data, such as fast forward, pause, stop screen projection, etc. For example, after establishing a connection through the communication protocol, the terminal device 500 such as a mobile phone and the display device 200 will negotiate the device functions and network conditions to select a suitable audio and video transmission format. After establishing the online protocol (Session) of the audio and video streaming, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the audio and video streaming.

[0079] Under other screen projection connection protocols such as Miracast, Airplay, NFC mirroring, and LeTV, the display device 200 can directly obtain the video data stream from the terminal device 500. The video data stream can be the video data stream corresponding to the final display screen of the display unit of the terminal device 500, that is, the video data stream includes the media content screen, UI interface, and other video screen content that can be displayed in the display unit of the terminal device 500. The video data stream can also be only the video data that can be obtained in the terminal device 500, such as the media video data being played. At this time, the terminal device 500 can be used as a signal source for the display device 200, and the display device 200 can continuously obtain video data through the screen projection data channel, and display it in real time on the display 260, so as to achieve the effect of synchronous display with the terminal device 500.

[0080] It should be noted that the above-mentioned screen projection protocol can not only obtain the video data stream in the terminal device 500, but also obtain the audio data stream in the terminal device 500, so that the video data and audio data can be sent to the display device 200 synchronously, so that the display device 200 can play the video and audio synchronously.

[0081] The terminal device 500 can obtain different projection data according to different projection protocols. For example, for system applications or direct projection at the system level, the terminal device 500 can use Miracast projection, Airplay projection or WiDi projection protocol to establish a projection data channel. Since the above-mentioned Miracast projection protocol and Airplay projection protocol require system-level application permissions, the Miracast projection protocol and Airplay projection protocol are only applicable to system applications, or the display device 200 and the terminal device 500 of the same manufacturer directly establish a projection connection through the built-in rules of the operating system.

[0082] The display device 200 can establish a screen projection connection with the terminal device 500 based on any of the screen projection methods provided in the above embodiments. In some embodiments, the display device 200 can also establish a screen projection connection relationship with multiple communication terminals through one or more of the above screen projection methods. For example, the display device 200 can simultaneously establish a screen projection connection with the first terminal device 510 and the second terminal device 520, and divide the display area in the display screen to display the projection screen shared by the first terminal device 510 and the second terminal device 520 respectively.

[0083] In order to display the projection screen shared by multiple terminal devices 500 on the display device 200, in some embodiments, when the display device 200 establishes a projection connection with multiple terminal devices 500, the display device 200 can display a multi-channel projection interface, which can include multiple projection windows. The projection window is used to display the projection screen of the corresponding terminal device 500 under a projection connection channel. For example, Figure 7 As shown, the multi-channel screen projection interface may include two screen projection windows, namely a first screen projection window and a second screen projection window. The first screen projection window is used to display the projection screen of the first terminal device 510; the second screen projection window is used to display the projection screen of the second terminal device 520.

[0084] It should be noted that the projection window in the multi-channel projection interface may not be strongly bound to the terminal device 500, that is, in some embodiments, the display content of the projection window may be determined according to the order in which the terminal device 500 establishes a projection connection with the display device 200. For example, when the first terminal device 510 establishes a projection connection with the display device 200 before the second terminal device 520, the first projection window on the left side of the multi-channel projection interface displays the projection screen of the first terminal device 510. When the first terminal device 510 establishes a projection connection with the display device 200 later than the second terminal device 520, the first projection window on the left side of the multi-channel projection interface displays the projection screen of the second terminal device 520.

[0085] In order to obtain a better picture display effect and improve the coordination of the multi-channel projection interface, in some embodiments, the display device 200 can also set the layout of the projection window. Figure 8 As shown, the display device 200 can have multiple functional modules built in, including: a data management module, a message processing module, a service management module, an algorithm processing module, a View management module, an application interaction processing module, a location management module, a replacement management module, a broadcast control management module, a time management module, a focus management module, and a screen projection management module, etc.

[0086] Among them, the data management module mainly manages all message data involved in the current template and updates the data information synchronously. The message processing module mainly transmits related messages between different modules, such as transmitting data changes to the View layer for refresh display, or synchronizing View changes to the data management module for change processing of broadcast control data, etc. The service management module is responsible for opening and closing the services of the entire application and some customized processing, such as starting specific services in specific scenarios. The algorithm processing module is responsible for all layout position calculations of the entire application. The View management module is mainly responsible for all UI displays of the entire template application, and receives data to refresh the View display.

[0087] The application interaction processing module is responsible for the processing of interactions with external applications. For example, any entry to start the current template application needs to interact with this module, and different customized management is performed according to the differences in the startup entries. The position management module is responsible for the rules for displaying the projection position in the left and right windows, such as on the left or on the right. The replacement management module is responsible for the replacement rules when the number of projection paths exceeds the maximum display number, such as how to replace when there is focus or no focus. The broadcast control management module is responsible for the broadcast control processing of different media types in the projection, such as fast forward, rewind, play pause, etc. The time management module is responsible for the focus display timing processing in the projection, whether it is the large frame focus or the broadcast control position focus. The focus management module is responsible for the focus display and button processing in the entire application. The projection management module is responsible for the projection protocol interaction and the client list management and window display content status information processing.

[0088] It should be noted that the multi-channel screen projection interface can realize screen display based on the multi-channel screen projection viewing application. The multi-channel screen projection viewing application is an application that can display multi-protocol multi-channel mirror projection and push content, and is an upper-level application inside the display device 200. The framework layer of the display device 200 can provide a Java interface for docking with hardware, etc. The multi-channel screen projection viewing application realizes docking with the lower layer and obtains information through the Java interface of the Framework layer. Other National, Runtime, HAL, and Linux kernel layers are all standard layers of the Android architecture, and each layer interacts according to the business process.

[0089] The display device 200 first starts the application through the service management module and performs some initial preparations, such as starting the monitoring mirror or pushing the corresponding protocol service; the View management module initializes the View displayed by the application layer; the data management module creates some data for the data processing module; the algorithm processing module performs algorithmic layout calculations on the projection content, and the position management module is responsible for positioning the left and right positions of the projection content.

[0090] The service management module can start the screen projection application and register the Session during the process of establishing the screen projection connection; Usession Connection Listener, Usession Prepare Listener, Usession DisplayListener and Usession Event Listener. Among them, Session represents a conversation between the client and the server, which can save the client's information. Usession Connection Listener is used to monitor the connection and disconnection events of the Session. Usession Prepare Listener interacts with the screen projection preparation of the Client. Usession DisplayListener monitors the changes in playback information and status and the status changes of the Client. Usession Event Listener processes information such as Client control.

[0091] After the initialization of View is completed, the display device 200 will display the default guide page when entering the application. The guide page will display some device information, such as the device name of the current display device 200 and the corresponding network information. Then, the guide page will be displayed in equal proportion to the left and right by controlling the standard control TextView and ImageView of the system and the corresponding Layout, such as Fig. 9 shown.

[0092] After displaying the guide page, the display device 200 can monitor and receive the screen projection connection instruction sent by the terminal device 500, and establish a screen projection connection channel according to the screen projection connection instruction, so as to display it in the screen projection window. For example, the screen projection management module is always ready for the access of a new Session. When Client1 is ready to start the connection, the screen projection management module monitors the onConnect connection of the new Session through UsessionConnectionListener, and then creates the playback control SurfaceView / TextureView. After the creation is successful, the position management module is started to set and manage the display position of the current Session. The position management module processes and obtains the position. The first screen projection is displayed on the left by default. The mViewPosition of Client1 is marked as 1, where 1 represents the left side and 2 represents the right side. It can also be represented by other parameters.

[0093] To facilitate the management of View and Session, in some embodiments, a management object MultiCastPlayView may be set, wherein the management object MultiCastPlayView may be associated with information such as Session, SurfaceView / TextureView, and mViewPosition. For example, for the first terminal device 510, the data corresponding to its Client1 may be represented as: Session:SessionClient1, View:SurfaceView / TextureView, MViewPosition:1.

[0094] When the display device 200 prepares the View as the server, it can notify the terminal device 500 as the mobile terminal to record or send the video stream through UsesessionPrepareListener. When the terminal device 500 is ready to send video information, the display device 200 knows that it will be cast through onFirstFrame() of UsesessionDisplayListener, so it can get the current width and height of Client1's video from the Session, determine the direction of the current video stream through the width and height, and then calculate the current position and size to be displayed through the algorithm.

[0095] That is, the first projection window and the second projection window can also present corresponding shapes according to the size of the displayed projection screen. For example, when the first projection screen and the second projection screen are both 1920×1080 horizontal screens, the display device 200 can present the first projection window and the second projection window with a size of 960×540 in the multi-channel projection interface.

[0096] In some embodiments, in order to distinguish the display range of the projection area, a certain blank area may be left in the first projection window and the second projection window, the projection window and the guide area. For example, the blank area is 12 pixels, and the resolution of the display device 200 is 1920×1080. Fig.10 As shown, when the first terminal device 510 (Client 1) is displayed in horizontal format, the size of the projection screen in the corresponding projection video stream is 1920×1080, and the guide page display size is 954×538. At this time, the video stream ratio and the right guide video stream ratio are close to 16:9 and are both horizontal. It can be calculated through the algorithm that the left and right sides can be displayed in proportion, that is, they can be displayed with equal width and height, that is, the width of the video stream and the guide page is 954, and the height is about 538.

[0097] Since the terminal device 500 may be placed in different states according to different usage modes, for example, a terminal device 500 such as a smart phone may be in a portrait state and a landscape state during use. The portrait state refers to a state in which the width of the displayed image is less than the height, and the landscape state refers to a state in which the width of the displayed image is greater than the height. In different placement states, the terminal device 500 may present different screen layouts.

[0098] Based on this, in some embodiments, when the display device 200 displays the projection screens of multiple terminal devices 500 by area division, each display area will present different screen ratios due to the different placement states of the terminal devices 500. For example, the first terminal device 510 is in a horizontal screen state, the first projection screen size is 1920×1080, and the second terminal device 520 is in a vertical screen state, and the second projection screen size is 512×1088. Then the first projection screen window can be 954×538, and the second projection screen window can be 508×1080.

[0099] In order to make full use of the display space of the multi-way projection interface and reduce large black screen areas, in some embodiments, when the sizes of the first projection screen and the second projection screen are inconsistent, the display device 200 can set the width or height of the first projection window and the second projection window to be consistent, and scale the projection windows according to the width or height.

[0100] For example, when the first terminal device 510 is in portrait mode, the size of the first projection screen is 512×1088; the second terminal device 520 is in landscape mode, the size of the second projection screen is 1920×1080, and the display resolution of the display device 200 is 1920×1080, the default size of the first projection window and the second projection window (i.e., the guide page display size) is 954×538. At this time, the video stream ratio is different from the right guide video stream ratio and the direction is different. It can be calculated through the algorithm that the left video stream display height is about 848 and the width is about 400. Excluding the blank area, the remaining width is the guide page width, and the height is consistent with the video stream.

[0101] To this end, the display device 200 can set the parameters of the first projection window, namely Client 1: width1 (width), height1 (height); Guide (guide page): width_guide, height_guide. The height of the projection screen video stream from the upper edge of the screen is set to top, because height1 and height_guide are ultimately the same height, and can be replaced by the same variable height. The video stream ratio ratio1 and the guide page ratio ratio_guide are both known constants. Through the following combined calculations, width1, top, height, and width_guide can be calculated; that is, width1+width_guide=1920; top*2+height=1080; width1 / height=ratio1; width_guide / height=ratio_guide.

[0102] Based on the above algorithm, the algorithm processing module of the display device 200 can obtain the corresponding width and height position of the content displayed on the left and right sides. That is, the width and height of Client1 can be calculated to be 848 and 400, and the corresponding position coordinates are (0, 116) from the upper left corner, and the guide page coordinates are (412, 116).

[0103] After the above position calculation is completed, the data management module of the display device 200 can sort and store, and then transmit the data, including: Session: SessionClient1, View: SurfaceView / TextureView, MViewPosition: 1, Position: (0, 116), Width: 400, Height: 848; Session: View: layout, MViewPosition: Position: (412, 116), Width: 1508, Height: 848. After receiving the data information, the View management module of the display device 200 sets the display parameters through LayoutParams, including the position (i.e., coordinates) from the upper and left boundaries, and the corresponding display width and height, and then displays the position through the setLayoutParams method. That is:

[0104] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0);lp1.width=400;lp1.height=848;lp1.leftMargin=0;lp1.topMargin=116;mPlayerView.setLayoutParams(lp1);

[0105] FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0);lp1.width=1508;lp1.height=848;lp1.leftMargin=412;lp1.topMargin=116;mPlayerView.setLayoutParams(lp1).

[0106] By setting the display position of the projection SurfaceView / TextureView and the display position of the guide Layout respectively, you can get the following Fig.11 The display effect of the multi-channel screen projection interface is shown.

[0107] In some embodiments, when the display device 200 displays the multi-channel projection interface, it can also display the broadcast control function control in the multi-channel projection interface, and the user can perform interactive operations through the broadcast control function control to input control instructions to control the display device 200 to perform interactive responses. For example, since the projection contents of the display device 200 and the terminal device 500 are various, there are mirrored video streams, pushed URL addresses, and different media types such as pictures and music, the broadcast control function controls of different media types are also different. For example, the mirrored video projection does not support fast forward and rewind operations nor does it support play and pause operations; the pushed URL address can fast forward, rewind, play, pause, etc.; the music can fast forward, rewind, play, pause, etc.; the picture does not support fast forward, rewind, play, pause, etc.

[0108] The display device 200 can dynamically adjust the broadcast control function controls according to the display area of ​​the projection window in the multi-channel projection interface, so that the display position of the broadcast control function controls is adapted to the projection window to facilitate the user to perform interactive operations. For example, the broadcast control function controls belonging to the same projection window can form a broadcast control list. The broadcast control list includes several button controls such as full screen, fast forward, fast rewind, play / pause, mute / unmute, and close. The size of each button control is 60×60, and the horizontal spacing of the button controls is 24. Combined with the left and right margins of the broadcast control list, the minimum display width for displaying six button controls is 528, the minimum display width for displaying three button controls is 276, the minimum display width for displaying two button controls is 192, and the minimum display width for displaying one button control is, and the same applies to other numbers.

[0109] In some embodiments, when the screen projection management module listens to the onConnect connection of the new Session through UsessionConnectionListener, the media resource type of the current Client can be obtained from the Session, such as VIDEO_URL\MIRROR_STREAM\MUSIC\PIC, etc. For example, if the currently displayed VIDEO_URL address is pushed, the display device 200 can perform full screen, fast forward, fast rewind, play / pause, mute / unmute, and close processing. At this time, the broadcast control data can be added to this Session, that is, the type data of each button control can be expressed as: DoublePushControlBean[mStatus=××,mType=××,mResId=××]. Based on this, the entire broadcast control list is the List list of the above-mentioned DoublePushControlBean, and the corresponding data can be expressed as: DoublePushControlListBean[List <doublepushcontrolbean>=××,Status=××].

[0110] For example, when there are 6 button controls below the projection window, the corresponding data format is:

[0111] Full screen button control: DoublePushControlBean[mStatus=true, mType=fullscreen, mResId=2131231132];

[0112] Replay button control: DoublePushControlBean[mStatus=true, mType=rewind, mResId=2131231110];

[0113] Fast rewind button control: DoublePushControlBean[mStatus=true, mType=playstatus, mResId=2131231149];

[0114] Fast forward button control: DoublePushControlBean[mStatus=true, mType=forward, mResId=2131231111];

[0115] Mute button control: DoublePushControlBean[mStatus=true, mType=mutestatus, mResId=2131231152];

[0116] Close button control: DoublePushControlBean[mStatus=true, mType=close, mResId=2131231121].

[0117] In order to display the above button controls, after the data management module obtains the display size of the window, it uses the View management module to layout and display the following: Fig.12 The effect shown in the figure is that the left window is the URL address media resource, which can be played and controlled. The algorithm processing module calculates that the left window size is 1508×848 and the right window size is 400×848. The following data results are obtained:

[0118] Session:SessionClient1,View:SurfaceView / TextureView,MViewPosition:1,Position:(0,116),Width:1508,Height:848,mDoublePushControlListBean:…

[0119] Session:SessionClient2,View:SurfaceView / TextureView,MViewPosition:2,Position:(1520,116),Width:400,Height:848,mDoublePushControlListBean:…

[0120] After calculating the window display position, the View management module will layout and display the left and right windows. During the display process, the playback control View needs to be initialized and displayed according to the playback control list data of the Session. For example, the display device 200 can use the Recycleview control for display, and the playback control menu is displayed by setting the margin and position of the Recycleview. The display parameters are also set through LayoutParams, including the position from the upper boundary and the left boundary, i.e., the coordinates, and the corresponding display width and height, and then the position is displayed through the setLayoutParams method.

[0121] After calculation by the algorithm processing module, the display area width 1508 of Client1 is larger than the minimum width 528 of the play control bar, so it can be fully displayed. Fig.12 The display effect can fully display the six play control buttons in the center, or display them in any other position. You can press the down button on the remote control to jump to the play control button for operation.

[0122] When the media content of the VIDEO_URL type changes or the direction of the video stream changes, the display device 200 can rearrange the display of the projection content, such as Fig.13 The left window shown is a VIDEO_URL type media asset, but the algorithm processing module calculates that the left window size is 459×816 and the right window size is 1449×816. The following data results are obtained:

[0123] Session:SessionClient1,View:SurfaceView / TextureView,MViewPosition:1,Position:(0,132),Width:459,Height:816,mDoublePushControlListBean:…

[0124] Session:SessionClient2,View:,MViewPosition:,Position:(1520,132),Width:1449,Height:816,mDoublePushControlListBean:…

[0125] After calculation by the algorithm processing module, the display area width of Client1 459 is less than the minimum width of the play control bar 528, but greater than 276, so some buttons can be displayed, such as Fig.13 The display effect can display the three playback control buttons completely in the center, or in any other position. You can jump to the playback control buttons by pressing the down button on the remote control to operate. That is, display the three button controls of full screen, mute, and close. The data format is as follows:

[0126] DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132];

[0127] DoublePushControlBean[mStatus=true,mType=mutestatus,mResId=2131231152];

[0128] DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0129] When the window size of the projection window changes, the display device 200 can notify the Recycleview to refresh the View through the notifyDataSetChanged of the Adapter, changing from 6 buttons to 3 buttons.

[0130] like Fig.12 In the play control list shown in the figure, when the user clicks the third play control button, Recycleview will listen to the user's onClick operation, and for the index type of the play control list, that is, the mType type is playstatus, then call the Session's setPlayerPause() interface to process the play control, and the same fast forward and rewind also call the Session's setPlayerSeekBackward() and setPlayerSeekForward() to process the play control. That is, modify the data format to: DoublePushControlBean[mStatus=true,mType=playstatus,mResId=2131231149].

[0131] The screen projection management module monitors the screen projection session status. When the video changes, the algorithm processing module is notified through the message processing module. The algorithm processing module calculates the screen projection display area and then informs the data management module through the message processing module. After the data management module organizes the data, it notifies the View management module through the message processing module to redraw and display the View.

[0132] Similarly, when the projection management module listens to the onConnect connection of the new Session through UsessionConnectionListener, it can obtain from the Session that the media resource type of the current Client is MIRROR_STREAM, then the large screen can be full screen, mute / unmute, and closed. At this time, the broadcast control data can be added to this Session as follows: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132];

[0133] DoublePushControlBean[mStatus=true,mType=mutestatus,mResId=2131231152];

[0134] DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0135] According to the data from the data management module, the right window size is 400×848, and the size of the three controls is 276. Since the right window width 400 is larger than the width 276 occupied by the three controls, the three mirrored play control buttons can be fully displayed. Then the Recycleview is initialized and displayed. The display parameters are set through LayoutParams, including the position from the upper and left boundaries, i.e. the coordinates, and the corresponding display width and height. Then the position is displayed through the setLayoutParams method. The corresponding play control list data is modified to Session:SessionClient2,View:SurfaceView / TextureView,MViewPosition:2,Position:(1520,116),Width:400,Height:848,mDoublePushControlListBean:…

[0136] It can be seen that in the above embodiment, the display device 200 can adjust the number of button controls from six to three for display when the width of the projection window cannot meet the arrangement of six button controls. However, when the width of the projection window cannot meet the three broadcast control button controls, it is necessary to continue to adjust the number of broadcast control buttons according to the projection window. If the right window is 258×864, the window width is less than the minimum display width of 276 when three button controls are displayed, and greater than the minimum display width of 192 when two button controls are displayed. Therefore, the display device 200 can display two broadcast control button controls below the projection window. In addition, since the volume can be mutually exclusive controlled by the other party's buttons, only the full screen and close button controls can be retained, that is:

[0137] DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132];

[0138] DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0139] Similarly, if the right window size is 156×864, the window width is less than the minimum display width of 192 required for two button controls, but greater than the minimum display width of 108 required for a button control, so one button can be displayed. Since closing the projection window can be controlled by stopping the projection operation on the terminal device 500, only the full screen button can be retained, that is: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132].

[0140] By analogy, if the right window size is 88×864, the window width is less than the display width 108 required for a button control, so the play control button cannot be displayed. Therefore, the display device 200 can cancel the display of the play control button control below the right window. It should be noted that the display method of the play control button control provided in the above embodiment is only an example. When the play control button control cannot be fully displayed horizontally, the display device 200 can also display it by vertical arrangement, can be arranged and displayed in a rectangular array or a circular array, or can be displayed by edge sliding, but the basic display principle is universal and is not limited to the above expression method.

[0141] In some embodiments, the user can input a control instruction based on the playback control button control in the multi-channel screen projection interface to control the display device 200 to switch interfaces. Fig.14 As shown, when the user clicks the full-screen button below the first projection window, the display device 200 can switch from the multi-way projection interface to the full-screen display of the projection screen in the first projection window, that is, display the projection screen of the first terminal device 510.

[0142] Since the display device 200 no longer displays the projection screen corresponding to the second projection window after switching to full-screen display of the projection screen of the first projection window, the display device 200 can close the projection data transmission channel corresponding to the second projection window, that is, stop the projection connection relationship with the second terminal device 520.

[0143] However, after the display device 200 switches to the full-screen display of the projection image in one projection window, the user may also switch back to the multi-channel projection interface as needed. For example, after controlling the display device 200 to display the projection image corresponding to the first projection window in full screen, if the user wants to watch the projection image in the second projection window, the user can control the display device 200 to switch back to the multi-channel projection interface by pressing the "return" key on the remote control of the display device 200. At this time, since the display device 200 has interrupted the projection connection relationship with the second terminal device 520, the display device 200 needs to re-establish the projection connection relationship and obtain the projection data sent by the second terminal device 520, so as to display the corresponding projection image through the second projection window in the multi-channel projection interface.

[0144] In some embodiments, the display device 200 can quickly switch multiple projection windows by executing a multiple window switching method. In order to meet the implementation of the multiple window switching method, the display device 200 should at least include a display 260, a communicator 220 and a controller 250.

[0145] The display 260 is configured to display a multi-screen projection interface, which includes a first projection window and a second projection window. The first projection window is used to display a first projection screen of a first terminal device; the second projection window is used to display a second projection screen of a second terminal device. The communicator 220 is configured to establish a communication connection with the first terminal device and the second terminal device respectively. Fig.15 As shown, the controller 250 is configured to execute program steps corresponding to the multi-window switching method, specifically including the following contents:

[0146] S100: In response to a full screen instruction for displaying the first projection window in full screen, obtaining a screen resolution.

[0147] When the display device 200 displays the multi-channel projection interface, the user can enter a full-screen command by clicking the broadcast control button control in the multi-channel projection interface. For example, if the first projection window is the window on the left side of the multi-channel projection interface, then when the multi-channel projection interface is displayed, the display device 200 can display a "full screen" button control below the first projection window according to the broadcast control button control display method provided in the above embodiment. If the user wants to display the left window in full screen and not watch the right window temporarily, the user can perform interactive operations through the full screen button control below the left window. The user can control the focus cursor to move to the "full screen" button control through the direction keys of the remote control, and press the "confirm" key on the remote control to enter the full screen command.

[0148] In this regard, the Recycleview of the playback control bar in the display device 200 will monitor the user's operations in real time. When the user operation type mType is a fullscreen operation and the user's OnClick event is monitored, DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132] is modified to display the selected projection screen in full screen.

[0149] The user can also input the full-screen command in other ways. For example, when the focus cursor selects the first projection window, the user can long press the "Confirm" key to control the display device 200 to display the projection screen corresponding to the first projection window in full screen, that is, input the full-screen command by long pressing the Confirm key; for the display device 200 that supports touch interactive operation, the user can directly click the "full screen" button control by touch to input the full-screen command; for the display device 200 that supports voice interactive operation, the user can input the full-screen command by inputting the voice of "display the left window in full screen".

[0150] In some embodiments, the full-screen instruction can also be automatically input by the display device 200 through the judgment of the display scene. For example, when the display device 200 displays a multi-channel projection interface, it can record the display time of the projection screen in each projection window. When the time for displaying a specific projection screen in the first projection window exceeds a preset time threshold, and the specific projection screen is not displayed in the second projection window (displaying the guide interface), the display device 200 can automatically generate a full-screen instruction and automatically display the first projection window in full screen.

[0151] After obtaining a full-screen instruction for full-screen display of the first projection window, the display device 200 can obtain the screen resolution in response to the full-screen instruction. The screen resolution can be the resolution of the display 260 or the resolution of the effective display area in the user interface. For example, for a display device 200 in which the entire screen is an effective display area, the screen resolution of 1920×1080 of the display 260 can be directly obtained after obtaining the full-screen instruction. For a special-shaped screen with a camera module on the top, the total resolution of the display 260 is 1920×1180, and the height occupied by the camera module is 100. Then, after obtaining the full-screen instruction, the display device 200 can obtain the screen resolution of the effective display area after removing the camera module area 100, that is, 1920×1080.

[0152] It should be noted that in the embodiment of the present application, the first projection window and the second projection window are only used to distinguish different projection windows, and the first projection window is located on the left side of the multi-way projection interface and the second projection window is located on the right side of the multi-way projection interface. It should be understood that the first projection window and the second projection window do not limit the specific projection window display position in the multi-way projection interface. The first projection window is only used to indicate the window that needs to display the projection screen in full screen, and the second projection window is only used to indicate the window that does not need to display the projection screen in full screen. Therefore, the first projection window can also refer to the window located on the right side of the multi-way projection interface, and the second projection window can also refer to the window located on the left side of the multi-way projection interface.

[0153] In some embodiments, in order to indicate a window that is displayed in full screen, the display device 200 may also set a conversion mode parameter for the window. That is, the display device 200 may set the conversion mode parameter of the first projection window to a first parameter value and the conversion mode parameter of the second projection window to a second parameter value in response to a full screen instruction for displaying the first projection window in full screen. The first parameter value is used to indicate that the projection window is displayed in full screen; the second parameter value is used to indicate that the projection window is hidden.

[0154] For example, when the mode management module of the display device 200 switches between the dual-screen projection mode and the full-screen mode, a SwitchMode attribute can be set. When SwitchMode is the full-screen mode, the parameter is the first parameter value full; when SwitchMode is the hidden mode, the parameter is the second parameter value Hidden; when SwithchMode is the two-way screen projection window mode, the parameter is the third parameter value double, which is used to characterize the various logics and display processing in the switching process.

[0155] S200: Setting a first display area of ​​the first projection window according to the screen resolution, and setting a second display area of ​​the second projection window.

[0156] After obtaining the screen resolution, the display device 200 can display the projection screen of the first projection window in full screen. However, since the screen resolution of the display device 200 is inconsistent with the size of the projection screen corresponding to the first projection window, the display device 200 needs to display the projection screen of the first projection window according to the screen resolution.

[0157] In some embodiments, when setting the first display area, the display device 200 may obtain the screen resolution of the first projection screen; read the screen width and screen height from the screen resolution; and then determine a reference size according to the screen resolution and the screen resolution, where the reference size is the screen width or the screen height. Then, based on the reference size, calculate the scaling ratio. Finally, scale the first projection screen according to the scaling ratio.

[0158] For example, the video stream size of Client 1 is 512×1088, while the screen resolution of the display device 200 is 1920×1080, and the guide page display size is 954×538. At this time, the video stream ratio is different from the guide video stream ratio on the right and the direction is different. The base size can be set according to the video height. Considering the page layout and the right window ratio, the left video stream display height can be set to 848, and the scaling ratio is calculated according to the display height 848 and the original video height, that is, 848 / 1088. Then, the display width is calculated according to this scaling ratio, that is, 512×848 / 1088=399.

[0159] In order to improve the switching speed when the display device 200 switches from full-screen display of the first projection screen to display of the multi-channel projection screen interface, the display device 200 may temporarily not close the second projection screen window while displaying the first projection screen in full screen, but hide the projection screen window. Therefore, the display device 200 can set the first display area of ​​the first projection screen window according to the screen resolution to achieve the effect of full-screen display of the first projection screen, and at the same time, set the second display area of ​​the second projection screen window. Among them, the second display area can complete the picture display with fewer pixels, so that the user cannot perceive the existence of the second projection screen. Therefore, the number of pixels contained in the second display area is less than or equal to the preset pixel threshold. The preset pixel threshold is the maximum number of pixels that the display device 200 can not be distinguished by the user at a preset viewing distance when displaying the picture. For example, for a display device 200 such as a TV, the user's viewing distance is 1-3m, then at this viewing distance, a picture with less than 4 pixels will not be perceived by the user, and the preset pixel threshold is 4.

[0160] In some embodiments, the display device 200 may determine a boundary area according to the first display area when calculating the second display area, wherein the boundary area is an area outside the first display area in the entire display area of ​​the display, or an area formed by edge pixels of the first display area. Then, according to the preset pixel threshold, the minimum number of pixels of the second display area is set, and according to the minimum number of pixels, the second display area is demarcated in the boundary area.

[0161] For example, the display device 200 sets the first display area to 1920×1080 and sets the second display area to 1×1. The first display area can realize full-screen display of the first projection screen. At the same time, the second projection screen is displayed by one pixel point, and one pixel point will not affect the display of the first projection screen, and a pixel point of the second projection screen can be set to be located in an area that is not easily noticed by the user, such as the upper left corner, upper right corner, lower left corner, lower right corner, etc. of the display 260.

[0162] S300: Control the display to display the first projection screen in the first display area, and control the display to display the second projection screen in the second display area.

[0163] After setting the first display area and the second display area, the display device 200 can display the first projection screen and the second projection screen respectively through the first display area and the second display area. Among them, the first display area includes more pixels, for example, the first display area can be the entire display screen area, or a larger proportion of the middle display area, etc. Therefore, the effect of full-screen display of the first projection screen can be achieved. The second display area is less pixel points, for example, the first display area is one pixel point, or two pixel points, etc. Therefore, the effect of hiding the second projection screen can be achieved.

[0164] For example, in the video stream projected by the first terminal device 510, the projected screen size is 1920×1080. According to the interface switching display mode provided in the above embodiment, it can be determined that the first display area Client1 is 1920×1080 and the second display area Client2 is 1×1.

[0165] At this time, Client2 can be ignored, and the default screen size of Client1 is 1920×1080. The full screen Client1 is calculated based on the video stream ratio ratio1 and the ratio of the entire screen ratio_fullscreen. Combined with the horizontal and vertical versions of Client1, it is determined that the current video ratio is consistent with the screen ratio, so the display area size of Client1 is 1920×1080, that is: Session:SessionClient1,View:SurfaceView / TextureView,MViewPosition:-1,Position:(0,0),Width:1920,Height:1080,SwitchMode:full…

[0166] Similarly, the second display area Session: SessionClient2, View: SurfaceView / TextureView, MViewPosition: -1, Position: (0, 0), Width: 1, Height: 1, SwitchMode: Hidden…

[0167] Since Client1 is displayed in full screen, when Client2 is displayed through one pixel, the display position can be displayed in any corner without affecting the user's viewing. For example, the display position Position of the second projection window can be (0,0), or (1919,0), (0,1079), (1919,1079), etc.

[0168] After setting the coordinates and width and height of the first display area and the second display area, the algorithm processing module notifies the data management module through the message processing module to organize the data, and then notifies the View management module. After receiving the data information, the View management module sets the display parameters through LayoutParams, including the position coordinates from the upper and left boundaries of the display 260, and the corresponding display width and height, and then displays the positions of the two Clients through the setLayoutParams method.

[0169] That is, for the first projection window: FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0); lp1.width=1920; lp1.height=1080; lp1.leftMargin=0; lp1.topMargin=0; mPlayerView.setLayoutParams(lp1).

[0170] For the second projection window: FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0); lp1.width=1; lp1.height=1; lp1.leftMargin=0; lp1.topMargin=0; mPlayerView.setLayoutParams(lp1).

[0171] It can be seen that in the above embodiment, the display device 200 can obtain the screen resolution after obtaining the full-screen instruction, and set the first display area according to the screen resolution to achieve the display effect of full-screen display of the first projection screen, and at the same time set the second display area containing pixels less than the preset pixel threshold to display the second projection screen through fewer pixels, so as to achieve the effect of hiding the second projection screen, and then display the first projection screen and the second projection screen based on the first display area and the second display area respectively, so as to achieve the purpose of full-screen display of one projection screen and hiding the other projection screen. After switching to the full-screen state, the projection connection relationship between the display device 200 and the second terminal device 520 is not closed. When switching back to the multi-projection interface, there is no need to re-establish the projection connection, and the multi-projection interface can be quickly presented.

[0172] Therefore, in some embodiments, the display device 200 can also obtain the first resolution of the first projection screen and the second resolution of the second projection screen in response to the multi-window instruction for exiting the full-screen display of the first projection window when switching back to the multi-projection window state in the full-screen state; reset the first display area according to the first resolution, and reset the second display area according to the second resolution; control the display to display the first projection screen in the first display area after the reset, and control the display to display the second projection screen in the second display area after the reset.

[0173] Since the multi-screen projection interface includes the first screen projection window and the second screen projection window, when switching back to the multi-screen projection interface, the display device 200 needs to reset the first display area and the second display area. The method of resetting the first display area and the second display area can be the same as the multi-window display method provided in the above embodiment, that is, obtaining the first resolution of the first screen projection picture and the second resolution of the second screen projection picture, and then resetting the first display area according to the first resolution, and resetting the second display area according to the second resolution.

[0174] For example, when the display device 200 displays the first projection screen in full screen, the user inputs a multi-window command by pressing the "return" button on the remote control to control the display device 200 to exit the full-screen state and switch back to the multi-way projection interface. In response to the multi-window command, the display device 200 can obtain the resolution of the projection screens of the first terminal device 510 and the second terminal device 520 respectively, that is, for the two horizontal projection screens, the resolution of the first projection screen and the second projection screen are both 1920×1080. At this time, the display device 200 can reset two display areas of size 954×538 according to the resolution of the first projection screen and the second projection screen, and the reset data content of the display area is:

[0175] For the first projection window: FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0); lp1.width=954; lp1.height=538; lp1.leftMargin=0; lp1.topMargin=271; mPlayerView.setLayoutParams(lp1).

[0176] For the second projection window: FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0); lp1.width=954; lp1.height=538; lp1.leftMargin=966; lp1.topMargin=271; mPlayerView.setLayoutParams(lp1).

[0177] In this way, when switching back to the multi-way screen projection interface, the display device 200 can directly complete the switch by resetting the first display area and the second display area, and in the switched back multi-way screen projection interface, the second screen projection window directly displays the projection screen, without the need to re-establish a screen projection connection with the second terminal device 520, nor to display a guide page, thereby improving the interface switching speed.

[0178] In order to improve the interface switching speed, in some embodiments, the display device 200 may also record the display size of each projection window in the multi-projection interface, that is, the display device 200 may respond to the projection instruction for displaying the multi-projection interface, and set the size of the first projection window and the size of the second projection window, that is, the first window size and the second window size, according to the screen resolution and the screen size of the first projection screen and the second projection screen. Then record the first window size and the second window size.

[0179] Among them, when setting the first window size and the second window size, the display device 200 can be dynamically set according to the screen size of the projection screen in the manner provided in the above embodiment to improve the space utilization of the display area. That is, the display device 200 can obtain the first aspect ratio of the first projection screen and the second aspect ratio of the second projection screen. If the first aspect ratio is not equal to the second aspect ratio, the first window size of the first projection window is set according to the first aspect ratio, and the second window size of the second projection window is set according to the second aspect ratio. Then control the display 260 to display the first projection window according to the first window size; and control the display 260 to display the second projection window according to the second window size. Record the first window size and the second window size.

[0180] When setting the first window size and the second window size, the display device can align and display the first projection window and the second projection window in the manner in the above-mentioned embodiments, that is, the window heights of the first window size and the second window size are equal, or the window widths of the first window size and the second window size are equal.

[0181] After recording the first window size and the second window size, the display device 200 can directly obtain the first window size and the second window size when resetting the first display area according to the first resolution and resetting the second display area according to the second resolution, and set the first display area according to the first window size, and set the second display area according to the second window size. The first window size is the window size of the first projection window recorded by the display device when displaying the multi-screen projection interface; the second window size is the window size of the second projection window recorded by the display device when displaying the multi-screen projection interface;

[0182] In some embodiments, after the display device 200 sets the conversion mode parameters for the projection window, the display device 200 can synchronously modify the conversion mode parameters after switching back to the multi-channel projection interface. That is, the display device 200 can respond to the multi-window instruction for exiting the full-screen display of the first projection window and set the conversion mode parameters of the first projection window and the second projection window to a third parameter value, wherein the third parameter value is used to indicate a multi-window display projection window.

[0183] For example, after switching back to the multi-channel screen projection interface, the display device 200 can modify the SwitchMode parameter of the first screen projection window from the first parameter value full to the third parameter value double, and at the same time, modify the SwitchMode parameter of the second screen projection window from the second parameter value Hidden to the third parameter value double, which is used to characterize various logics and display processing during the switching process.

[0184] In some embodiments, after the display device 200 switches to the full-screen state, it can also record the full-screen display duration of the first projection screen in full screen, and judge the full-screen display duration. If the full-screen display duration is greater than or equal to the preset duration threshold, that is, after the user switches to the full-screen display of the first projection screen, the probability of switching back to the multi-channel projection interface for a long time is small. Therefore, in order to save the computing resources of the display device 200, the projection connection channel with the second terminal device 520 can be closed, so as to control the display 260 to stop displaying the second projection screen in the second display area.

[0185] For example, when the user clicks the full-screen button control below the first projection window to control the display device 200 to display the first projection screen in full screen, the display device 200 can record the full-screen display duration of the first projection screen. When the full-screen display duration exceeds 30 minutes, the projection connection channel with the second terminal device 520 can be directly closed so that the display device 200 only displays the first projection screen to reduce the resource occupation of the second projection screen processing process.

[0186] In some embodiments, after the display device 200 sets the first display area of ​​the first projection window according to the screen resolution, it can also detect the display content of the second projection window; if the display content of the second projection window is a guide screen, the second projection window is closed; if the display content of the second projection window is a projection screen, the step of setting the second display area of ​​the second projection window is executed.

[0187] For example, the display device 200 switches the mode from only one client, that is, one channel is used for projecting the screen and the other channel is used for guiding the page, such as Fig.10 At this time, the system is in dual-screen projection mode. After the user selects the Recycleview control of the broadcast control on Client1, Client1 needs to be displayed in full screen. The View management module needs to synchronize the user operation to the mode management module. The mode management module changes the SwitchMode parameter from double to full and synchronizes it to the data management module to update the data, that is:

[0188] Client1 data: Session:SessionClient1,View:SurfaceView / TextureView,MViewPosition:1,Position:(0,272),Width:954,Height:537,SwitchMode:double…

[0189] Switch to the following data: Session: SessionClient1, View: SurfaceView / TextureView, MViewPosition: 1, Position: (0, 0), Width: 1920, Height: 1080, SwitchMode: full…

[0190] At this time, the right projection window displays the guide page, not the Client, so there is no need to perform display processing of extremely small pixels, and it does not involve playback. The View of the Guide on the right can be directly processed as INVISIBLE. That is, after the View management module receives the data information, it sets the display parameters through LayoutParams, including the position from the upper boundary and the left boundary, i.e. the coordinates, and the corresponding display width and height, and then displays the position of Client1 through the setLayoutParams method, as follows:

[0191] FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0);lp1.width=1920;lp1.height=1080;lp1.leftMargin=0;lp1.topMargin=0;mPlayerView.setLayoutParams(lp1);

[0192] Then set the Layout of the right guide page to the INVISIBLE state through the setVisibility method of View, such as Fig.16 shown.

[0193] Similarly, if you need to switch the full-screen display of the first projection screen back to the multi-screen projection interface, you can also trigger the window switching action through specific interaction methods, such as the menu key or any other interaction, such as voice, etc. For example, the default size of the guide page is 954×538, that is, Guide:954×538; the display size of Client1 is 1920×1080, that is, Client1:1920×1080.

[0194] Through the algorithm provided in the above embodiment, the current video and the guide page can be displayed in a horizontal version of 1:1, the full-screen mode data is switched to two-window data, and then the SwitchMode parameter is synchronously changed from full to double. After the View management module receives the data information, the display parameters are set through LayoutParams, including the position from the upper boundary and the left boundary, that is, the coordinates, and the corresponding display width and height. Then, the position of Client1 is displayed through the setLayoutParams method, as follows:

[0195] FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0); lp1.width=1920; lp1.height=1080; lp1.leftMargin=0; lp1.topMargin=0; mPlayerView.setLayoutParams(lp1);

[0196] Then set the Layout of the right guide page to the VISIBLE state through the setVisibility method of View.

[0197] Based on the multi-window switching method provided in the above embodiment, some embodiments of the present application further provide a display device 200, the display device 200 includes a display 260, a communicator 220 and a controller 250, the display 260 is configured to display a multi-screen projection interface, the multi-screen projection interface includes a first screen projection window and a second screen projection window, the first screen projection window is used to display a first screen projection picture of a first terminal device; the second screen projection window is used to display a second screen projection picture of a second terminal device; the communicator 220 is configured to establish a communication connection with the first terminal device and the second terminal device respectively; the controller 250 is configured to execute the following program steps:

[0198] S100: In response to a full screen instruction for displaying the first projection window in full screen, obtaining a screen resolution;

[0199] S200: Setting a first display area of ​​the first projection window according to the screen resolution, and setting a second display area of ​​the second projection window, wherein the number of pixels included in the second display area is less than or equal to a preset pixel threshold;

[0200] S300: Control the display to display the first projection screen in the first display area, and control the display to display the second projection screen in the second display area.

[0201] It can be seen from the above technical solution that the display device provided by the above embodiment can obtain the screen resolution of the display in response to the full-screen instruction for full-screen display of the first projection window when the display device displays the multi-way projection interface. Then set the first display area of ​​the first projection window according to the screen resolution, and set the second display area with a number of pixels less than or equal to the preset pixel threshold. Thereby, the first projection screen is displayed through the first display area, and the second projection screen is displayed through the second display area. When the display device displays the first projection window in full screen, it does not directly close the second projection window, but displays the second projection screen by setting the second display area and using a small number of pixels that are difficult for the user to perceive, so as to realize a multi-way projection interface that can freely and quickly switch between full screen and multi-window according to the usage scenario, thereby improving the user experience.

[0202] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0203] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0205] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.< / doublepushcontrolbean>

Claims

1. A display device, characterized in that: include: A display, configured to display a multi-screen projection interface, wherein the multi-screen projection interface includes a first screen projection window and a second screen projection window, wherein the first screen projection window is used to display a first screen projection picture of a first terminal device; The second screen projection window is used to display a second screen projection screen of the second terminal device; A communicator, configured to establish communication connections with the first terminal device and the second terminal device respectively; The controller is configured as: In response to a full screen instruction for displaying the first projection window in full screen, obtaining a screen resolution; Setting a first display area of ​​the first projection window according to the screen resolution, and setting a second display area of ​​the second projection window, wherein the number of pixels included in the second display area is less than or equal to a preset pixel threshold; Control the display to display the first projection screen in the first display area, and control the display to display the second projection screen in the second display area.

2. The display device according to claim 1, characterized in that The controller is also configured to: In response to a multi-window instruction for exiting full-screen display of a first projection window, obtaining a first resolution of the first projection picture and a second resolution of the second projection picture; resetting the first display area according to the first resolution, and resetting the second display area according to the second resolution; Control the display to display the first projection screen in the first display area after reset, and control the display to display the second projection screen in the second display area after reset.

3. The display device according to claim 2, characterized in that The controller performs resetting of the first display area according to the first resolution and resetting of the second display area according to the second resolution, and is further configured to: Obtain a first window size and a second window size, wherein the first window size is the window size of the first projection window recorded by the display device when the multi-projection interface is displayed; the second window size is the window size of the second projection window recorded by the display device when the multi-projection interface is displayed; The first display area is set according to the first window size, and the second display area is set according to the second window size.

4. The display device according to claim 3, characterized in that The controller is also configured to: In response to a screen projection instruction for displaying a multi-screen projection interface, obtaining a first aspect ratio of the first projection screen and a second aspect ratio of the second projection screen; If the first aspect ratio is not equal to the second aspect ratio, a first window size of the first projection window is set according to the first aspect ratio, and a second window size of the second projection window is set according to the second aspect ratio, and the window heights of the first window size and the second window size are equal, or the window widths of the first window size and the second window size are equal; Controlling the display to display the first screen projection window according to the first window size; and controlling the display to display the second screen projection window according to the second window size; The first window size and the second window size are recorded.

5. The display device according to claim 1, characterized in that After the controller controls the display to display the first projection image in the first display area, the controller is further configured to: Recording the full-screen display duration of the first projection image; If the full-screen display duration is greater than or equal to a preset duration threshold, closing the screen projection connection channel with the second terminal device; Control the display to stop displaying the second projection screen in the second display area.

6. The display device according to claim 1, characterized in that The controller executes setting the first display area of ​​the first projection window according to the screen resolution, and is further configured to: Obtaining the image resolution of the first projection image; Read the screen width and screen height from the screen resolution; Determine a reference size according to the screen resolution and the picture resolution, the reference size being the screen width or the screen height; Based on the reference size, calculating a scaling ratio; Scaling the first projection screen according to the scaling ratio.

7. The display device according to claim 1, characterized in that The controller is also configured to: In response to a full screen instruction for full screen displaying the first projection window, setting a conversion mode parameter of the first projection window to a first parameter value, wherein the first parameter value is used to indicate full screen displaying of the projection window; Setting the conversion mode parameter of the second projection window to a second parameter value, where the second parameter value is used to indicate hiding the projection window; And, in response to a multi-window instruction for exiting full-screen display of the first projection window, the conversion mode parameters of the first projection window and the second projection window are set to a third parameter value, and the third parameter value is used to indicate a multi-window display of the projection window.

8. The display device according to claim 1, characterized in that The controller executes setting a second display area of ​​the second projection window, and is further configured to: Determine a boundary area according to the first display area, where the boundary area is an area outside the first display area in the entire display area of ​​the display, or an area formed by edge pixels of the first display area; According to the preset pixel threshold, setting the minimum number of pixels in the second display area; The second display area is delineated in the boundary area according to the minimum number of pixels.

9. The display device according to claim 1, characterized in that: After the controller sets the first display area of ​​the first projection window according to the screen resolution, the controller is further configured to: Detecting the display content of the second projection window; If the display content of the second screen projection window is a guide screen, close the second screen projection window; If the display content of the second projection window is a projection screen, execute the step of setting a second display area of ​​the second projection window.

10. A multi-window switching method, characterized in that: The display device according to any one of claims 1 to 9, the display device comprising a display, a communicator and a controller, the display being configured to display a multi-way screen projection interface, the multi-way screen projection interface comprising a first screen projection window and a second screen projection window, the first screen projection window being used to display a first screen projection picture of a first terminal device; The second screen projection window is used to display a second screen projection screen of the second terminal device; The communicator is configured to establish communication connections with a first terminal device and a second terminal device respectively; the method comprises: In response to a full screen instruction for displaying the first projection window in full screen, obtaining a screen resolution; Setting a first display area of ​​the first projection window according to the screen resolution, and setting a second display area of ​​the second projection window, wherein the number of pixels included in the second display area is less than or equal to a preset pixel threshold; Control the display to display the first projection screen in the first display area, and control the display to display the second projection screen in the second display area.

Images