Display device and multi-channel screen mirroring control method

Abstract

The present disclosure provides a display device and a multi-channel screen mirroring control method. The method comprises: in response to a screen mirroring connection instruction sent by a first terminal device, creating a first screen mirroring window and releasing a first guide window; acquiring a first screen mirroring conversation sent by the first terminal device, and measuring a screen resolution and a reference resolution of a guide page; and calculating a first display area and a second display area on the basis of the screen resolution, a first resolution, and the reference resolution, and displaying the first screen mirroring window in the first display area, and displaying a second guide window in the second display area. The height of the first display area is equal to that of the second display area. By means of the process, when a plurality of windows are displayed at the same height, the display area of each window is calculated on the basis of a screen resolution, the resolution of a screen mirroring picture, and the resolution of a guide page, so that the screen mirroring picture can be displayed in a multi-channel screen mirroring interface to the maximum extent, improving the user experience.

Description

Display device and multi-channel screen projection control method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese applications filed on November 16, 2023, with application numbers 202311529991.8; filed on November 16, 2023, with application numbers 202311533170.1; filed on November 16, 2023, with application numbers 202311530019.2; filed on November 16, 2023, with application numbers 202311530008.4; and filed on November 16, 2023, with application numbers 202311533185.8, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the technical field of display devices, and in particular to a display device and a multi-channel screen projection control method. Background Art

[0004] Display devices are smart devices that can present user interfaces and support user interaction. They can establish communication connections with mobile devices and other end-devices, obtain projection data from these end-devices based on specific projection protocols, and then display the projection image of the end-device based on the projection data.

[0005] Some display devices can establish simultaneous communication connections with multiple devices and receive projection data from multiple devices, creating a multi-channel projection scenario. However, in a multi-channel projection scenario, the display device can only display the projection window of a single device. If projection windows are created for multiple devices simultaneously, since the projection windows cannot communicate directly with each other, the display device may experience interference between the projection windows.

[0006] Summary of the Invention

[0007] In a first aspect, an embodiment of the present disclosure provides a display device, which may include: a display, which may be configured to display a multi-screen projection interface, and the multi-screen projection interface may include a first guide window and a second guide window, and the first guide window and the second guide window may be used to display a guide page; the first guide window and the second guide window are the same as the root node of the multi-screen projection interface; a user input interface, which may be configured to receive instructions from a user; a communication device, which may be configured to establish a communication connection with a terminal device; a memory, which may be configured to store computer instructions and data associated with the display device; at least one processor, which may be connected to the display, the user input interface, the communication device and the memory, and is configured to execute computer instructions so that the display device executes: in response to the projection connection instruction sent by the first terminal device, creating a first projection window , and release the first guide window, and keep the second guide window displayed in the multi-way screen projection interface; the first screen projection window is used to display the first screen projection screen of the first terminal device, and the first screen projection window is the same as the root node of the multi-way screen projection interface; obtain the first screen projection session sent by the first terminal device, and the first screen projection session includes the first resolution of the first screen projection screen; detect the screen resolution and the benchmark resolution of the guide page; calculate the window display area according to the screen resolution, the first resolution and the benchmark resolution, the window display area includes a first display area and a second display area, and the height of the first display area is equal to that of the second display area; control the display to display the first screen projection window in the first display area, and display the second guide window in the second display area.

[0008] In a second aspect, an embodiment of the present disclosure provides a multi-screen projection control method, which may include: creating a first projection window in response to a projection connection instruction sent by a first terminal device, releasing the first guide window, and retaining the display of the second guide window in the multi-screen projection interface; the first projection window is used to display the first projection screen of the first terminal device, and the first projection window is the same as the root node of the multi-screen projection interface; the multi-screen projection interface includes a first guide window and a second guide window, and the first guide window and the second guide window are used to display the guide page; the first guide window and the second guide window are the same as the root node of the multi-screen projection interface; obtaining the first projection session sent by the first terminal device; the first projection session includes the first resolution of the first projection screen; detecting the screen resolution and the base resolution of the guide page; calculating the window display area according to the screen resolution, the first resolution and the base resolution; the window display area includes the first display area and the second display area, and the height of the first display area and the second display area are equal; displaying the first projection window in the first display area, and controlling the display to display the second guide window in the second display area. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG1 is a schematic diagram of a usage scenario of a display device according to some embodiments;

[0010] FIG2 is a block diagram of a hardware configuration of a display device according to some embodiments;

[0011] FIG3 is a block diagram of a hardware configuration of a control device according to some embodiments;

[0012] FIG4 is a block diagram of a software configuration of a display device according to some embodiments;

[0013] FIG5 is a schematic diagram of mirrored screen projection according to some embodiments;

[0014] FIG6 is a schematic diagram of push screen projection according to some embodiments;

[0015] FIG7 is a schematic diagram of a multi-screen projection interface according to some embodiments;

[0016] FIG8 is a schematic diagram of functional modes of a display device according to some embodiments;

[0017] FIG9 is a schematic diagram of a guide page according to some embodiments;

[0018] FIG10 is a schematic diagram of a multi-channel screen projection control process executed by a display device according to some embodiments;

[0019] FIG11 is a schematic diagram illustrating the display effect of a multi-screen projection interface when one screen is projected according to some embodiments;

[0020] FIG12 is a schematic diagram of a process of displaying a second projection window according to some embodiments;

[0021] FIG13 is a schematic diagram illustrating the display effect of a multi-screen projection interface during two-screen projection according to some embodiments;

[0022] FIG14 is another schematic diagram of the display effect of the multi-screen projection interface during two-screen projection according to some embodiments;

[0023] FIG15 is a schematic diagram showing the display effect of a floating display control function control according to some embodiments;

[0024] FIG16 is a schematic diagram illustrating the display effect of displaying a playback control function control in an edge area according to some embodiments;

[0025] FIG17 is a schematic diagram illustrating the display effect after compressing the projection window according to some embodiments;

[0026] FIG18 is another schematic diagram of a display device executing a multi-channel screen projection control process according to some embodiments;

[0027] FIG19 is a flow chart illustrating a method for setting replacement mode parameters according to some embodiments;

[0028] FIG20 is a schematic diagram showing the display effect of locking a control according to some embodiments;

[0029] FIG21 is a schematic diagram showing the display effect of a locked control in an unlocked state according to some embodiments;

[0030] FIG22 is a schematic diagram illustrating the display effect of a third projection window replacing the second projection window according to some embodiments;

[0031] FIG23 is a schematic diagram of an interface display effect in the second mode according to some embodiments;

[0032] FIG24 is a schematic diagram illustrating the display effect of a third projection window replacing the first projection window according to some embodiments;

[0033] FIG25 is another schematic diagram of the display effect of the multi-screen projection interface when one screen is projected according to some embodiments;

[0034] FIG26 is another schematic diagram of a display device executing a multi-channel screen projection control process according to some embodiments;

[0035] FIG27 is a schematic diagram showing the effects of six button controls according to some embodiments;

[0036] FIG28 is a schematic diagram illustrating the effect of dynamically adjusting the number of button controls according to some embodiments;

[0037] FIG29 is a schematic diagram showing the effects of three button controls according to some embodiments;

[0038] FIG30 is a schematic diagram showing the effects of two button controls according to some embodiments;

[0039] FIG31 is a schematic diagram showing the effect of a button control according to some embodiments;

[0040] FIG32 is a schematic diagram of a display device switching to a full-screen effect according to some embodiments;

[0041] FIG33 is another schematic diagram of a display device executing a multi-channel screen projection control process according to some embodiments;

[0042] FIG34 is a schematic diagram of a full-screen display effect according to some embodiments;

[0043] FIG35 is a flow chart of sound playback control of a display device according to some embodiments;

[0044] FIG36 is a schematic diagram illustrating a display effect of a sound view according to some embodiments;

[0045] FIG37 is a schematic diagram illustrating the effect of dynamically adjusting a sound view according to some embodiments;

[0046] FIG38 is a schematic diagram illustrating a display effect when one screen projection path is disconnected according to some embodiments;

[0047] FIG39 is a flowchart illustrating sound switching when screen projection is disconnected according to some embodiments;

[0048] FIG40 is a schematic diagram illustrating the effect of locking the sound mode of the projection window according to some embodiments;

[0049] FIG41 is another schematic diagram of a display device switching to a full-screen effect according to some embodiments. DETAILED DESCRIPTION

[0050] Figure 1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments. As shown in Figure 1, a user can operate a display device 200 through touch operation, a mobile terminal 300, and a control device 100. The control device 100 may be a remote control, a stylus, or the like.

[0051] In some embodiments, the mobile terminal 300 can install software applications on the display device 200 and connect to the display device 200 through a network communication protocol to achieve one-to-one control operations and data communication. It is also possible to transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronous display.

[0052] 1 , the display device 200 may also communicate data with the server 400 via various communication methods. For example, the display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), or other networks.

[0053] In addition to providing broadcast reception television functions, the display device 200 may also provide intelligent network television functions with computer support functions, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0054] Figure 2 is a hardware configuration block diagram of the display device 200 shown in Figure 1. As shown in Figure 2, the display device 200 may include at least one of a tuner and demodulator 210, a communication device 220, a detector 230, an external device interface 240, a processor 250, a display 260, an audio output interface 270, a memory, a power supply, and a user input interface.

[0055] In some embodiments, detector 230 can be used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, such as a sensor for collecting ambient light intensity; or detector 230 may include an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 may include a sound collector, such as a microphone, for receiving external sounds.

[0056] In some embodiments, the display 260 may include a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals, and displaying video content, image content, and a menu control interface, as well as a user control UI interface.

[0057] In some embodiments, the communication device 220 is a component used to communicate with an external device or server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 depending on the supported communication methods. For example, if the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including WiFi functionality. If the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including Bluetooth functionality.

[0058] The communication device 220 can establish a communication connection between the display device 200 and an external device or server 400 via a wireless or wired connection. A wired connection can connect the display device 200 to an external device via a data cable, an interface, or other components. A wireless connection can connect the display device 200 to an external device via a wireless signal or wireless network. The display device 200 can establish a connection with an external device directly or indirectly through a gateway, router, or connection device.

[0059] In some embodiments, the display device 200 may also support establishing communication connections with multiple external devices at the same time. Multiple external devices may be connected through the same. For example, the external device is a terminal device 500 such as a mobile phone or a tablet computer. The first terminal device 510 and the second terminal device 520 may establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices may also be connected to the display device 200 through different types of connection methods. For example, the first terminal device 510 is connected to the display device 200 via a wireless local area network; the second terminal device 520 is connected to the display device 200 via Bluetooth.

[0060] To achieve communication between the display device 200 and the terminal device 500, the display device 200 and the terminal device 500 are each equipped with a communication device that supports the same type of communication. For example, if the display device 200 is equipped with a communication device 220 including a WiFi module, the terminal device 500 should also be equipped with a communication device including a WiFi module. Furthermore, while the same type of communication device is provided, a specific communication transmission protocol is required for data transmission between the display device 200 and the terminal device 500. Examples include the WiFi protocol, Bluetooth connection protocol, ZigBee protocol, and NFC protocol.

[0061] In some embodiments, the processor 250 may include a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output. The processor 250 may control the operation of the display device and respond to user operations through various software control programs stored in the memory. The processor 250 may control the overall operation of the display device 200.

[0062] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

[0063] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0064] In some embodiments, the user input interface 280 may be an interface operable to receive control input.

[0065] Figure 3 is a block diagram of the hardware configuration of the control device shown in Figure 1. As shown in Figure 3, the control device 100 may include a processor 110, a communication device 130, a user input / output interface, a memory, and a power supply.

[0066] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0067] In some embodiments, the control device 100 may be a smart device. For example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0068] In some embodiments, as shown in FIG. 1 , the mobile terminal 300 or other intelligent electronic device can perform similar functions to the control device 100 after installing an application for controlling the display device 200 .

[0069] The processor 110 may include a RAM 113 and a ROM 114, a communication device 130, and a communication bus. The processor 110 is used to control the operation and operation of the control device 100, as well as the communication and coordination between internal components and the external and internal data processing functions.

[0070] Under the control of the processor 110, the communication device 130 communicates control signals and data signals with the display device 200. The communication device 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, or other near field communication modules.

[0071] The user input / output interface 140 includes at least one of a microphone 141 , a touch panel 142 , a sensor 143 , a button 144 and other input interfaces.

[0072] In some embodiments, the control device 100 includes at least one of a communication device 130 and an input / output interface 140. The control device 100 is configured with the communication device 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via the WiFi protocol, Bluetooth protocol, or NFC protocol and send them to the display device 200.

[0073] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the processor. The memory 190 can store various control signal instructions input by the user.

[0074] The power supply 180 is used to provide operating power support for various components of the control device 100 under the control of the processor.

[0075] Figure 4 is a software configuration block diagram of a display device according to some embodiments. As shown in Figure 4, the built-in system of the display device can be divided into four layers, from top to bottom, namely the application layer (referred to as the "application layer"), the application framework layer (referred to as the "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as the "system runtime library layer"), and the kernel layer.

[0076] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0077] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0078] In some embodiments, the application framework layer in the embodiments of the present disclosure includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to the system location service; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0079] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0080] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0081] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG4 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0082] After establishing a communication connection with the terminal device 500, the display device 200 can establish a screen projection connection channel and transmit projection data based on the projection connection so that the display device 200 can display the projection image of the terminal device 500. In some embodiments, to establish a screen projection connection relationship, the user can issue a screen projection connection request through the terminal device 500 and send the screen projection connection request to the display device 200 via the WiFi network. The display device 200 then completes the transmission protocol configuration based on the screen projection connection request, thereby establishing a transmission channel for projection data between the display device 200 and the terminal device 500.

[0083] The display device 200 can also achieve a screen projection connection with the terminal device 500 through different connection methods. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, a screen projection connection can be established based on the WiFi network. For another example, when both the display device 200 and the terminal device 500 are provided with NFC (Near Field Communication) components, a screen projection connection relationship can be established through the NFC component. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the terminal device 500 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network and other connection methods that can be associated with those skilled in the art based on the connection methods provided in the above embodiments.

[0084] After establishing the screen projection connection, the display device 200 can receive the screen projection related data from the terminal device 500 through the screen projection data channel, thereby playing the screen projection related content on the display device 200. Under different screen projection data transmission protocols, the screen projection data obtained by the display device 200 is in different forms. For example, the display device 200 and the terminal device 500 can establish a screen projection connection through the Digital Living Network Alliance (DLNA) protocol, Miracast protocol, AirPlay push protocol, WiDi screen projection, NFC mirror screen projection, LeTV screen projection, and other protocols.

[0085] The DLNA protocol is a protocol used to share media resources between devices. This protocol uses network technology (wired or wireless) to interconnect various devices and exchange data through standard protocols, allowing media resources to be shared between devices. The DLNA protocol is based on the UPnP protocol, enabling devices to discover and control each other, achieving seamless transmission and sharing of media resources. The DLNA protocol supports a variety of devices, including TVs, speakers, mobile phones, computers, and more, allowing users to seamlessly switch and share media content between different devices.

[0086] The Miracast protocol is a wireless display standard developed by the Wi-Fi Alliance and based on Wi-Fi Direct. The Miracast screen projection protocol is compatible with Android devices, allowing Android devices to share video images through Miracast. Smart devices running Android 5.0 and higher have the Miracast protocol built-in, allowing direct wireless screen projection without installing any software, using the Wi-Fi P2P function between the projection device and the receiving device's Wi-Fi network card. Miracast also supports simultaneous audio and video transmission.

[0087] AirPlay is a wireless playback technology introduced by Apple, suitable for wireless screen projection on iOS and Mac systems. Apple devices have built-in AirPlay projection functionality. When used with a screen projector that supports the AirPlay server service, the display is clear and stable. The protocol also includes a built-in projection code function. However, the Apple device and screen projector must be on the same local area network (LAN segment) and cannot be used across different network segments or VLANs.

[0088] WiDi is an Intel-developed wireless streaming technology based on Wi-Fi Direct. Windows 8 / 10 / 11 laptops natively support WiDi, allowing you to cast your screen directly without installing any software. It also supports USB reverse control.

[0089] In some embodiments, after the display device 200 establishes a projection connection with the terminal device 500, projection data can be transmitted through mirror mode and push mode. Among them, mirror mode means that the display device 200 synchronously displays the screen content displayed by the terminal device 500. For example, as shown in Figure 5, the display device displays the main interface of the terminal device 500, and when the user performs an interactive operation on the terminal device 500, causing the content of the main interface to change, the content displayed by the display device also changes. To this end, the terminal device 500 can generate a projection data stream in the form of a video based on the display content, and send the projection data stream to the display device for display.

[0090] Push mode means that the display device accesses the network link pushed by the terminal device 500 and obtains the corresponding projection image. For example, as shown in Figure 6, when the terminal device 500 is playing video media, a projection control is provided in the upper right corner of the playback interface. After the user clicks the projection control and selects the display device as the device to display the projection image, the terminal device 500 can send the URL address of the currently playing video media to the display device. After receiving the URL address, the display device accesses the URL address to obtain the video media data and plays the video media.

[0091] Among the screen projection protocols supported by the display device and the terminal device 500, some protocols are based on the mirror mode, and some protocols are based on the push mode. For example, under the DLNA protocol, the display device can obtain the link address of a resource in the terminal device 500 through the screen projection data channel, and obtain the media content data by accessing the link address, thereby playing the media data. At the same time, the display device can also obtain the control instructions sent by the terminal device 500 through the screen projection data channel when playing the media data, such as fast forward, pause, stop screen projection, etc. For example, after establishing a connection through the communication protocol, the terminal device 500 such as a mobile phone and the display device will negotiate the device functions and network conditions to select a suitable audio and video transmission format. After establishing the online protocol (Session) for the audio and video stream, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the audio and video stream.

[0092] Under other screen projection connection protocols such as Miracast, Airplay, NFC mirroring, and LeTV, the display device can obtain the video data stream directly from the terminal device 500. The video data stream can be the video data stream corresponding to the final display screen of the display unit of the terminal device 500, that is, the video data stream includes the media content screen, UI interface, and other video screen content that can be displayed in the display unit of the terminal device 500. The video data stream can also be only the video data that can be obtained in the terminal device 500, such as the media video data being played. At this time, the terminal device 500 can be used as a signal source for the display device, and the display device can continuously obtain video data through the screen projection data channel and display it in real time on the display, thereby achieving a synchronous display effect with the terminal device 500.

[0093] It should be noted that the above-mentioned screen projection protocol can not only obtain the video data stream in the terminal device 500, but also obtain the audio data stream in the terminal device 500, so that the video data and audio data can be sent to the display device synchronously, so that the display device can play the video and audio synchronously.

[0094] The terminal device 500 can obtain different projection data according to different projection protocols. For example, for system applications or direct projection at the system level, the terminal device 500 can use Miracast projection, Airplay projection or WiDi projection protocol to establish a projection data channel. Since the above-mentioned Miracast projection protocol and Airplay projection protocol require system-level application permissions, the Miracast projection protocol and Airplay projection protocol are only applicable to system applications, or the display device and terminal device 500 of the same manufacturer directly establish a projection connection through the built-in rules of the operating system.

[0095] The display device can establish a screen projection connection with the terminal device 500 based on any of the screen projection methods provided in the above embodiments. In some embodiments, the display device can also establish a screen projection connection relationship with multiple communication terminals through one or more of the above screen projection methods. For example, the display device can simultaneously establish a screen projection connection with the first terminal device 510 and the second terminal device 520, and divide the display area in the display screen for respectively displaying the screen projection screen shared by the first terminal device 510 and the second terminal device 520.

[0096] In order to be able to display the projection screen shared by multiple terminal devices 500 on the display device, in some embodiments, when the display device establishes a projection connection with multiple terminal devices 500, the display device can display a multi-way projection interface, which can include multiple projection windows. The projection window is used to display the projection screen of the corresponding terminal device 500 under a projection connection channel. For example, as shown in Figure 7, the multi-way projection interface can include two projection windows, namely the first projection window and the second projection window. Among them, the first projection window is used to display the projection screen of the first terminal device 510; the second projection window is used to display the projection screen of the second terminal device 520.

[0097] It should be noted that the projection window in the multi-way projection interface may not be strongly bound to the terminal device 500. That is, in some embodiments, the display content of the projection window may be determined according to the order in which the terminal device 500 establishes a projection connection with the display device. For example, when the first terminal device 510 establishes a projection connection with the display device before the second terminal device 520, the first projection window on the left side of the multi-way projection interface displays the projection screen of the first terminal device 510. When the first terminal device 510 establishes a projection connection with the display device later than the second terminal device 520, the first projection window on the left side of the multi-way projection interface displays the projection screen of the second terminal device 520.

[0098] In order to obtain a better picture display effect and improve the coordination of the multi-channel projection interface, in some embodiments, the display device can also set the layout of the projection window. To this end, as shown in Figure 8, the display device can have multiple functional modules built into the same application, such as a multi-channel projection application. The multi-channel projection application includes: data management module, message processing module, service management module, algorithm processing module, View management module, application interaction processing module, location management module, replacement management module, broadcast control management module, time management module, focus management module and projection management module, etc.

[0099] Among them, the data management module mainly manages all message data involved in the current template and updates the data information synchronously. The message processing module is mainly responsible for transmitting related messages between different modules, such as transmitting data changes to the View layer for refresh display, or synchronizing View changes to the data management module for change processing of broadcast control data, etc. The service management module is responsible for opening and closing services of the entire application and some customized processing, such as starting specific services in specific scenarios. The algorithm processing module is responsible for all layout position calculations of the entire application. The View management module is mainly responsible for all UI displays of the entire template application, receiving data to refresh the View display.

[0100] The application interaction processing module is responsible for processing interactions with external applications. For example, any entry that starts the current template application needs to interact with this module, and different customized management is performed according to the differences in the startup entry. The position management module is responsible for the rules for displaying the projection position in the left and right windows, such as on the left or right. The replacement management module is responsible for the replacement rules when the number of projection paths exceeds the maximum display number, such as how to replace when there is focus or no focus. The broadcast control management module is responsible for the broadcast control processing of different media types in the projection, such as fast forward and rewind, play pause, etc. The time management module is responsible for the focus display timing processing in the projection, whether it is the large frame focus or the broadcast control position focus. The focus management module is responsible for the focus display and button processing in the entire application. The projection management module is responsible for the projection protocol interaction and the client list management and the status information processing of the window display content.

[0101] It should be noted that the multi-channel screen projection interface can realize screen display based on the multi-channel screen projection viewing application. The multi-channel screen projection viewing application is an application that can display multi-protocol multi-channel mirror projection and push content, and is an upper-layer application within the display device. The framework layer of the display device can provide a Java interface for docking with hardware, etc. The multi-channel screen projection viewing application uses the Java interface of the Framework layer to achieve docking with the lower layer and obtain information. Other National, Runtime, HAL, and Linux kernel layers are all standard layers of the Android architecture, and each layer interacts according to the business process.

[0102] The display device first starts the application through the service management module and performs some initial preparations, such as starting the monitoring mirror or pushing the corresponding protocol service; the View management module initializes the View displayed by the application layer; the data management module creates some data for the data processing module; the algorithm processing module performs algorithmic layout calculations on the projection content, and the position management module is responsible for positioning the left and right positions of the projection content.

[0103] The service management module can start the screen projection application and register the Session during the establishment of the screen projection connection; Usession Connection Listener, Usession Prepare Listener, Usession Display Listener and Usession Event Listener listening. Among them, Session represents a conversation between the client and the server, which can save the client's information. Usession Connection Listener is used to monitor the connection and disconnection events of the Session. Usession Prepare Listener interacts with the screen projection preparation of the Client. Usession Display Listener monitors the playback information and status changes as well as the status changes of the Client. Usession Event Listener processes information such as Client control.

[0104] After the View is initialized, the display device displays a default guide window when entering the application. The guide window is used to display the guide page, which displays some device information, such as the device name of the current display device and its corresponding network information. The guide page includes multiple system-standard TextView and ImageView controls. The display device can display the guide page in equal proportions on the left and right sides by controlling the TextView and ImageView controls and their corresponding Layout, as shown in Figure 9. The guide page includes a baseline resolution. When displaying the guide page through the guide window, the display device needs to scale the guide page according to the baseline resolution.

[0105] In some embodiments, the guide window and the multi-screen projection interface share a root, that is, the guide window and the root node of the multi-screen projection interface are the same. For example, after the user runs the multi-screen projection application in the display device, the display device controls the display to display the multi-screen projection interface in full screen, and displays the guide window in the corresponding position of the multi-screen projection interface. Among them, the multi-screen projection interface is the homepage interface of the multi-screen projection application, and the guide window and the homepage interface of the multi-screen projection application use the same root.

[0106] After displaying the guide window, the display device can monitor and receive the screen projection connection instruction sent by the terminal device 500, and establish a screen projection connection channel according to the screen projection connection instruction, thereby displaying the screen projection image of the terminal device in the screen projection window.

[0107] Some embodiments of the present disclosure provide a display device, including a display, a user input interface, a communication device, a memory, and at least one processor. The display is configured to display a multi-channel screen projection interface, and the multi-channel screen projection interface includes a first guide window and a second guide window; wherein the first guide window and the second guide window are used to display a guide page, and the first guide window, the second guide window and the root node of the multi-channel screen projection interface are the same; the communication device is configured to establish a communication connection with the terminal device 500. In a multi-channel screen projection scenario, the display device can calculate the display area of ​​each window according to the screen resolution, the projection screen and the resolution of the guide page, on the premise of displaying the projection windows of multiple terminal devices at the same height, so that the projection screen can be displayed in the multi-channel screen projection interface to the greatest extent, thereby improving the user experience. Specifically, as shown in Figure 10, it includes the following steps:

[0108] S1001: In response to the screen projection connection instruction sent by the first terminal device, create a first screen projection window, release the first guide window, and keep the second guide window displayed in the multi-screen projection interface.

[0109] After the display device displays the default guide window, the user can establish a communication connection between the terminal device 500 and the display device based on the prompt information in the guide window. When the user sends a screen projection connection instruction to the display device through the first terminal device 510, the display device responds to the screen projection connection instruction and creates a first screen projection window. Among them, the first screen projection window is used to display the first screen projection picture, and the first screen projection window is the same as the root node of the multi-screen projection interface, that is, the first screen projection window also uses the same root as the multi-screen projection interface.

[0110] For example, the screen projection management module of the display device is always ready to access a new Session. When the first terminal device 510Client1 is ready to start the connection, the screen projection management module listens to the onConnect connection of the new Session through UsessionConnectionListener, and executes the creation of the first screen projection window. The first screen projection window can include the playback control SurfaceView / TextureView, and the display device can set the first projection screen to the corresponding position of SurfaceView / TextureView.

[0111] When creating the first projection window, the display device also releases the first guide window so that it is not displayed in the multi-screen projection interface. At the same time, the second guide window remains displayed in the multi-screen projection interface. When releasing the first guide window, the display device may release the related resources of the window object corresponding to the first guide window.

[0112] In order to facilitate the management of the display orientation of the projection window, in some embodiments, when creating the first projection window, the display device also sets the orientation identifier of the first projection window through the position management module. The orientation identifier is used to characterize the display orientation of the projection window, and the display device can display the first projection window according to the orientation identifier. When setting the orientation identifier, the display device can set the orientation identifier of the projection window according to the projection order of the terminal device 500 according to the default priority order. The orientation identifier can be different numbers or English characters. For example, the display orientation of the projection window includes the left and right sides, and the default priority order is left>right. The orientation identifier of the left side is 1, and the orientation identifier of the right side is 2; After receiving the projection connection instruction from the first terminal device 510, the position management module sets the orientation identifier of the first projection window to 1 according to the default priority order, that is, sets mViewPosition:1.

[0113] To facilitate management of the first projection session between the first projection window and the first terminal device 510, in some embodiments, the display device further includes a management object, MultiCastPlayView. The MultiCastPlayView management object can be associated with information such as Session, SurfaceView / TextureView, and mViewPosition. For example, for the first terminal device 510, the data corresponding to its Client1 can be represented as: Session: SessionClient1, View: SurfaceView / TextureView, MViewPosition: 1.

[0114] S1002: Obtain a first screen projection session sent by a first terminal device.

[0115] After completing the creation of the first projection window, the display device obtains the first projection session sent by the first terminal device 510 to obtain the projection data information sent by the first terminal device 510 through the first projection session. The projection data information includes information associated with the projection connection between the first terminal device and the display device, such as the resolution of the first projection screen and other attributes, that is, the first projection session includes the first resolution of the first projection screen. By obtaining the first projection session SessionClient1, the display device can obtain the first resolution of the first projection screen, and then calculate the display area of ​​the first projection window based on the first resolution.

[0116] For example, after the display device prepares the first projection window view, it can notify the first terminal device Client1 through UsessionPrepareListener to record or send the video stream. When the terminal device 500 is ready to send video information, the display device learns through onFirstFrame() of UsessionDisplayListener that the first projection screen will be played. Therefore, it obtains the first resolution of the current Client1 corresponding to the first projection screen from SessionClient1, that is, the width and height of the video stream. It then determines the direction of the current video stream and uses an algorithm to calculate the display position and size of the current view.

[0117] S1003: Detecting the screen resolution and the base resolution of the guide page.

[0118] After the display device releases the first guide window and creates the first projection window, the windows displayed in the multi-screen projection interface include the first projection window and the second guide window. Therefore, after the display device obtains the first resolution of the first projection screen in the first projection session, it detects the screen resolution of the display device and the baseline resolution of the guide page, and determines the display areas of the first projection window and the second guide window based on the screen resolution, the first resolution of the first projection screen, and the baseline resolution of the guide page.

[0119] The screen resolution and the baseline resolution of the boot page are pre-stored information in the display device. The display device can detect the screen resolution of the display device and the baseline resolution of the boot page in a designated storage location. For example, the baseline resolution of the boot page can be stored in the layout file of the multi-screen projection application. The display device can detect the layout information of the boot page in the layout file to obtain the baseline resolution.

[0120] S1004: Calculate the window display area according to the screen resolution, the first resolution, and the reference resolution.

[0121] After obtaining the screen resolution, the first resolution, and the reference resolution, the display device can calculate the display area of ​​each window according to the above resolutions. The window display area includes a first display area and a second display area. The first display area is used to display the first projection window, and the second display area is used to display the second guide window. To improve the user's visual experience, the first display area and the second display area are equal in height so that each window can maintain a consistent height and avoid the problem of users repeatedly switching visual heights when viewing multiple windows simultaneously.

[0122] In some embodiments, when the display device calculates the window display area based on the screen resolution, the first resolution, and the reference resolution, it reads the horizontal pixel value and the vertical pixel value of the screen resolution. It then calculates the first aspect ratio based on the ratio of the horizontal pixel value to the vertical pixel value in the first resolution, and calculates the second aspect ratio based on the ratio of the resolution to the vertical pixel value in the reference resolution. For example, the screen resolution of the display device is 1920×1080, the first resolution of the first projection screen is 1920×1080, and the reference resolution of the guide page is 954×538. The horizontal pixel value in the screen resolution is 1920, and the vertical pixel value is 1080; the horizontal pixel value in the first resolution is 1920, and the vertical pixel value is 1080. The ratio of the two is calculated to obtain a first aspect ratio of 16 / 9; the horizontal pixel value in the reference resolution is 954, and the vertical pixel value is 1080. The ratio of the two is calculated to obtain a second aspect ratio of 477 / 269, which is close to 16 / 9.

[0123] Then, set the first and second layout parameters. The first layout parameters include a first layout width and a first layout height, and the second layout parameters include a second layout width and a second layout height. The first layout height is equal to the second layout height, the sum of the first layout width and the second layout width is equal to the horizontal pixel value of the screen resolution, the first layout height and the second layout height are less than or equal to the vertical pixel value of the screen resolution, the ratio of the first layout width to the first layout height is equal to the first aspect ratio, and the ratio of the second layout width to the second layout height is equal to the second aspect ratio. Then, generate the first display area according to the first layout parameters, and generate the second display area according to the second layout parameters.

[0124] To improve the standardization of window layouts, in some embodiments, the multi-screen projection interface of a display device includes a margin area. This margin area is the display area of ​​the multi-screen projection interface, excluding the display window and the guide window, and can be referred to as a white space. Furthermore, the margin area can also be used to display playback control controls or prompts. Therefore, the sum of the layout height and the margin area height is set to equal the vertical pixel value of the screen.

[0125] In some embodiments, when setting the first layout parameters, the display device reads a first preset height of an edge area. The edge area includes the display area of ​​the multi-screen projection interface excluding the first display window and the second guide window. The sum of the first layout height and the first preset height is then set equal to the vertical pixel value of the screen resolution to generate the first layout height based on the first preset height.

[0126] Taking a display device with a screen resolution of 1920×1080 as an example, when setting the first and second layout parameters, the display device sets Client1's width1 and height1 based on the first and base resolutions; and Guide (guide page): width_guide and height_guide. Width1 and height1 correspond to the first layout width and height, respectively; width_guide and height_guide correspond to the second layout width and height, respectively. The height of the first projection window from the top edge of the screen, or the first preset height, is set to top. Since height1 and height_guide are the same height, the same variable height can be used instead. The first aspect ratio (ratio1) of the first projection window and the second aspect ratio (ratio_guide) of the guide page are both known constants. The values ​​of the width1, top, height, and width_guide parameters can be calculated using the following combined calculations: width1 + width_guide = 1920; top * 2 + height = 1080; width1 / height = ratio1; width_guide / height = ratio_guide.

[0127] In order to facilitate the distinction between each window, in some embodiments, a blank area is provided between the first projection window and the second guide window, and / or a blank area is provided between the first projection window and the second guide window and the left and right edges of the screen. When setting the layout parameters, the display device also sets the sum of the layout height and the width of the blank area to be equal to the horizontal pixel value of the screen. For example, the width of the projection screen from the left edge is left, the width of the guide page from the right edge is right, and the blank space between the first projection window and the guide window is 12 pixels. If the screen resolution of the display device is 1920×1080, set width1+width_guide+left+right+12=1920.

[0128] S1005: Control the display to display a first projection window in the first display area, and control the display to display a second guide window in the second display area.

[0129] After calculating the first and second display areas, the display device displays the first projection window in the first display area and the second display window in the second display area. By calculating the first and second display areas using the aforementioned algorithm and displaying the first projection window in the first display area and the second guide window in the second display area, the first projection window can be displayed to the greatest extent possible while maintaining the same height for both windows, thus standardizing the layout of each window and improving the user's visual experience.

[0130] In some embodiments, when the display device displays the first projection window in the first display area, the display device generates the layout position coordinates of the first projection window according to the first layout parameters, records the first layout parameters and the layout position coordinates, and then draws the first projection window according to the first layout parameters and the layout position coordinates.

[0131] Similarly, in some embodiments, when the display device displays the second guide window in the second display area, it generates the layout position coordinates of the second guide window based on the second layout parameters, records the second layout parameters and the layout position coordinates, and then draws the second projection window based on the second layout parameters and the layout position coordinates.

[0132] For example, the screen resolution of the display device is 1920×1080, the first resolution of the first projection screen is 512×1088, the baseline resolution of the guide page is 954×538, and the width of the blank area between the first projection window and the second guide window is 12 pixels. Based on the width1+width_guide=1920;top*2+height=1080;width1 / height=ratio1;width_guide / height=ratio_guide algorithm, the first layout parameters and the second layout parameters are calculated, where top=116. The first layout height in the first layout parameters is calculated to be 848 and the first layout width is 400. According to the above layout parameters, the upper left corner coordinates of the first projection window (0, 116) and the upper left corner coordinates of the guide window (412, 116) are obtained, and then the above layout parameters and upper left corner coordinates are stored in the data management module, and the above data is transmitted to the View management module through the message processing module, so that the View management module draws the first projection window and the second guide window according to the layout parameters and the upper left corner coordinates.

[0133] That is, the display device's data management module can organize and store layout parameters and layout position coordinates before transmitting the data, including: Session:SessionClient1, View:SurfaceView / TextureView, MViewPosition:1, Position:(0,116), Width:400, Height:848; Session:View:layout, MViewPosition:Position:(412,116), Width:1508, Height:848. After receiving the data information, the display device's View management module uses LayoutParams to set the display parameters, including the upper-left corner coordinates and the corresponding layout width and height, and then displays the position using the setLayoutParams method. That is: FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width, height, 0); lp1.width=400; lp1.height=848; lp1.leftMargin=0; lp1.topMargin=116; mPlayerView.setLayoutParams(lp1);

[0134] FrameLayout.LayoutParamslp1=newFrameLayout.LayoutParams(width,height,0);lp1.width=1508;lp1.height=848;lp1.leftMargin=412;lp1.topMargin=116;mPlayerView.setLayoutParams(lp1).

[0135] By setting the display position of SurfaceView / TextureView and the display position of the guide page Layout respectively through the View management module, you can obtain the display effect of the multi-screen projection interface as shown in Figure 11.

[0136] After displaying the multi-screen projection interface shown in Figure 11, the display device can also receive a screen projection connection instruction sent by other terminal devices 500 and establish a screen projection connection relationship with the terminal device. For example, as shown in Figure 12, the display device can also create a second screen projection window and release the second guide window in response to the screen projection connection instruction sent by the second terminal device 520. The second screen projection window is used to display the second screen projection screen of the second terminal device 520, and the second screen projection window is the same as the root node of the multi-screen projection interface. Then obtain the second screen projection session sent by the second terminal device 520. The second screen projection session includes the second resolution of the second screen projection screen. Then detect the screen resolution and the first resolution, and calculate the window display area based on the screen resolution, the first resolution and the second resolution. The window display area includes a third display area and a fourth display area, and the height of the third display area is equal to that of the fourth display area. Then control the display to display the first screen projection window in the third display area, and control the display to display the second screen projection window in the fourth display area.

[0137] In some embodiments, when calculating the window display area based on the screen resolution, the first resolution, and the second resolution, the display device may calculate a third aspect ratio based on the ratio of the horizontal pixel value to the vertical pixel value in the second resolution. Third and fourth layout parameters are set. The third layout parameters include a third layout width and a third layout height, and the fourth layout parameters include a fourth layout width and a fourth layout height, with the third layout height being equal to the fourth layout height. The sum of the third layout width and the fourth layout width is equal to the horizontal pixel value of the screen resolution, the third layout height and the third layout height are less than or equal to the vertical pixel value of the screen resolution, the ratio of the third layout width to the third layout height is equal to the first aspect ratio, and the ratio of the fourth layout width to the fourth layout height is equal to the third aspect ratio.

[0138] For example, the screen resolution of the display device can be 1920×1080, and the first resolution of the first projection screen can be 512×1088. After the display device displays the projection window of the first terminal device Client1, it listens to the projection connection instruction of the new second terminal device Client2, and then synchronously creates the second projection window View corresponding to the second terminal device 520. After the View is successfully created, the position management module is started to set and manage the display position of SessionClient2, and it is processed and obtained through the position management module. Since the first projection window has been displayed on the left, the orientation mark of the second projection window is set to the right, that is, the mViewPosition of Client2 is marked as 2.

[0139] The View management data of the management object includes: Session:SessionClient2; View:SurfaceView / TextureView; MViewPosition:2, and the second resolution of the second projection screen is obtained as 1552×1088. The steps of calculating the layout parameters and layout position coordinates of the first projection window and the second projection window according to the above algorithm and storing the obtained layout parameters and layout position coordinates in the data management module can be expressed as follows:

[0140] Session: SessionClient1; View: SurfaceView / TextureView; MViewPosition: 1; Position: (0, 37); Width: 473; Height: 1006;

[0141] Session: SessionClient2; View: SurfaceView / TextureView; MViewPosition: 2; Position: (485, 37); Width: 1435; Height: 1006.

[0142] After the data management template records the above data information, it transmits the data information to the View management module through the message management module. The View management module then displays the position according to the transmitted data information through the setLayoutParams method.

[0143] That is: FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.width=473; lp1.height=1006; lp1.leftMargin=0; lp1.topMargin=37; mPlayerView.setLayoutParams(lp1);

[0144] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.width=1435; lp1.height=1006; lp1.leftMargin=485; lp1.topMargin=37; mPlayerView.setLayoutParams(lp1).

[0145] By setting the display positions of the first and second projection windows corresponding to SurfaceView / TextureView through the View management module, you can obtain the display effect of the multi-screen projection interface as shown in Figure 13. The display device displays the first and second projection windows at the same height, and at the same time, both windows can be displayed to the maximum extent in the multi-screen projection interface.

[0146] After the display device displays the projection window of the terminal device 500, it can also respond to the corresponding projection termination instruction of the terminal device 500 to terminate the projection connection between the display device and the terminal device 500 and remove the projection window of the terminal device 500. In some embodiments, the projection termination instruction can be input from the display device, can be based on input from the terminal device, or can be automatically generated due to external reasons such as the network.

[0147] Therefore, in some embodiments, the display device monitors the screen projection connection session of the terminal device 500. After monitoring the termination screen projection instruction of the first screen projection session, the first screen projection window is removed and the first guide window is called in response to the termination screen projection instruction of the first screen projection session. Since the first guide window is released when the first screen projection window is displayed, the first guide window needs to be re-called after the display device removes the first screen projection window. For example, the relevant resources of the window object corresponding to the first guide window are re-called. Then the screen resolution and the second resolution are detected, and the window display area is calculated according to the screen resolution, the second resolution and the reference resolution. Among them, the window display area includes a fifth display area and a sixth display area, and the height of the fifth display area is equal to that of the sixth display area. After calculating the fifth display area and the sixth display area, the display is controlled to display the first guide window in the fifth display area, and the display is controlled to display the second screen projection window in the sixth display area.

[0148] In some embodiments, when calculating the window display area based on the screen resolution, the second resolution, and the reference resolution, the display device sets fifth and sixth layout parameters. The fifth layout parameters include a fifth layout width and a fifth layout height, and the sixth layout parameters include a sixth layout width and a sixth layout height, with the fifth layout height being equal to the sixth layout height. The sum of the fifth layout width and the sixth layout width is equal to the horizontal pixel value of the screen resolution, the fifth layout height and the fifth layout height are less than or equal to the vertical pixel value of the screen resolution, the ratio of the fifth layout width to the fifth layout height is equal to the third aspect ratio, and the ratio of the sixth layout width to the sixth layout height is equal to the second aspect ratio.

[0149] It is understandable that when the display device calculates the fifth layout parameter and the sixth layout parameter, the algorithm adopted is the same as the algorithm used to calculate the first layout parameter, the second layout parameter, the third layout parameter, and the fourth layout parameter, which will not be described in detail here.

[0150] Because the terminal device 500 can be placed in different positions depending on how it is used, for example, a terminal device such as a smartphone can be in either a portrait or landscape position during use. The portrait position refers to a state where the width of the displayed image is smaller than its height, while the landscape position refers to a state where the width of the displayed image is larger than its height. In different placement positions, the terminal device 500 can present different screen layouts.

[0151] Based on this, in some embodiments, when the display terminal device 500 is placed in different states, the resolution of the projected screen will also be different, that is, the projected screen will be presented in different aspect ratios in different placement states. For example, if the first terminal device 510 is in landscape mode, the resolution of the first projected screen is 1920×1080. When the first terminal device 510 is switched to portrait mode, the size of the first projected screen is 1080×1920.

[0152] Obviously, when the resolution of the projection screen changes, the display device also needs to recalculate the display area of ​​the projection window according to the resolution of the projection screen after the change. Therefore, in some embodiments, when only the first projection window is displayed in the multi-channel projection interface, the display device also monitors the change event of the first projection session. When the change event of the first resolution is monitored, the changed resolution of the first projection screen is detected. Among them, the changed resolution is the resolution of the first projection screen after the change event is executed. The window display area is then calculated based on the screen resolution, the changed resolution and the reference resolution. The window display area includes the seventh display area and the eighth display area, and the height of the seventh display area is equal to that of the eighth display area. Then control the display to display the first projection window in the seventh display area, and control the display to display the second guide window in the eighth display area.

[0153] Similarly, in some embodiments, when only the second projection window is displayed in the multi-channel display interface, the display device also monitors the change event of the second projection session. When the change event of the second resolution is monitored, the changed resolution of the second projection screen is detected. Herein, the changed resolution is the resolution of the second projection screen after the change event is executed. Then, the window display area is calculated based on the screen resolution, the changed resolution and the reference resolution. The window display area includes the ninth display area and the tenth display area, and the height of the ninth display area is equal to that of the tenth display area. Then, the display is controlled to display the first guide window in the ninth display area, and the display is controlled to display the second projection window in the tenth display area.

[0154] In some embodiments, when multiple display interfaces simultaneously display a first projection interface and a second projection interface, the display device monitors change events of the first projection session and the second projection session. When a change event of the first resolution and / or the second resolution is monitored, the window display area is calculated according to the refresh resolution and the screen resolution after the change event is executed. That is, when the display device monitors a resolution change event in any projection session, the window display area is recalculated to display the projection window according to the newly calculated window display area.

[0155] For example, after the display device establishes a projection connection relationship with the first terminal device 510 and the second terminal device 520, the multi-channel projection interface shown in Figure 13 is displayed, and the projection management module can monitor the first projection session through the onVideoChanged() method of UsessionDisplayListener. When it is monitored that the first terminal device 510 changes from a vertical screen state to a horizontal screen state, that is, when the first resolution changes, the message processing module notifies the algorithm processing module of the changed first resolution, so that the algorithm processing module recalculates the layout parameters and layout position coordinates of each window through the algorithm provided by the above embodiment. After the calculation is completed, the algorithm processing module notifies the View management module of the data information through the message processing module, and the View management module then re-layouts the first projection window and the second projection window according to the transmitted data information, displaying the effect shown in Figure 14.

[0156] In order to make full use of the display space of the multi-channel projection interface and reduce large blank areas, in some embodiments, after the display device displays the first projection window, it also calculates the window aspect ratio of the first projection window based on the first layout width and the first layout height, and compares the window aspect ratio with the first aspect ratio. If the window aspect ratio is not equal to the first aspect ratio, the first projection screen in the first projection window is scaled. That is to say, when the aspect ratio of the projection window is inconsistent with the aspect ratio of the projection screen, it means that in addition to the display area of ​​the projection screen in the projection window, there are other blank areas. The display device can then scale the projection screen in the projection screen, thereby reducing the blank area in the projection window.

[0157] In order to facilitate the user's interaction with the projection window, in some embodiments, when the display device displays the multi-channel projection interface, it can also display the broadcast control function control in the multi-channel projection interface, and the user can perform interactive operations through the broadcast control function control to input control instructions and control the display device to perform interactive responses. For example, since the projection content of the display device and the terminal device 500 is diverse, there are mirrored video streams, pushed URL addresses, and different media types such as pictures and music, the broadcast control function controls for different media types are also different. For example, the mirrored video projection does not support fast forward and rewind operations nor does it support play and pause operations; the pushed URL address can fast forward, rewind, play, pause, etc.; the music can fast forward, rewind, play, pause, etc.; the pictures do not support fast forward, rewind, play, pause, etc.

[0158] In some embodiments, the display device's playback control function controls can be displayed above the projection screen in a floating manner, as shown in Figure 15. Alternatively, the display device's playback control function controls can also be displayed in the edge area of ​​the multi-channel projection interface, as shown in Figure 16. For example, the display device can display the above playback control function controls when detecting a focus event in the projection window.

[0159] In order to take into account the display area of ​​the projection window and the display effect of the broadcast control function control, for the scenario where the broadcast control function control is displayed in the edge area, in some embodiments, after the display device displays the first projection window in the first display area, the display device responds to the focus event of the first projection window and controls the display to display the broadcast control function control of the first projection window in the edge area. Read the second preset height of the edge area. The second preset height is greater than the first preset height. Then set the sum of the first layout height and the second preset height to be equal to the vertical pixel value of the screen resolution to generate the first layout height based on the second preset height. That is to say, when the display device displays the broadcast control function control in the edge area, it can also compress the display area of ​​the projection window or the guide window in proportion, thereby increasing the edge area of ​​the multi-channel projection interface to facilitate the display of the broadcast control function control.

[0160] Alternatively, the display device may control the display to display the playback control function controls of the first projection window in an edge area in response to a focus event of the first projection window, detect the maximum display height when displaying the playback control function controls, and then calculate the first layout height of the first projection window based on the maximum display height.

[0161] For example, after the display device establishes a screen projection connection relationship with the first terminal device 510 and the second terminal device 520, the multi-screen projection interface shown in Figure 13 is displayed. At this time, the layout height of the first and second screen projection windows is 1006. At this time, the user clicks on the first screen projection window, and the display device detects that the maximum display height when displaying the broadcast control function control is 864. It is necessary to reset the layout parameters of the first and second screen projection windows, set the layout height of the first and second screen projection windows from 1006 to 864, and compress the first and second screen projection windows in equal proportions. Set Client 1: width1, height1; Client 2: width2, height2; Through the following combined calculations, width1, top, height, and width_guide can be calculated; that is, width1 / height1=ratio1; width2 / height2=ratio2; reset height1 and height2 to 864, and then width1 and width2 can be obtained according to the above ratios. At this time, the data stored in the data management module includes: Session: SessionClient1; View: SurfaceView / TextureView; MViewPosition: 1; Position: (0, 108); Width: 406; Height: 864,

[0162] Session: SessionClient2; View: SurfaceView / TextureView; MViewPosition: 2; Position: (418, 108); Width: 1232; Height: 864.

[0163] The data management module notifies the View management module of the above data information, and sets the display positions of the first projection window and the second projection window corresponding to SurfaceView / TextureView respectively, so as to obtain the display effect of the multi-screen projection interface as shown in Figure 17.

[0164] The display device provided by the embodiment of the present disclosure can also detect the projection replacement mode in response to the projection connection instruction sent by the third terminal device. If the projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement projection window at the focus position is removed, and the third projection window is displayed according to the orientation mark of the first replacement projection window. If the projection replacement mode is the second mode, the locking attribute of the projection window is detected, and the second replacement projection window is removed according to the locking attribute, and the third projection window is displayed according to the orientation mark of the second replacement projection window. Among them, the second replacement projection window is a projection window whose locking attribute is in an unlocked state. Therefore, according to different projection replacement modes, the projection window can be replaced by the window locking attribute or the focus position, so that the display device can quickly switch the third projection window to the position desired by the user, thereby improving the user experience. Specifically, as shown in Figure 18, it includes the following steps:

[0165] S1801: In response to the screen projection connection instruction sent by the third terminal device, detect the screen projection replacement mode.

[0166] Due to the limitation of screen resolution, the number of projection windows that the display device can display at the same time is limited. In some embodiments, the display device can display at most the first projection window and the second projection window at the same time. The sum of the number of the first projection window and the second projection window is the target number. Therefore, after the display device has displayed the first projection window and the second projection window, if it receives the projection connection instruction sent by the third terminal device 530, the display device needs to replace the displayed first projection window or the second projection window, then detects the current projection replacement mode of the display device.

[0167] For example, if the target number is 2, when the display device receives the screen projection connection instruction sent by the terminal device 500, it detects the number of terminal devices currently connected to the display device. If the number of terminal devices currently connected to the display device is greater than 2, the screen projection replacement mode is detected to replace the first screen projection window or the second screen projection window through the screen projection replacement mode; if the number of terminal devices currently connected to the display device is greater than 2, the orientation identifier of the screen projection window corresponding to the terminal device is set according to the default priority order, so that the screen projection window is displayed through the orientation identifier.

[0168] For ease of description, the target number in the embodiment of the present disclosure is 2, that is, the first projection window represents one projection window, and the second projection window represents another projection window; the first terminal device 510 represents one terminal device, and the second terminal device 520 represents another terminal device.

[0169] Therefore, when the first and second projection windows are already displayed in the multi-screen projection interface of the display device, it means that the display device has reached the maximum number of projection windows that can be displayed. At this time, when the display device receives the projection connection instruction sent by the third terminal device 530, it detects the current projection replacement mode and executes different interactive programs according to the detected projection replacement mode.

[0170] As shown in Figure 19, the projection replacement mode of the display device can be set based on the locking attribute of the projection window, that is, in some embodiments, the display device also obtains the locking attribute of the projection window. The locking attribute includes a locked state and an unlocked state, and the locked state and the unlocked state correspond to different parameters. For example, the locked state of the locking attribute is lock, and the unlocked state is unlock. If the locking attributes of the projection window are all unlocked, set the replacement mode parameter to the first parameter; if the locking attributes of the projection window include a locked state, set the replacement mode parameter to the second parameter. Among them, the first parameter is used to characterize the projection replacement mode as the first mode, and the second parameter is used to characterize the projection replacement mode as the second mode. The first mode queries the projection window that needs to be replaced according to the focus position of the focus frame, and the second mode queries the projection window that needs to be replaced according to the state of the locking attribute.

[0171] In some embodiments, when the display device displays a multi-channel projection interface, it can also display a broadcast control function control in the multi-channel projection interface, and the user can perform interactive operations through the broadcast control function control to input control instructions and control the display device to perform interactive responses. For example, since the projection content of the display device and the terminal device 500 is diverse, there are mirrored video streams, pushed URL addresses, and different media types such as pictures and music, the broadcast control function controls for different media types are also different. For example, mirrored video projection does not support fast forward and rewind operations nor does it support play and pause operations; pushed URL addresses can fast forward, rewind, play, pause, etc.; music can fast forward, rewind, play, pause, etc.; pictures do not support fast forward, rewind, play, pause, etc.

[0172] Since the lock attribute is a user-customizable attribute, the broadcast control function control can also include a lock control. Therefore, in some embodiments, when the display device displays the screen projection window, it also controls the display 260 to display a lock control in the screen projection window. Among them, the lock control is associated with the screen projection window. The lock control can be displayed floating above the screen projection window; or, as shown in Figure 20, the lock control can also be displayed in the edge area above or below the screen projection window in the multi-channel screen projection interface.

[0173] In some embodiments, when a user performs a selection operation on a lock control, the display device automatically generates a selection instruction for the lock control and refreshes the current value of the lock control and the lock attribute of the projection window in response to the selection instruction. After the display device refreshes the current value of the lock control, the display content presented by the lock control will also change. For example, when the display device displays the lock control shown in Figure 20, the user performs a selection operation on the lock control on the right, and the display device refreshes the lock attribute of the projection window to unlock and refreshes the current value of the lock control, so that the lock control presents the display content shown in Figure 21.

[0174] S1802: If the projection replacement mode is the first mode, detect the focus position of the focus frame, remove the first replacement projection window at the focus position, and display the third projection window according to the orientation mark of the first replacement projection window.

[0175] When the display device detects that the projection replacement mode is the first mode, it indicates that the lock attributes of the first projection window and the second projection window are both unlocked. At this time, the display device detects the focus position of the focus frame and removes the first replacement projection window at the focus frame position. For example, if the focus position of the focus frame is on the first projection window, the first projection window is removed; if the focus position of the focus frame is on the second projection window, the second projection window is removed.

[0176] While removing the first replacement projection window, the display device also creates a third projection window. The third projection window is used to display the projection screen of the third terminal device 530, and the third projection window is the same as the root node of the multi-way projection interface. Then display the third projection window according to the orientation identifier of the first replacement projection window. Therefore, if the projection replacement mode is the first mode, when the display device displays the third projection window according to the orientation identifier of the first replacement window, it can record the first orientation identifier of the first replacement window and create a third projection window. Then set the orientation identifier of the third projection window to the first orientation identifier, and draw the third projection window according to the first orientation identifier.

[0177] That is, when the projection management module listens to the onConnect connection of the Session corresponding to the third terminal device Client3 through UsessionConnectionListener, it calls the Session.finish() interface to perform a disconnection process on the Session corresponding to the projection window selected by the focus frame. For example, when the projection window selected by the focus frame is the second projection window, the projection management module notifies the View management module to remove the View of Client2 through rootview.removeView(). At the same time, the View management module also stores the information of Client3 and creates the third projection window playback control SurfaceView / TextureView. After the SurfaceView / TextureView is successfully created, the display device sets the orientation identifier of Client3 to the orientation identifier of Client2 by starting the position management module, thereby replacing the position of the second projection window. Therefore, for the third terminal device, the data corresponding to its Client3 can be represented as: Session:SessionClient3; View:SurfaceView / TextureView; MViewPosition:2.

[0178] For example, when the display device establishes a screen projection connection with the first terminal device 510 and the second terminal device 520, a multi-screen projection interface as shown in Figure 21 is displayed. Among them, the left side is the first screen projection window, and the right side is the second screen projection window. The lock attributes of the first screen projection window and the second screen projection window are both unlocked. At this time, a screen projection connection instruction is sent to the display device through the third terminal device 530, and the display device detects the focus position of the focus frame according to the first mode. As shown in Figure 21, the focus position is located in the second screen projection window, that is, the second screen projection window is the first replacement screen projection window. The display device removes the second screen projection window and sets the orientation identifier of the third screen projection window to the orientation identifier of the second screen projection window, so that the third screen projection window replaces the position of the second screen projection window, presenting a multi-screen projection interface as shown in Figure 22.

[0179] It is understandable that when the display device replaces the first replacement projection window with the third projection window, the display area of ​​the third projection window can be calculated based on the algorithm provided in the above embodiment, which will not be elaborated here.

[0180] Obviously, to facilitate the user's modification of the lock properties of the third projection window, the display device also generates a lock control associated with the third projection window after displaying the third projection window. Therefore, in some embodiments, the display device further controls the display 260 to display a target lock control, and in response to a user input selection instruction based on the target lock control, refreshes the current value of the lock control and the lock properties of the third projection window. The target lock control is associated with the third projection window.

[0181] S1803: If the projection replacement mode is the second mode, detect the locking attribute of the projection window, remove the second replacement projection window according to the locking attribute, and display the third projection window according to the orientation mark of the second replacement projection window.

[0182] When the display device detects that the projection replacement mode is the first mode, it indicates that the lock attribute of the first projection window and / or the second projection window is locked. At this time, the display device detects the lock attribute of each projection window and determines a second replacement projection window based on the lock attribute of each window. The second replacement projection window is a projection window whose lock attribute is unlocked.

[0183] Since the lock attribute is user-definable, the lock attributes of the first projection window and the second projection window may both be in a locked state. At this time, the display device cannot replace any displayed projection window with a third projection window. Therefore, in some embodiments, if the projection replacement mode is the second mode, the display device traverses the lock attributes of the projection window. If the lock attribute of the projection window does not include an unlocked state, the control communication device sends a projection failure notification to the third terminal device, so that the third terminal device prompts the user with information about the projection failure.

[0184] For example, the display device can display a multi-channel screen projection interface as shown in Figure 23. Among them, the left side is the first screen projection window, and the right side is the second screen projection window. The lock properties of the first screen projection window and the second screen projection window are both in the locked state. At this time, after the screen projection management module listens to the onConnect connection of the Session corresponding to the third terminal device Client3 through UsessionConnectionListener, it will not perform disconnection processing on the first terminal device Client1 and the second terminal device Client2. The screen projection management module will not send any notification to the View management module, and the display device will not create a view for Client3. Instead, it notifies Client3 that the current mode can no longer project the screen. After receiving this message, Client3 will prompt the user on the mobile terminal that the screen projection has failed.

[0185] If the lock attribute of the projection window includes an unlocked state, in some embodiments, when the display device displays the third projection window based on the orientation identifier of the second replacement projection window, the orientation identifier of the second replacement projection window is recorded, and the second replacement projection window is the projection window with the unlocked lock attribute. A third projection window is created, and the orientation identifier of the third projection window is set to the second orientation identifier. The third projection window is then drawn based on the second orientation identifier.

[0186] That is, when the projection management module listens to the onConnect connection of the Session corresponding to the third terminal device Client3 through UsessionConnectionListener, the index detection finds the unlocked projection window, and calls the Session.finish() interface to perform the disconnection process on the Session corresponding to the unlocked projection window. For example, when the unlocked projection window is the first projection window, the projection management module notifies the View management module to remove the View of Client1 through rootview.removeView(). At the same time, the View management module also stores the information of Client3 and creates the third projection window playback control SurfaceView / TextureView. After the SurfaceView / TextureView is successfully created, the display device sets the orientation identifier of Client3 to the orientation identifier of Client1 by starting the position management module, thereby replacing the position of the first projection window. Therefore, for the third terminal device, the data corresponding to its Client3 can be represented as: Session:SessionClient3; View:SurfaceView / TextureView; MViewPosition:1.

[0187] For example, when the display device establishes a screen projection connection with the first terminal device 510 and the second terminal device 520, a multi-screen projection interface as shown in Figure 20 is displayed. Among them, the left side is the first screen projection window, the right side is the second screen projection window, the first screen projection window is in an unlocked state, and the locking attribute of the second screen projection window is in a locked state. At this time, a screen projection connection instruction is sent to the display device through the third terminal device 530, and the display device detects the locking attribute of the screen projection window according to the second mode. The first screen projection window with an unlocked locking attribute is the second replacement screen projection window. The display device removes the first screen projection window and sets the orientation identifier of the third screen projection window to the orientation identifier of the first screen projection window, so that the third screen projection window replaces the position of the first screen projection window, presenting a multi-screen projection interface as shown in Figure 24.

[0188] It can be understood that the embodiments of the present disclosure are described with reference to the target number. In some application scenarios, if the target number set by the display device is greater than 2, the lock attribute of the projection window is in a locked state, and the number of unlocked projection windows is greater than 1, the display device can mark the highest-priority unlocked projection window as the second replacement window according to the preset priority order, so that the second replacement window can be replaced by the third projection window.

[0189] In addition, when the display device displays the first projection window and the second projection window, if the user disconnects the projection connection relationship between one of the terminal devices, the projection window of the remaining terminal device may move. For example, when the projection connection relationship between the first terminal device 510 and the display device is disconnected, the display device removes the first projection window and displays the guide window and the second projection window in the multi-projection interface. Since the display device has a default priority order, when the display device recalculates the display area of ​​each window, it may set the orientation identifier of the second projection window to an orientation identifier with a higher priority, causing the second projection window to suddenly switch to the position of the first projection window. In this way, when the display device displays multiple projection windows at the same time, if the projection connection relationship between one of the terminal devices 500 is disconnected, the position of the projection window will switch back and forth, and the user's line of sight also needs to follow the position of the projection window to switch back and forth, affecting the user's viewing experience.

[0190] In order to improve the problem of switching back and forth between the above-mentioned projection windows, in some embodiments, after displaying the third projection window, the display device can also delete the target projection session in response to the termination projection instruction of the target projection session. The target projection session is a communication session generated based on the third terminal device. The third position identifier of the third projection window is recorded, and the third projection window is removed, and the guide window is called, wherein the guide window is used to display the guide page, and the root node of the guide window is the same as the root node of the multi-way projection interface. Then control the display 260 to display the guide window according to the third position identifier.

[0191] For example, the display device is already connected to two Clients, namely the first terminal device Client1 and the third terminal device Client3. In the case of the multi-screen projection interface shown in Figure 22 above, the left side is the first projection window of Client1, and the right side is the third projection window of Client3. The projection management service can listen to the onDisConnect() request of Client3 to disconnect through UsessionConnectionListener, and notify the data management module to remove the Session of Client3. After the data management module successfully removes the information of Client3, the data management module synchronously notifies the View management module so that the View management module executes the rootview.removeView() removal operation on the corresponding View, and retains the data information of Client1, including: Session: SessionClient1; View: SurfaceView / TextureView; MViewPosition: 1. At the same time, the guide window on the right can be re-called according to the orientation identifier of Client3, so that the multi-screen projection interface can present a display effect such as that shown in Figure 25. In the display effect shown in Figure 25, the size of the projection screen in the video stream corresponding to the projection of the first terminal device 510 (Client1) is 1920×1080, and the guide page display size is 954×538. At this time, the video stream ratio and the guide video stream ratio on the right are close to 16 / 9, and both use the same layout display, for example, both are displayed in horizontal format.

[0192] In some embodiments, the guide window and the multi-screen projection interface also share the same root, that is, the root node of the guide window and the multi-screen projection interface is also the same. When calling the guide window, the display device can set the guide window to a display state; or re-call the relevant resources of the window object corresponding to the guide window.

[0193] In order to ensure the standardization of the layout of each window, when displaying the guide window, the display device also calculates the display area of ​​the guide window based on the resolution of the guide page, that is, in some embodiments, the display device detects the screen resolution, the reference resolution and the projection screen resolution. Among them, the reference resolution is the resolution of the guide page, and the projection screen resolution is the resolution of the projection screen in the multi-channel projection interface, that is, the projection screen corresponding to the terminal device 500 currently connected to the display device. The window layout parameters of the guide window are then calculated based on the screen resolution, the reference resolution and the projection screen resolution. Among them, the window layout parameters include layout width and layout height.

[0194] After calculating the window layout parameters for the guide window, the display device may display the guide window according to the window layout parameters. Therefore, in some embodiments, when displaying the guide window according to the third-party position identifier, the display device generates the layout position coordinates of the guide window based on the window layout parameters and the third-party position identifier, and records the window layout parameters and the layout position coordinates. The guide window is then drawn according to the window layout parameters and the layout position coordinates.

[0195] For example, when the display device displays the multi-screen projection interface as shown in Figure 22 above, the first screen projection window of Client1 is on the left, and the third screen projection window of Client3 is on the right. At this time, the screen projection connection between Client3 and the display device is disconnected, the display device deletes the Session of Client3, and removes the View corresponding to Client3. Based on the algorithm provided in the above embodiment, the width and height of each window, as well as the coordinates of the upper left corner of the window are calculated according to the screen resolution of the display device, the picture resolution of the first projection screen and the reference resolution of the guide window, and the above data information is stored through the data management module. The data information is then sent to the View management module, so that the View management module displays the display position of the projection SurfaceView / TextureView and the display position of the guide Layout through the setLayoutParams method, and the display effect shown in Figure 25 above can be obtained.

[0196] In some embodiments, when the multi-way projection interface of the display device includes only one projection window, that is, the display device only establishes a projection connection relationship with one terminal device 500, the projection connection relationship between the terminal device 500 and the display device is disconnected, and the display device deletes the projection session of the terminal device and removes the projection window to display the default guide window, which can be as shown in the aforementioned Figure 9.

[0197] For example, the screen resolution is 1920×1080 and the baseline resolution of the guide page is 954×538. When the display device displays the default guide window, it calculates the window layout parameters and layout position coordinates based on the algorithm provided in the above embodiment. Among them, the data transmitted by the data transmission module includes: Session; View: layout; Position: (0, 271); Width: 954; Height: 538; Session; View; Position: (966, 271); Width: 954; Height: 538. The View management module displays the display position of the guide layout through the setLayoutParams method, and the display effect shown in Figure 9 above can be obtained.

[0198] As mentioned above, since the terminal device 500 may be in different placement states according to different usage methods, for example, for terminal devices 500 such as smartphones, they may be in portrait and landscape states during use. When the display device displays the projection images of multiple terminal devices 500 by area division, each display area will present a different screen ratio due to the different placement states of the terminal devices 500. For example, the first terminal device 510 is in landscape state, and the first projection screen size is 1920×1080, and the second terminal device 520 is in portrait state, and the second projection screen size is 512×1088. Then the first projection window can be 954×538, and the second projection window can be 508×1080.

[0199] In order to make full use of the display space of the multi-screen projection interface and reduce large black screen areas, in some embodiments, the display device can also set the width or height of the first projection window and the second projection window to be consistent when the sizes of the first projection screen and the second projection screen are inconsistent, and scale the projection windows according to the width or height.

[0200] For example, when the first terminal device 510 is in portrait mode, the size of the first projection screen is 512×1088; when the second terminal device 520 is in landscape mode, the size of the second projection screen is 1920×1080, and the display resolution of the display device is 1920×1080, the default size of the first projection window and the second projection window (i.e., the guide page display size) is 954×538. At this time, the video stream ratio is different from the guide video stream ratio on the right and the direction is different. It can be calculated through the algorithm that the display height of the left video stream is about 848 and the width is about 400. Excluding the blank area, the remaining width is the guide page width, and the height is consistent with the video stream.

[0201] The display device dynamically adjusts the broadcast control function controls according to the display area of ​​the projection window in the multi-channel projection interface, so that the display position of the broadcast control function controls is adapted to the projection window to facilitate user interaction. For example, the broadcast control function controls belonging to the same projection window can form a broadcast control list. The broadcast control list includes several button controls such as full screen, fast forward, fast rewind, play / pause, mute / unmute, and close. The size of each button control is 60×60, and the horizontal spacing of the button controls is 24. Combined with the left and right margins of the broadcast control list, the minimum display width for displaying six button controls is 528, the minimum display width for displaying three button controls is 276, the minimum display width for displaying two button controls is 192, and the minimum display width for displaying one button control is, and the same applies to other numbers.

[0202] Due to the diversity of screen projection content between the display device and the terminal device, the displayed broadcast control function controls may not match the screen projection content, reducing the user experience. Based on this, the display device provided by the embodiment of the present disclosure can also obtain the media type of the screen projection data in response to a connection event for establishing a screen projection connection relationship between the display device and the fourth terminal device. Extract the broadcast control data associated with the media type in the broadcast control list, and create a broadcast control view based on the broadcast control data. Then set the display position of the broadcast control view according to the display area of ​​the screen projection window, and finally, control the display to display the broadcast control view at the display position. Among them, the broadcast control list includes the broadcast control data of the preset broadcast control view, and the broadcast control view is a control in the screen projection interface for performing playback control on the screen projection data. Through the above process, the display device can dynamically adjust the display of the broadcast control function controls according to the media type and the screen projection display area to enhance the user experience. Specifically, as shown in Figure 26, it includes the following steps:

[0203] S2601, in response to a connection event for establishing a screen projection connection relationship between the display device and the fourth terminal device, obtaining a media asset type of the screen projection data;

[0204] S2602: Extracting broadcast control data associated with the media asset type from the broadcast control list, and creating a broadcast control view based on the broadcast control data. The broadcast control list includes the broadcast control data of a preset broadcast control view. The broadcast control view is a control in the screen projection interface for performing playback control on the projection data.

[0205] To facilitate the processing of broadcast control data, the display device can generate a broadcast control sublist based on the broadcast control data associated with the media asset type in the broadcast control list, and create a broadcast control view based on the broadcast control sublist. When subsequent logical processing of the broadcast control view is performed, only the broadcast control data in the broadcast control sublist needs to be processed. In other words, the broadcast control list is a master list containing broadcast control data for all types of broadcast control views, and the broadcast control sublist is a sublist containing the broadcast control data for the broadcast control view related to the current screen projection connection of the fourth terminal device.

[0206] The generation of the broadcast control sublist can include multiple methods. After the display device obtains the media asset type of the projection data, it calls the broadcast control list, copies the broadcast control list, and deletes the broadcast control data not associated with the media asset type from the broadcast control list to generate a broadcast control sublist. The display device can also create a broadcast control sublist and add the broadcast control data associated with the media asset type in the broadcast control list to the broadcast control sublist.

[0207] For example, when the screen projection management module listens to the onConnect connection of the new Session through UsessionConnectionListener, it can obtain the media type of the currently connected terminal device (i.e., the fourth terminal device Client4) from the Session, such as VIDEO_URL (video URL)\MIRROR_STREAM (mirror stream)\MUSIC (music)\PIC (photo), etc. For example, if the display device currently displays the media data obtained based on the VIDEO_URL address pushed by the fourth terminal device, then full screen, fast forward, fast rewind, play / pause, mute / unmute, and close operations can be performed on the display device.

[0208] At this point, you can add play control data to this session. For example, the play control sublist includes six play control function controls associated with the media asset type, such as six button controls. The data format corresponding to these button controls can be expressed as follows: full screen button control: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132]; rewind button control: DoublePushControlBean[mStatus=true,mType=rewind,mResId=2131231110]; play button control: DoublePushControlBean[mStatus=true,mType=playstatus,mResId=2 131231149]; fast forward button control: DoublePushControlBean[mStatus=true,mType=forward,mResId=2131231111]; mute button control: DoublePushControlBean[mStatus=true,mType=mutestatus,mResId=2131231152]; close button control: DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0209] In this way, the display device can create and display the play control function controls related to the media asset type of the current Client 4 according to the play control sub-list, so that the user can perform interactive operations.

[0210] S2603: Setting a display position of the broadcast control view according to the display area of ​​the projection window, and controlling the display to display the broadcast control view at the display position;

[0211] In some embodiments, when multiple screens are being projected, to facilitate user differentiation between the playback control controls corresponding to each projection, the display device can dynamically adjust the playback control controls based on the display area of ​​the projection window in the multi-screen projection interface, so that the display position of the playback control controls matches the display area of ​​the projection window. Specifically, the display device can set the display position of the playback control view based on the display area of ​​the projection window and control the display to display the playback control view at that position.

[0212] The play control view size can be pre-set based on actual needs. For example, the play control list includes play control controls such as full screen, fast forward, fast rewind, play / pause, mute / unmute, and off. Each play control can be set to 60×60 pixels, with a horizontal spacing of 24 pixels. Considering the left and right margins of the play control list, the minimum display width for six play control controls is 528 pixels, for three controls is 276 pixels, for two controls is 192 pixels, and for one control is 108 pixels. The same applies to other numbers.

[0213] To improve the display effect, the broadcast control view can be displayed in a horizontal or vertical arrangement. The display device can plan the display position of the broadcast control view based on the display area of ​​the projection window. It can also calculate the maximum number of broadcast control views that can be displayed in horizontal or vertical arrangement based on the display area size of the projection window and the preset view size of the broadcast control view. Therefore, the display device can obtain the area size of the display area of ​​the projection window and the view size of the preset broadcast control view. The display number of the playback view is calculated based on the area size and view size, and the broadcast control data associated with the media asset type is extracted from the broadcast control list according to the display number.

[0214] When the playback control controls are displayed in landscape mode, the display device can obtain the width of the projection window display area and the total width of the playback control bar. The playback control bar is arranged at preset spacing based on the playback control views associated with the media asset type. The width of the projection window display area is compared with the total width of the playback control bar. If the width is greater than or equal to the total width, indicating that the playback control views associated with the media asset type can be fully displayed in landscape mode, the playback control data associated with the media asset type is subsequently extracted from the playback control list, and a playback control view is created and displayed based on the playback control data.

[0215] If the area width is smaller than the total width, it means that the broadcast control views associated with the media asset type cannot be fully displayed in the horizontal version. Then, the number of broadcast control views that can be displayed is calculated based on the area width and the total width. According to the number of displays, the broadcast control data associated with the media asset type is extracted from the broadcast control list, and the broadcast control views of that number are created and displayed based on the broadcast control data.

[0216] It is understood that when the broadcast control function controls are displayed in landscape mode, the maximum number of broadcast control views that can be displayed in landscape mode can be calculated based on the width of the projection window display area. When the broadcast control function controls are displayed in portrait mode, the maximum number of broadcast control views that can be displayed in portrait mode can also be calculated based on the height of the projection window display area.

[0217] It should be noted that the display method of the broadcast control function controls provided in the above embodiment is only an example. The broadcast control function controls of the display device are not limited to the above-mentioned horizontal / vertical arrangement, but can also be displayed in other forms. For example, they can be arranged and displayed in a rectangular array or a circular array, or by edge sliding, but the basic display principles are universal.

[0218] In some embodiments, to meet actual user needs, weight values ​​can be set for the play control views in the play control list. When the display device extracts the play control data associated with the media asset type from the play control list according to the display quantity, the weight values ​​of the play control data in the play control list can be traversed. The play control data in the play control list are sorted in descending order of weight value. From the sorted play control list, the play control data associated with the media asset type for the first display quantity is extracted.

[0219] In other words, the control views in the playlist are prioritized based on their weights. If not all control views can be displayed, the view with the highest weight is selected for display. For example, the control views in the playlist are sorted from highest to lowest weight as follows: Full Screen, Off, Mute / Unmute, Play / Pause, Fast Forward, and Fast Rewind. If only three control views can be displayed, the Full Screen, Off, Mute / Unmute, and Play / Pause views are displayed.

[0220] For example, taking the VIDEO_URL address pushed as the media resource type, to display the aforementioned playback control controls, after the data management module obtains the display size of the projection window, the View management module formats and displays the display effect shown in Figure 27. The left projection window displays the media resource image obtained based on the pushed URL address, and supports playback controls such as full screen, fast forward, rewind, play / pause, mute / unmute, and off. The algorithm processing module calculates that the left projection window size is 1508×848, and the right projection window size is 400×848.

[0221] After calculating the display positions of the left and right projection windows, the View Management Module will layout and display the left and right projection windows. At the same time, the playback control view will be initialized and displayed based on the Session's playback control sublist data. For example, a display device can use a RecycleView control to display the playback control view. The RecycleView control's display parameters, including the position (i.e., coordinates) from the top and left boundaries, and the corresponding display width and height, can then be displayed using the setLayoutParams method.

[0222] Regarding the number of buttons displayed when the broadcast control view is displayed horizontally, the algorithm processing module calculates that the display area width of the left projection window (1508) is greater than the total width of the broadcast control bar (528) when displaying six broadcast control function controls, indicating that all broadcast control function controls can be displayed. Therefore, the broadcast control view created and displayed based on the session's broadcast control list data presents the display effect shown in Figure 27, with the six broadcast control function buttons centered at the bottom of the left projection window. Of course, they can also be displayed in any other position.

[0223] When the display device displays the broadcast control view, the user can input control commands based on the broadcast control view to control the display device to perform interactive responses. In response to a click command on the target broadcast control view, the display device can obtain the broadcast control type of the target broadcast control view from the broadcast control list, call the processing interface associated with the broadcast control type, and perform the processing operation associated with the target broadcast control view on the screen projection data based on the processing interface.

[0224] For example, as shown in Figure 27, the user can use the arrow keys on the remote control compatible with the display device to move the focus cursor to the "Play" button and press the "Confirm" button on the remote control to enter the pause command. At this time, RecycleView will listen for the user's onClick operation and call the Session's setPlayerPause() interface to pause playback based on the index type of the play control list (i.e., mType type is playstatus). Similarly, fast forward and rewind can also be performed by calling Session's setPlayerSeekBackward() and setPlayerSeekForward() to perform the corresponding play control processing.

[0225] In some embodiments, when the display area of ​​the projection window changes, the playback control function controls also need to be dynamically adjusted based on the display area of ​​the projection window. That is, the display device can respond to the notification of the change in the display area of ​​the projection window, obtain the changed display area, reset the display position of the playback control view according to the changed display area, and recalculate the number of playback views displayed based on the area size of the changed display area.

[0226] In this regard, the screen projection management module can monitor the screen projection session status. When the video changes, the algorithm processing module is notified through the message processing module. The algorithm processing module calculates the screen projection display area and then informs the data management module through the message processing module. After the data management module organizes the data, it notifies the View management module through the message processing module to redraw and display the View.

[0227] For example, if the content of the VIDEO_URL media asset changes or the direction of the video stream changes, the display device can rearrange the projection content, presenting the display effect shown in Figure 28, where the left projection window displays the VIDEO_URL media asset. The algorithm processing module calculates that the size of the left projection window is 459×816, and the size of the right projection window is 1449×816.

[0228] Regarding the display capacity when the playback control view is displayed in landscape orientation, the algorithm processing module calculates that the display area width of the left projection window (459) is less than the total width of the playback control bar (528) when displaying six playback control controls, but greater than the total width of the playback control bar (276) when displaying three playback control controls, indicating that three playback control controls can be displayed. Therefore, the three playback control controls (fullscreen, mute / unmute, and off) can be displayed. The data format is as follows: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132]; DoublePushControlBean[mStatus=true,mType=mutestatus,mResId=2131231152]; DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0229] The View management module can refresh the View based on the above broadcast control data by notifying RecycleView through the Adapter's notifyDataSetChanged method, replacing six broadcast control function controls with three, resulting in a display effect as shown in Figure 28, where three broadcast control function buttons are centered below the left projection window. As can be seen, in the above embodiment, the display device can adjust the number of broadcast control function controls from six to three for display when the projection window width cannot accommodate the arrangement of six broadcast control function controls.

[0230] Similarly, when the projection management module listens to the onConnect connection of the new Session through UsessionConnectionListener, it can obtain from the Session that the media type of the current Client4 is MIRROR_STREAM, and the View management module is used to layout and display the display effect shown in Figure 29. The projection window on the right displays the mirrored media screen, which can be full screen, muted / unmuted, and closed on the display device. At this time, the broadcast control data can be added to this Session. The data format is as follows: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132]; DoublePushControlBean[mStatus=true,mType=mutestatus,mResId=2131231152]; DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0231] The algorithm processing module calculates that the size of the right projection window is 400×848, which yields the following data: Session: SessionClient4, View: SurfaceView / TextureView, MViewPosition: 2, Position: (1520, 116), Width: 400, Height: 848, mDoublePushControlListBean: ...

[0232] The algorithm processing module calculates the projection display area and then informs the data management module through the message processing module. According to the data from the data management module, the size of the projection window on the right is 400×848, and the size of the three broadcast control function controls is 276. Since the display area width of the projection window on the right, 400, is greater than the total width of 276 occupied by the three broadcast control function controls, the three broadcast control function buttons can be fully displayed.

[0233] The View management module can initialize and display Recycleview, set the display parameters through LayoutParams, including the position (i.e. coordinates) from the upper and left boundaries, and the corresponding display width and height, and then display the position through the setLayoutParams method to present the display effect as shown in Figure 29, with three playback control function buttons displayed in the center below the right projection window.

[0234] If the width of the projection window doesn't allow for three buttons, the number of control buttons can be adjusted based on the width of the projection window's display area. For example, using the display shown in Figure 30, if the right projection window is 258×864, the display area width of 258 is less than the total width of 276 for three control buttons, but greater than the total width of 192 for two control buttons. Therefore, the display device can display two control buttons below the right projection window. Furthermore, since the volume can be controlled exclusively by the control buttons in the left projection window, only the fullscreen and close control buttons can be retained. The corresponding control list data is: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132]; DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0235] Similarly, taking the display effect shown in Figure 31 as an example, if the size of the right projection window is 156×864, the display area width 156 of the right projection window is less than the total width 192 when two playback control function controls are displayed, and greater than the total width 108 when one playback control function control is displayed. Therefore, the display device can display a playback control function control below the right projection window. Since closing the projection window can be controlled by stopping the projection operation on the fourth terminal device, only the full-screen button can be retained. That is, the corresponding playback control list data is: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132].

[0236] By analogy, if the size of the right window is 88×864, the display area width 88 of the right projection window is smaller than the total width 108 when a broadcast control function control is displayed, and the button control cannot be displayed. Therefore, the display device can cancel the display of the broadcast control function control below the right projection window.

[0237] In the above process, in response to the diversity of the screen projection content between the display device and the fourth terminal device, playback control function controls adapted to the screen projection content of the fourth terminal device are displayed. Different playback control function controls are displayed for different media types such as mirrored video streams, pushed URL addresses, pictures, and music, thereby reducing the phenomenon of errors in user-entered playback control operations and allowing users to intuitively distinguish different screen projection types when multiple screens are projected. For example, if mirrored video projection does not support fast forward and rewind operations, nor does it support play and pause operations, the playback control function controls for fast forward, rewind, play, and pause will not be displayed; if pushed URL addresses can perform fast forward, rewind, play, pause, etc., the playback control function controls for fast forward, rewind, play, and pause will be displayed; if music can perform fast forward, rewind, play, pause, etc., the playback control function controls for fast forward, rewind, play, and pause will be displayed; if pictures do not support fast forward, rewind, play, pause, etc., the playback control function controls for fast forward, rewind, play, and pause will not be displayed. Therefore, the display of the broadcast control function controls is dynamically adjusted according to the media type and the projection display area to improve the user experience.

[0238] As mentioned above, in a multi-channel projection scenario, the display device can only display the projection image of a single terminal device. If the display device is connected to multiple terminal devices (such as the first terminal device and the second terminal device mentioned above) at the same time, when the user wants to switch between multiple projection channels, it is necessary to frequently switch the projection signal source, and each switching process requires re-establishing the projection connection, resulting in a long waiting time for the switching process, which is not conducive to fast switching and reduces the user experience.

[0239] In some embodiments, the user can input control instructions based on the button controls in the multi-screen projection interface to control the display device to switch interfaces. For example, as shown in Figure 32, when the user clicks the full-screen button below the first projection window, the display device can switch from the multi-screen projection interface to full-screen display of the projection screen in the first projection window, that is, display the projection screen of the first terminal device 510. Since the display device will no longer display the projection screen corresponding to the second projection window after switching to full-screen display of the projection screen of the first projection window, the display device can close the projection data transmission channel corresponding to the second projection window, that is, stop the projection connection relationship with the second terminal device 520.

[0240] However, after the display device switches to the full-screen display of the projection image in one projection window, the user may also switch back to the multi-screen projection interface as needed. For example, after controlling the display device to display the projection image corresponding to the first projection window in full screen, if the user wants to watch the projection image in the second projection window, the user can control the display device to switch back to the multi-screen projection interface by pressing the "return" button on the remote control that comes with the display device. At this time, since the display device has interrupted the projection connection relationship with the second terminal device 520, the display device needs to re-establish the projection connection relationship and obtain the projection data sent by the second terminal device 520 to display the corresponding projection image through the second projection window in the multi-screen projection interface.

[0241] Based on this, the display device provided by the embodiment of the present disclosure can also obtain the screen resolution of the display in response to the full-screen instruction for full-screen display of the first projection window when displaying a multi-way projection interface. Then, the first target display area of ​​the first projection window is set according to the screen resolution, and the second target display area of ​​the second projection window is set, wherein the number of pixels contained in the second target display area is less than or equal to the preset pixel threshold. Thus, the first projection screen is displayed through the first target display area, and the second projection screen is displayed through the second target display area. Therefore, when the first projection window is displayed in full screen, the second projection window is not directly closed. Instead, the second projection screen is displayed by setting the second target display area using a small number of pixels that are difficult for the user to perceive, so that the display device can freely and quickly switch between full-screen and multi-window projection interfaces according to the usage scenario, thereby improving the user experience. Specifically, as shown in Figure 33, it includes the following steps:

[0242] S3301: In response to a full-screen instruction for displaying the first projection window in full screen, obtain a screen resolution.

[0243] When the display device displays the multi-screen projection interface, the user can enter a full-screen command by clicking the button control in the multi-screen projection interface. For example, if the first projection window is the window on the left side of the multi-screen projection interface, then when the multi-screen projection interface is displayed, the display device can display a "full screen" button control below the first projection window in accordance with the display process of the button control provided in the above embodiment. If the user wants to display the left window in full screen and not watch the right window temporarily, the user can perform interactive operations through the full-screen button control below the left window. The user can use the direction keys of the remote control to control the focus cursor to move to the "full screen" button control, and press the "confirm" key on the remote control to enter the full-screen command.

[0244] In this regard, the Recycleview component of the playback control bar in the display device will monitor the user's operations in real time. When it detects that the operation type mType triggered by the user is the fullscreen operation type and the user triggers the OnClick event, it will modify DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132] to display the selected projection screen in full screen.

[0245] Users can also enter full-screen commands in other ways. For example, when the focus cursor selects the first projection window, the user can long press the "Confirm" key to control the display device to display the projection screen corresponding to the first projection window in full screen, that is, enter the full-screen command by long pressing the Confirm key; for display devices that support touch interaction operations, users can directly click the "Full Screen" button control by touch to enter the full-screen command; for display devices that support voice interaction operations, users can enter the full-screen command by inputting the voice of "Display the left window in full screen".

[0246] In some embodiments, the full-screen instruction can also be automatically input by the display device through its judgment of the display scene. For example, when displaying a multi-screen projection interface, the display device can record the display time of the projection images in each projection window. When the time for displaying a specific projection image in the first projection window exceeds a preset duration threshold, and the specific projection image is not displayed in the second projection window (displaying the guidance interface), the display device can automatically generate a full-screen instruction and automatically display the first projection window in full screen.

[0247] After obtaining a full-screen instruction for full-screen display of the first projection window, the display device can obtain the screen resolution in response to the full-screen instruction. The screen resolution can be the resolution of the display or the resolution of the effective display area in the user interface. For example, for a display device where the entire screen is an effective display area, the screen resolution of 1920×1080 of the display can be directly obtained after obtaining the full-screen instruction. For a special-shaped screen with a camera module on the top, the total resolution of the display is 1920×1180, and the height occupied by the camera module is 100. Then, after obtaining the full-screen instruction, the display device can obtain the screen resolution of the effective display area after removing the camera module area 100, that is, 1920×1080.

[0248] It should be noted that the first projection window and the second projection window in the embodiment of the present disclosure are only used to distinguish different projection windows, and the first projection window is located on the left side of the multi-way projection interface and the second projection window is located on the right side of the multi-way projection interface for example. It should be understood that the first projection window and the second projection window do not limit the specific projection window display position in the multi-way projection interface. The first projection window only represents the window that needs to display the projection screen in full screen, and the second projection window only represents the window that does not need to display the projection screen in full screen. Therefore, the first projection window can also refer to the window located on the right side of the multi-way projection interface, and the second projection window can also refer to the window located on the left side of the multi-way projection interface.

[0249] In some embodiments, to indicate which window is being displayed full screen, the display device may further set a conversion mode parameter for the window. Specifically, in response to a full screen instruction for displaying the first projection window in full screen, the display device may set the conversion mode parameter for the first projection window to a first parameter value, and set the conversion mode parameter for the second projection window to a second parameter value. The first parameter value indicates that the projection window is being displayed full screen, while the second parameter value indicates that the projection window is being hidden.

[0250] For example, when the mode management module of the display device switches between dual-screen projection mode and full-screen mode, a SwitchMode attribute can be set. When SwitchMode is full-screen mode, the parameter is the first parameter value full; when SwitchMode is hidden mode, the parameter is the second parameter value Hidden; when SwithchMode is two-way screen projection window mode, the parameter is the third parameter value double, which is used to characterize various logics and display processing during the switching process.

[0251] S3302: Set the first target display area of ​​the first projection window according to the screen resolution, and set the second target display area of ​​the second projection window.

[0252] After obtaining the screen resolution, the display device can display the projection image of the first projection window in full screen. However, since the screen resolution of the display device is inconsistent with the size of the projection image corresponding to the first projection window, the display device needs to display the projection image of the first projection window according to the screen resolution.

[0253] In some embodiments, when setting the first target display area, the display device may obtain the screen resolution of the first projected image; read the screen width and screen height from the screen resolution; then determine a reference size based on the screen resolution and the image resolution, where the reference size is the screen width or the screen height. Based on the reference size, a scaling ratio is calculated. Finally, the first projected image is scaled according to the scaling ratio.

[0254] For example, the video stream size of the first terminal device is 512×1088, while the screen resolution of the display device is 1920×1080, and the guide page display size is 954×538. At this time, the video stream ratio is different from the guide video stream ratio on the right and the direction is different. The benchmark size can be set according to the video height. Considering the page layout and the right window ratio, the left video stream display height can be set to 848, and the scaling ratio is calculated according to the display height 848 and the original video height, that is, 848 / 1088. Then, the display width is calculated according to this scaling ratio, that is, 512×848 / 1088=399.

[0255] In order to improve the switching speed of the display device when switching from full-screen display of the first projection screen to displaying the multi-channel projection interface, the display device can temporarily not close the second projection window while displaying the first projection screen in full screen, but hide the projection window. Therefore, the display device can set the first target display area of ​​the first projection window according to the screen resolution to achieve the effect of full-screen display of the first projection screen, and at the same time, set the second target display area of ​​the second projection window. Among them, the second target display area can complete the picture display with fewer pixels, so that the user cannot perceive the existence of the second projection screen. Therefore, the number of pixels contained in the second target display area is less than or equal to the preset pixel threshold. The preset pixel threshold is the maximum number of pixels that the display device can not be distinguished by the user at a preset viewing distance when displaying the picture. For example, for display devices such as TVs, the user's viewing distance is 1-3m. At this viewing distance, pictures with less than 4 pixels will not be perceived by the user, so the preset pixel threshold is 4.

[0256] In some embodiments, when calculating the second target display area, the display device may determine a boundary area based on the first target display area, where the boundary area is the area outside the first target display area within the entire display area of ​​the display, or the area formed by the edge pixels of the first target display area. A minimum number of pixels in the second target display area is then set based on a preset pixel threshold, and the second target display area is demarcated within the boundary area based on the minimum number of pixels.

[0257] For example, the display device sets the first target display area to 1920×1080 and the second target display area to 1×1. The first target display area can display the first projection screen in full screen. At the same time, the second projection screen is displayed by a single pixel. This single pixel does not affect the display of the first projection screen, and the single pixel for displaying the second projection screen can be set to an area that is not easily perceived by the user, such as the upper left corner, upper right corner, lower left corner, or lower right corner of the display.

[0258] S3303: Control the display to display the first projection screen in the first target display area, and control the display to display the second projection screen in the second target display area.

[0259] After setting the first target display area and the second target display area, the display device can display the first projection screen and the second projection screen respectively through the first target display area and the second target display area. Among them, the first target display area includes more pixels, for example, the first target display area can be the entire display screen area, or a larger proportion of the middle display area, etc. Therefore, the effect of full-screen display of the first projection screen can be achieved. The second target display area is less pixel points, for example, the first target display area is one pixel point, or two pixels point, etc. Therefore, the effect of hiding the second projection screen can be achieved.

[0260] For example, in the video stream projected by the first terminal device 510, the projected screen size is 1920×1080. According to the interface switching display mode provided in the above embodiment, it can be determined that the first target display area is 1920×1080; the second target display area is 1×1. At this time, the second terminal device can be ignored, and the window of the first target display area defaults to the entire screen size of 1920×1080. The full-screen first target display area is calculated based on the video stream ratio ratio1 and the ratio of the entire screen ratio_fullscreen, and then combined with the horizontal and vertical versions of Client1, it is determined that the current video ratio is consistent with the screen ratio.

[0261] Therefore, the display area size of the first target display area should be 1920×1080, that is: Session:SessionClient1,View:SurfaceView / TextureView,MViewPosition:-1,Position:(0,0),Width:1920,Height:1080,SwitchMode:full... Based on the same logic, the second target display area is: Session:SessionClient2,View:SurfaceView / TextureView,MViewPosition:-1,Position:(0,0),Width:1,Height:1,SwitchMode:Hidden...

[0262] Since the first target display area is displayed in full screen, when the second target display area is displayed through one pixel, the display position can be displayed in any corner without affecting the user's viewing. For example, the display position of the second projection window can be (0,0), or (1919,0), (0,1079), (1919,1079) and other positions.

[0263] After setting the coordinates and width and height of the first target display area and the second target display area, the algorithm processing module notifies the data management module through the message processing module to organize the data, and then notifies the View management module. After receiving the data information, the View management module sets the display parameters through LayoutParams, including the position coordinates from the upper and left boundaries of the display, and the corresponding display width and height, and then displays the positions of the two target display areas through the setLayoutParams method.

[0264] For the first projection window: FrameLayout.LayoutParamslp1 = newFrameLayout.LayoutParams(width, height, 0); lp1.width = 1920; lp1.height = 1080; lp1.leftMargin = 0; lp1.topMargin = 0; mPlayerView.setLayoutParams(lp1).

[0265] For the second projection window: FrameLayout.LayoutParamslp1 = newFrameLayout.LayoutParams(width, height, 0); lp1.width = 1; lp1.height = 1; lp1.leftMargin = 0; lp1.topMargin = 0; mPlayerView.setLayoutParams(lp1).

[0266] In the above process, the display device can obtain the screen resolution after obtaining the full-screen instruction, and set the first target display area according to the screen resolution to achieve the display effect of full-screen display of the first projection screen. At the same time, the second target display area containing pixels smaller than the preset pixel threshold is set to display the second projection screen with fewer pixels, thereby achieving the effect of hiding the second projection screen. The first projection screen and the second projection screen are then displayed based on the first target display area and the second target display area respectively, so as to achieve the purpose of full-screen display of one projection screen and hiding the other projection screen. After switching to the full-screen state, the projection connection relationship between the display device and the second terminal device 520 is not closed. When switching back to the multi-projection screen interface, there is no need to re-establish the projection connection, and the multi-projection screen interface can be quickly presented.

[0267] In some embodiments, the display device can also obtain the first resolution of the first projection screen and the second resolution of the second projection screen in response to the multi-window instruction for exiting the full-screen display of the first projection window when switching back to the multi-projection window state in the full-screen state; reset the first target display area according to the first resolution, and reset the second target display area according to the second resolution; control the display to display the first projection screen in the first target display area after the reset, and control the display to display the second projection screen in the second target display area after the reset.

[0268] Since the multi-screen projection interface includes the first projection window and the second projection window, when switching back to the multi-screen projection interface, the display device needs to reset the first target display area and the second target display area. The method of resetting the first target display area and the second target display area can be the same as the multi-window display method provided in the above embodiment, that is, obtaining the first resolution of the first projection screen and the second resolution of the second projection screen, and then resetting the first target display area according to the first resolution, and resetting the second target display area according to the second resolution.

[0269] For example, when the display device displays the first projection screen in full screen, the user presses the "return" button on the remote control to enter the multi-window command to control the display device to exit the full-screen state and switch back to the multi-way projection interface. In response to the multi-window command, the display device can obtain the resolution of the projection screens of the first terminal device 510 and the second terminal device 520 respectively, that is, for the two horizontal projection screens, the resolution of the first projection screen and the second projection screen are both 1920×1080. At this time, the display device can reset two display areas with a size of 954×538 according to the resolution of the first projection screen and the second projection screen, and the reset data content of the display area is:

[0270] For the first projection window: FrameLayout.LayoutParamslp1 = newFrameLayout.LayoutParams(width, height, 0); lp1.width = 954; lp1.height = 538; lp1.leftMargin = 0; lp1.topMargin = 271; mPlayerView.setLayoutParams(lp1).

[0271] For the second projection window: FrameLayout.LayoutParamslp1 = newFrameLayout.LayoutParams(width, height, 0); lp1.width = 954; lp1.height = 538; lp1.leftMargin = 966; lp1.topMargin = 271; mPlayerView.setLayoutParams(lp1).

[0272] In this way, when switching back to the multi-way projection interface, the display device can directly complete the switch by resetting the first target display area and the second target display area, and in the multi-way projection interface switched back, the second projection window directly displays the projection screen, without the need to re-establish the projection connection with the second terminal device 520, nor the need to display the guide page, thereby improving the interface switching speed.

[0273] To improve the interface switching speed, the display device may further record the display size of each projection window in the multi-projection interface. That is, the display device may respond to a projection instruction for displaying the multi-projection interface and set the size of the first projection window and the size of the second projection window, i.e., the first window size and the second window size, according to the screen resolution and the screen size of the first projection screen and the second projection screen. The first window size and the second window size are then recorded.

[0274] Among them, when setting the first window size and the second window size, the display device can be dynamically set according to the screen size of the projection screen in the manner provided in the above embodiment to improve the space utilization of the display area. That is, the display device can obtain the first aspect ratio of the first projection screen and the second aspect ratio of the second projection screen. If the first aspect ratio is not equal to the second aspect ratio, the first window size of the first projection window is set according to the first aspect ratio, and the second window size of the second projection window is set according to the second aspect ratio. Then control the display to display the first projection window according to the first window size; and control the display to display the second projection window according to the second window size. Record the first window size and the second window size.

[0275] When setting the first window size and the second window size, the display device can align and display the first projection window and the second projection window in the manner in the above-mentioned embodiment, that is, the window heights of the first window size and the second window size are equal, or the window widths of the first window size and the second window size are equal.

[0276] After recording the first window size and the second window size, the display device can directly obtain the first window size and the second window size when resetting the first target display area according to the first resolution and resetting the second target display area according to the second resolution, and set the first target display area according to the first window size, and set the second target display area according to the second window size. The first window size is the window size of the first projection window recorded by the display device when displaying the multi-screen projection interface; the second window size is the window size of the second projection window recorded by the display device when displaying the multi-screen projection interface;

[0277] In some embodiments, after the display device sets the conversion mode parameters for the projection window, the display device can synchronously modify the conversion mode parameters after switching back to the multi-screen projection interface. That is, in response to the multi-window instruction for exiting full-screen display of the first projection window, the display device can set the conversion mode parameters of the first projection window and the second projection window to a third parameter value, where the third parameter value represents the multi-window display projection window.

[0278] For example, after switching back to the multi-screen projection interface, the display device can modify the SwitchMode parameter of the first projection window from the first parameter value full to the third parameter value double, and at the same time, modify the SwitchMode parameter of the second projection window from the second parameter value Hidden to the third parameter value double, which is used to represent various logic and display processing during the switching process.

[0279] In some embodiments, after switching to full-screen mode, the display device may also record the full-screen display duration of the first projection image and determine the full-screen display duration. If the full-screen display duration is greater than or equal to a preset duration threshold, that is, after the user switches to full-screen display of the first projection image and watches it for a long time, the probability of switching back to the multi-screen projection interface is low. Therefore, in order to conserve the computing resources of the display device, the projection connection channel with the second terminal device 520 may be closed, thereby controlling the display to stop displaying the second projection image in the second target display area.

[0280] For example, when a user clicks the full-screen button control below the first projection window to control the display device to display the first projection screen in full screen, the display device can record the full-screen display duration of the first projection screen. When the full-screen display duration exceeds 30 minutes, the projection connection channel with the second terminal device 520 can be directly closed, so that the display device only displays the first projection screen, thereby reducing the resource usage of the second projection screen processing process.

[0281] In some embodiments, after the display device sets the first target display area of ​​the first projection window according to the screen resolution, it can also detect the display content of the second projection window; if the display content of the second projection window is a guide screen, the second projection window is closed; if the display content of the second projection window is a projection screen, the step of setting the second target display area of ​​the second projection window is executed.

[0282] The display device switches modes from a single client mode, with one channel casting the screen image and the other channel guiding the page, as shown in Figure 25 above. At this point, the display device is in dual-channel projection mode. After the user selects the RecycleView control in the playback control on the first and terminal devices, they need to display the first target display area in full screen. The View Management Module synchronizes the user operation to the Mode Management Module. The Mode Management Module changes the SwitchMode parameter from double to full and synchronizes it to the Data Management Module for data update. The data update process involves switching the data of the first target display area: Session: SessionClient1, View: SurfaceView / TextureView, MViewPosition: 1, Position: (0, 272), Width: 954, Height: 537, SwitchMode: double… The data after switching can be expressed in the following form: Session: SessionClient1, View: SurfaceView / TextureView, MViewPosition: 1, Position: (0, 0), Width: 1920, Height: 1080, SwitchMode: full…

[0283] At this point, the right-side projection window displays the guide page, not the target display area. Therefore, there's no need for micro-pixel display processing, and playback isn't involved. The right-side Guide View can be directly INVISIBLEed. That is, after receiving the data, the View Management Module sets the display parameters using LayoutParams. These parameters include the position (coordinates) from the top and left edges, as well as the corresponding display width and height.

[0284] Based on this, the position of the first target display area can be displayed through the setLayoutParams method. That is, FrameLayout.LayoutParamslp1 = newFrameLayout.LayoutParams(width, height, 0); lp1.width = 1920; lp1.height = 1080; lp1.leftMargin = 0; lp1.topMargin = 0; mPlayerView.setLayoutParams(lp1); Then, the layout of the right guide page can be set to the INVISIBLE state through the View's setVisibility method, as shown in Figure 34.

[0285] Similarly, if you need to switch the full-screen first projection screen back to the multi-screen projection interface, you can also trigger the window switch through specific interaction methods, such as the menu key or any other interaction, such as voice. For example, the default size of the guide page is 954×538, that is, Guide:954×538; the display size of the first target display area is 1920×1080, that is, Client1:1920×1080.

[0286] Through the algorithm provided in the above embodiment, the current video and the guide page can be displayed in a horizontal version of 1:1, the full-screen mode data can be switched to two-window data, and then the SwitchMode parameter is synchronously changed from full to double. After the View management module receives the data information, the display parameters are set through LayoutParams, including the position from the upper boundary and the left boundary, that is, the coordinates, and the corresponding display width and height.

[0287] After setting the display parameters, you can use the setLayoutParams method to display the position of the first target display area. That is, FrameLayout.LayoutParamslp1 = newFrameLayout.LayoutParams(width, height, 0); lp1.width = 1920; lp1.height = 1080; lp1.leftMargin = 0; lp1.topMargin = 0; mPlayerView.setLayoutParams(lp1); Then, use the View's setVisibility method to set the layout of the right guide page to the VISIBLE state.

[0288] Another consideration is that during the audio transmission process of multi-channel screen projection, if the audio playback of multi-channel screen projection is not controlled, the audio of multiple screen projections will be played at the same time, resulting in sound conflicts, thereby reducing the user experience.

[0289] Based on this, the display device of the embodiment of the present disclosure can also maintain a global projection list (SessionList) for managing the access and removal operations of the Session, that is, the projection list is used to record the terminal devices 500 that establish projection connections with the display device, and the connection order in which the terminal devices 500 establish projection connections with the display device. The display device can adjust the sound mode of multiple projections through the projection list, set the sound mode of one terminal device 500 in the projection list to non-silent mode, and set the other terminal devices 500 to silent mode, so that the display device only plays the sound of one projection.

[0290] Among them, the sound management module of the display device can set an audio parameter when setting the sound mode. When the sound mode is non-silent mode, the audio parameter can be a first audio parameter, and the first audio parameter is used to indicate that audio data is allowed to be played. When the sound mode is silent mode, the audio parameter can be a second audio parameter, and the second audio parameter is used to indicate that audio data is not allowed to be played. This is used to characterize various logics and display processing during the sound mode switching process.

[0291] Setting the sound mode of a terminal device 500 in the screen projection list to silent mode can be achieved in a variety of ways. After the display device establishes a screen projection connection with the terminal device 500, when the display device obtains the screen projection data containing video data and audio data, it can filter out the audio data and only receive the video data, thereby muting the sound of the screen projection. The display device can also receive video data and audio data, but does not play the audio data through the audio output interface 270, thereby muting the sound of the screen projection. The display device can also, based on the screen projection protocol, cause the terminal device 500 that establishes a screen projection connection with the display device not to send audio data, thereby muting the sound of the screen projection. For example, in some screen projection protocols, it can be set to transmit only video data, so that the terminal device 500 only sends video data and does not send audio data.

[0292] In some embodiments, the display device may set the sound mode of the first projection connection to a non-mute mode, i.e., the terminal device 500 (Client) that has the priority to take over the sound will keep the sound. Alternatively, the display device may set the sound mode of the last projection connection to a non-mute mode, i.e., the terminal device 500 (Client) that has the most recent projection connection to keep the sound.

[0293] Through the above process, the display device of the embodiment of the present disclosure can record the terminal devices that establish a screen projection connection with the display device based on the screen projection list, as well as the connection order of the terminal devices and the display device to establish the screen projection connection. When the display device establishes a screen projection connection with at least two terminal devices, the sound mode of the multi-channel screen projection is adjusted through the screen projection list, and the sound mode of one terminal device in the screen projection list is set to non-silent mode, and the other terminal devices are set to silent mode, so that the display device only plays the sound of one channel of screen projection. In this way, the sound can be dynamically set according to different screen projection scenarios, and the sound can also be dynamically controlled according to the user's independent choice, solving the problem of sound conflict during multi-channel screen projection.

[0294] Figure 35 is a schematic diagram of the sound playback control process according to some embodiments. As shown in Figure 35, when the display device has no screen projection connection, it can respond to a connection event for establishing a screen projection connection relationship between the display device and the fifth terminal device and execute step S3501: set the audio parameter of the fifth terminal device to the first audio parameter. When the display device has a screen projection connection, it can respond to a connection event for establishing a screen projection connection relationship between the display device and the sixth terminal device and execute step S3502: set the audio parameter of the fifth terminal device to the second audio parameter, and set the audio parameter of the sixth terminal device to the first audio parameter, or execute step S3503: set the audio parameter of the sixth terminal device to the second audio parameter. Among them, the fifth terminal device is the terminal device 500 whose current audio parameter is the first audio parameter.

[0295] For example, when UsessionConnectionListener listens to a Session connection (i.e., connected to the fifth terminal device Client5), SessionList adds Client5 and sets the sound mode of Client5 to non-mute mode through Session's setAudioOutputState, that is, the first screen projection makes a sound. At this time, if UsessionConnectionListener listens to another Session connection (i.e., connected to the sixth terminal device Client6), SessionList adds Client6. Because the first screen projection is sound-activated, if the two channels make sound at the same time, it will cause sound confusion. Therefore, after the second Session is connected, the sound mode of Client6 will be set to mute mode through Session's setAudioOutputState, that is, the second screen projection does not make a sound.

[0296] In order to facilitate users to distinguish between the projection end that makes sound and the projection end that does not make sound, the display device can create a sound view in response to a connection event that establishes a projection connection relationship between the display device and the terminal device 500. The sound view is a control for performing playback control on audio data in the projection interface. The sound view may include a first icon and a second icon, wherein the first icon is used to represent a non-silent mode and the second icon is used to represent a silent mode. When the sound mode of the terminal device 500 in the projection list is a non-silent mode, the control display displays the first icon of the sound view. When the sound mode of the terminal device 500 in the projection list is a silent mode, the control display displays the second icon of the sound view, thereby intuitively presenting the sound mode of the current projection to the user through the icon of the sound view.

[0297] To facilitate switching of the sound view icon, an association relationship may be established between the state identifier of the sound view and the audio parameter of the terminal device 500. That is, the display device may read the audio parameter of the terminal device 500 in the screen projection list. If the audio parameter is the first audio parameter, the state identifier of the sound view is set to the first identifier value. If the audio parameter is the second audio parameter, the state identifier of the sound view is set to the second identifier value.

[0298] When displaying the sound view, the display device may read a status identifier of the sound view, and control the display to display a first icon of the sound view if the status identifier is a first identifier value, and control the display to display a second icon of the sound view if the status identifier is a second identifier value.

[0299] For example, using the above embodiment, when UsessionConnectionListener detects a session connection, it adds Client5 to the SessionList and sets Client5's audio mode to non-mute mode via the Session's setAudioOutputState method, effectively emitting sound from the first projection. As shown in Figure 36, the first projection content is displayed via the left projection window. Based on the display area of ​​the left projection window, playback control views are created and displayed, including full screen, off, mute / non-mute, play / pause, fast forward, and rewind. Among them, since the sound mode of the projection window on the left is non-silent mode, the status identifier of the sound view representing "silent / non-silent" in the play control view is set to the first identifier value, that is, mStatus = false, indicating that the current mode is non-silent, that is, the play control list data of Client5 is as follows: DoublePushControlBean[mStatus = true, mType = fullscreen, mResId = 2131231132], DoublePushControlBean[mStatus = true, mType = rewind, mResId = 2131231110], DoublePushControlBean[mStatus = true, mType = playstatus, mResId = 2131231149], DoublePushControlBean[mStatus = true, mType = forward, mResId = 2131231111], DoublePushControlBean[mStatus=false,mType=mutestatus,mResId=2131231152], DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121]

[0300] At this time, if UsessionConnectionListener detects another Session connection (i.e., the sixth terminal device), Client6 will be added to SessionList. Because the first projection is sound-enabled, if both channels emit sound at the same time, it will cause sound confusion. Therefore, after the second Session is connected, the sound mode of Client6 will be set to silent mode through the Session's setAudioOutputState, that is, the second projection will not emit sound. As shown in Figure 36, the second projection content is displayed through the right projection window. According to the display area of ​​the right projection window, the full screen, off, mute / unmute, play / pause, fast forward, and fast rewind playback control views are created and displayed. Among them, since the sound mode of the projection window on the right is silent mode, the status identifier of the sound view representing "silent / non-silent" in the play control view is set to the second identifier value, that is, mStatus=true, indicating that it is currently in silent mode, that is, the play control list data of Client6 is as follows: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132], DoublePushControlBean[mStatus=true,mType=rewind,mResId=2131231110], DoublePushControlBean[mStatus=true,mType=playstatus,mResId=2131231149], DoublePushControlBean[mStatus=true,mType=forward,mResId=2131231111], DoublePushControlBean[mStatus=true,mType=mutestatus,mResId=2131231152], DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121]

[0301] According to the above-mentioned play control list data of Client5 and Client6, the icon display status of the mutestatus type view in the Recycleview of the left projection window and the right projection window is set respectively, and the display of the play control view is set by setBackgroundResource, presenting the display effect shown in Figure 36, wherein the sound view below the left projection window displays the first icon, and the sound view below the right projection window displays the second icon.

[0302] When the display device displays the sound view, the user can control the sound mode of the corresponding projection window based on the sound view. That is, the display device can monitor the user's operation through the onClick event of RecycleView and mute / unmute the selected client based on the user's selection, and perform the opposite operation on the unselected clients.

[0303] For example, in the sound view shown in Figure 36, the user can use the direction keys on the remote control compatible with the display device to control the focus cursor to move to the "Mute / Unmute" playback control function button below the left projection window, and press the "Confirm" key on the remote control to enter the mute command. At this time, Recycleview will monitor the user's onClick operation, set the sound mode of Client5 from non-mute mode to mute mode through Session's setAudioOutputState, and set the sound mode of Client6 from mute mode to non-mute mode. At the same time, the display of the playback control view is set through setBackgroundResource, presenting the display effect shown in Figure 37, where the sound view below the left projection window displays the second icon, and the sound view below the right projection window displays the first icon.

[0304] In some embodiments, when a screen projection with a non-silent sound mode in the projection list is disconnected, the sound can be dynamically switched according to the projection list. That is, the display device can respond to the exit event of disconnecting the projection connection between the display device and the fifth terminal device by removing the fifth terminal device from the projection list, determining the seventh terminal device in the projection list according to the connection order, and setting the audio parameter of the seventh terminal device to the first audio parameter. Among them, the fifth terminal device is the terminal device 500 whose current audio parameter in the projection list is the first audio parameter.

[0305] For example, taking the two-way projection shown in Figure 36 as an example, Client5 in the first projection has sound. At this time, Client5 is disconnected and Client5 in the SessionList is removed. Only Client6 remains in the SessionList. At this time, Client6 can be unmuted. First, the sound of the second projection is unmuted through the setAudioOutputState method of Session in the projection management module. This message is notified to the data management module through the message management module. The data management module updates the playlist data of Client6. The updated data is as follows: DoublePushControlBean[mStatus=false,mType=mutestatus,mResId=2131231152]

[0306] After the data update is completed, the message processing module notifies the View management module of the message, and the View management module performs dynamic refresh display processing of Recycleview, presenting the display effect shown in Figure 38.

[0307] The above implementation can realize dynamic switching of sound, but in some scenarios, users do not want to switch the sound. For example, taking the two-way screen projection shown in Figure 36 as an example, Client5 in the first-way screen projection is in non-silent mode, and Client6 in the second-way screen projection is in silent mode. During the screen projection process, if the media type of Client5 in the first-way screen projection is push type, it may switch the episodes of the film and television series. During the episode switching process, there will be a session switching situation depending on the differences between different mobile terminals, which is equivalent to a new Client access. There are also some mobile terminals that perform mirror projection and then switch to push type. In this case, there will also be a new session access, which is equivalent to a new Client access.

[0308] In the above scenario, according to the previous processing logic, after the Client5 in the first projection is disconnected, the display device will unmute the Client6 in the second projection to switch the sound. When UsesionConnectionListener monitors a new Session connection, based on the principle of giving priority to the projection that holds the sound, the display device may mute the newly connected third projection (i.e., the seventh terminal device Client7). In other words, according to the above processing logic, it may cause the sound mode to switch from non-muted mode to silent mode after the number of episodes is switched or the media type is changed in one projection. But for the user, the user's goal is to continue playing in the previous sound mode after the number of episodes is switched or the media type is changed in one projection, that is, Client7 in the third projection continues to use the sound mode of Client5 in the first projection.

[0309] Therefore, the display device can set a time threshold, and the sound will not be switched within the time threshold of the exit event, and the sound can be switched after the time threshold of the exit event. The implementation can be shown in Figure 39, including the following steps:

[0310] S3901, in response to an exit event of disconnecting the screen projection connection relationship between the display device and the fifth terminal device, removing the fifth terminal device from the screen projection list, and recording the exit time of the exit event;

[0311] S3902: Whether a connection event for establishing a screen projection connection relationship between the display device and the sixth terminal device is detected within a preset time threshold after the exit time;

[0312] S3903: If a connection event is detected for establishing a screen projection connection relationship between the display device and the sixth terminal device, the sixth terminal device is added to the screen projection list;

[0313] S3904, setting the audio parameter of the sixth terminal device to the first audio parameter;

[0314] In some embodiments, after the audio parameter of the sixth terminal device is set to the first audio parameter, if a connection event is detected for establishing a screen projection connection relationship between the display device and the eighth terminal device, the audio processing logic may continue to be pre-set, that is, the display device may set the audio parameter of the sixth terminal device to the second audio parameter and the audio parameter of the eighth terminal device to the first audio parameter in response to the connection event for establishing a screen projection connection relationship between the display device and the eighth terminal device. Alternatively, the audio parameter of the eighth terminal device may be set to the second audio parameter.

[0315] S3905: If no connection event is detected to establish a screen projection connection relationship between the display device and the sixth terminal device, determine the seventh terminal device in the screen projection list according to the connection order;

[0316] S3906: Set the audio parameter of the seventh terminal device to the first audio parameter.

[0317] There are multiple ways to determine the seventh terminal device in the screen projection list based on the connection order. In some embodiments, the display device can select the terminal device 500 that establishes a screen projection connection with the display device earliest in the screen projection list according to the connection order as the seventh terminal device. Alternatively, the terminal device that establishes a screen projection connection with the display device latest in the screen projection list can be selected as the seventh terminal device according to the connection order.

[0318] For the broadcast control list data, if the media type of the currently exited screen projection is the same as the media type of the newly connected screen projection within the time threshold, the broadcast control list data can be directly reused. That is, the display device can record the first media type and broadcast control list of the screen projection data of the fifth terminal device in response to the exit event of disconnecting the screen projection connection relationship between the display device and the fifth terminal device, and the broadcast control list includes the broadcast control data of the broadcast control view associated with the first media type. If a connection event is detected to establish a screen projection connection relationship between the display device and the sixth terminal device within the preset time threshold after the exit time, the second media type of the screen projection data of the sixth terminal device is obtained. If the first media type is the same as the second media type, the broadcast control list is set for the sixth terminal device.

[0319] For example, using the two-way projection shown in Figure 36, if Client5 disconnects via UsesessionConnectionListener, SessionList will remove Client5. After removal, the CountTimeUtils timer is started, set to 2 seconds, and the audio switching method is locked within 2 seconds. The audio switching operation between the left and right projection windows will not be performed, which means that the Session's setAudioOutputState method will be invalid. If a new Client7 connects within 2 seconds, SessionList will add Client7 and perform View layout operations based on Client7's VideoChanged. At the same time, Client7 will continue to use the original Client5 audio mode. If the media asset type of Client 7 is the same as that of Client 5, the play control list data is reused. The play control list data corresponding to Client 7 is as follows: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132], DoublePushControlBean[mStatus=true,mType=rewind,mResId=2131231110], DoublePushControlBean[mStatus=true,mType=playstatus,mResId=2131231149], DoublePushControlBean[mStatus=true,mType=forward,mResId=2131231111], DoublePushControlBean[mStatus=false,mType=mutestatus,mResId=2131231152], DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121]

[0320] If UsessionConnectionListener has not detected any new Client7 access after the 2S timer expires, the sound of Client6 in SessionList will be reset, and the sound mode of Client6 will be set to non-silent mode through the setAudioOutputState method of Session. The data will be synchronized to the data management module for updating, and then synchronized to the View management module through the message processing module to refresh the View display.

[0321] To improve the display effect, the display device may further record the display position parameters of a target projection window in response to an exit event that disconnects the projection connection between the display device and the fifth terminal device, where the target projection window is the projection window displaying the projection image of the fifth terminal device. If a connection event is detected within a preset time threshold after the exit time that establishes a projection connection between the display device and the sixth terminal device, the display is controlled to display the projection image of the sixth terminal device in the target projection window according to the display position parameters.

[0322] For example, as shown in Figure 38, the media assets of Client 6 in the second projection are displayed in the right projection window. When Client 6 disconnects via UsesexConnectionListener, SessionList removes Client 6 and records the display position parameters of the right projection window. After removal, it starts the CountTimeUtils timer with a duration of 2 seconds. If Client 7 connects within 2 seconds, SessionList adds Client 7 and sets its display position according to the recorded display position parameters, so that Client 7's media assets are displayed in the right projection window.

[0323] In some embodiments, the display device also supports locking the sound mode of the projection window. After the sound mode of a projection window is locked, the projection window can always maintain the sound mode before the current lock without switching the sound. In this regard, the projection management module of the display device monitors the Session and sets the sound mode through the sound management module. After the user operates the View, the View management module monitors the user action and then refreshes the View. At the same time, the message processing module notifies the data management module to update the data. When the projection management module monitors the exit event of the projection connection relationship, it notifies the time management module through the message processing module to monitor the time to determine whether to switch the sound. At the same time, the above operations are all dynamic sound switching when the mode management module notifies the current projection window that the sound mode is not locked. If the sound mode of the current projection window is locked, the locked projection window will not switch the sound, and the projection window on the other side will always maintain the default sound mode for display and will not switch the sound mode.

[0324] Therefore, the display device can respond to the lock instruction of the specified projection window, record the current target audio parameters of the specified projection window, and set a lock value for the specified projection window. The lock value indicates that the window state of the specified projection window is locked, which is used to indicate that the audio parameters of the terminal device displaying the projection screen in the specified projection window are set to the target audio parameters.

[0325] When the display device receives a display instruction to display the projection screen of the target terminal device in a specified projection window, it can respond to the display instruction and read the lock value of the specified projection window to detect the window state of the specified projection window. If the window state is locked, the target audio parameters of the specified projection window are obtained, and the audio parameters of the target terminal device are set to the target audio parameters. The target audio parameters are the audio parameters of the terminal device whose projection screen is displayed in the specified projection window when the window state is set to the locked state.

[0326] For example, taking the two-way screen projection shown in Figure 37 above as an example, the audio mode of Client 5 in the first screen projection is silent mode, and Client 6 in the second screen projection is non-silent mode. At this time, the user long-presses the "Voice" play control function control below the left screen projection window shown in Figure 37 to input the lock command for the left screen projection window. In response to this lock command, the display device can lock the current silent mode of the left screen projection window and set the mBlockStatus in the current DoublePushControlBean of the mutestatus type to true. That is, after the View management module monitors the user's long-press action, it refreshes the View, presenting the display effect shown in Figure 40, with the third icon displayed in the audio view below the left screen projection window. The message processing module then notifies the data management module of this operation, and the data management module sets mBlockStatus to true. After the setting is successful, the audio mode of the locked left screen projection window is remembered, that is, the global memory mIsPlayControlVoiceBlocked is true and mIsPlayControlVoiceBlockedLeft is true. The playback control list data of Client5 is as follows: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132], DoublePushControlBean[mStatus=true,mType=rewind,mResId=2131231110], DoublePushControlBean[mStatus=true,mType=playstatus,mResId=2131231149], DoublePushControlBean[mStatus=true,mType=forward,mResId=2131231111], DoublePushControlBean[mStatus=true,mType=mutestatus,mBlockStatus=true,mResId=213 1231152], DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121]

[0327] After the above operation, the sound mode of the left projection window will not change regardless of whether new clients are added or disconnected. If Client 6 is disconnected, Client 6 in the SessionList is removed. At this time, although only Client 5 in silent mode remains in the SessionList, the sound will not be reset because mIsPlayControlVoiceBlocked is true. And when the media image of the newly connected eighth terminal device Client8 is displayed in the left projection window, because mIsPlayControlVoiceBlocked and mIsPlayControlVoiceBlockedLeft are true, the sound mode of Client8 is still silent mode. The specific data are as follows: DoublePushControlBean[mStatus=true,mType=fullscreen,mResId=2131231132]; DoublePushControlBean[mStatus=true,mType=rewind,mResId=2131231110]; DoublePushControlBean[mStatus=true,mType=playstatus,mResId=2131231149]; DoublePushControlBean[mStatus=true,mType=forward,mResId=2131231111], DoublePushControlBean[mStatus=true,mType=mutestatus,mBlockStatus=true,mResId=213 1231152]; DoublePushControlBean[mStatus=true,mType=close,mResId=2131231121].

[0328] In some embodiments, when the screen projection is switched from a small window to a full-screen window, the sound also needs to be logically processed accordingly. The display device can respond to a full-screen instruction for full-screen display of a specified screen projection window, delete other terminal devices other than the target terminal device in the screen projection list, and disconnect the screen projection connection with other terminal devices. Among them, the target terminal device is a terminal device whose screen projection image is displayed in the specified screen projection window. Then read the audio parameter of the target terminal device. If the audio parameter is the second audio parameter, set the audio parameter of the target terminal device to the first audio parameter, and display the specified screen projection window in full screen. If the audio parameter is the first audio parameter, display the specified screen projection window in full screen.

[0329] For example, taking the two channels shown in Figure 36 above as an example, the sound mode of Client 5 in the first channel is non-mute mode, and the sound mode of Client 6 in the second channel is mute mode. At this time, as shown in Figure 41, the user clicks the "full screen" playback control function control below the right screen projection window shown in Figure 36. First, remove Client 5 from the SessionList, and then remove the View corresponding to Client 5 from the ViewList of the display window. Then reset the display window and set the display area and position of Client 6 according to the window size. At the same time, use the setAudioOutputState method of Session to unmute Client 6, presenting the display effect shown in Figure 41.

[0330] Through the above process, the display device can dynamically set the sound according to different projection scenarios, and can also dynamically control the sound according to the user's independent choice, thereby solving the problem of sound conflicts during multi-channel projection. It should be noted that the specific solution for sound processing is not only the above example content, but multiple projection windows and multiple projection protocols can also refer to the above rules. In addition, the sound processing can not only be muted / unmuted, but also the volume can be set, such as entering the secondary menu through the sound view or adding other display buttons to adjust the volume.

Claims

1. A display device, comprising: A display, configured to display a multi-channel screen projection interface, the multi-channel screen projection interface comprising a first guide window and a second guide window, the first guide window and the second guide window are used to display a guide page; the first guide window, the second guide window and the root node of the multi-channel screen projection interface are the same; A user input interface configured to receive instructions from a user; A communication device configured to establish a communication connection with a terminal device; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, the user input interface, the communication device and the memory, is configured to execute computer instructions to cause the display device to perform: In response to the screen projection connection instruction sent by the first terminal device, a first screen projection window is created, the first guide window is released, and the display of the second guide window on the multi-way screen projection interface is retained; the first screen projection window is used to display the first screen projection screen of the first terminal device, and the first screen projection window is the same as the root node of the multi-way screen projection interface; Obtain a first screen projection session sent by the first terminal device, where the first screen projection session includes a first resolution of a first screen projection picture; Detecting a screen resolution and a reference resolution of the guide page; calculating a window display area according to the screen resolution, the first resolution and the reference resolution, wherein the window display area includes a first display area and a second display area, and the first display area and the second display area have the same height; The display is controlled to display the first projection window in the first display area, and to display the second guide window in the second display area.

2. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to the screen projection connection instruction sent by the second terminal device, create a second screen projection window, and release the second guide window; the second screen projection window is used to display the second screen projection screen of the second terminal device, and the second screen projection window is the same as the root node of the multi-channel screen projection interface; Acquire a second screen projection session sent by the second terminal device, where the second screen projection session includes a second resolution of the second screen projection picture; detecting the screen resolution and the first resolution; Calculate a window display area according to the screen resolution, the first resolution, and the second resolution, wherein the window display area includes a third display area and a fourth display area, and the third display area and the fourth display area have the same height; Control the display to display the first projection window in the third display area, and control the display to display the second projection window in the fourth display area.

3. The display device according to claim 2, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a screen projection termination instruction of the first screen projection session, removing the first screen projection window and calling the first guide window; detecting the screen resolution and the second resolution; The window display area is calculated according to the screen resolution, the second resolution and the reference resolution, the window display area includes a fifth display area and a sixth display area, The fifth display area is equal to the sixth display area in height; The display is controlled to display a first guide window in the fifth display area, and the display is controlled to display the second projection window in the sixth display area.

4. The display device according to claim 1, wherein the at least one processor is specifically configured to execute computer instructions so that the display device calculates the window display area according to the screen resolution, the first resolution and the reference resolution in the following manner: Read the horizontal pixel value and the vertical pixel value of the screen resolution; Calculating a first aspect ratio according to a ratio of horizontal pixel values ​​to vertical pixel values ​​in the first resolution, and calculating a second aspect ratio according to a ratio of horizontal pixel values ​​to vertical pixel values ​​in the reference resolution; Setting first layout parameters and second layout parameters, wherein the first layout parameters include a first layout width and a first layout height, and the second layout parameters include a second layout width and a second layout height; the first layout height is equal to the second layout height, the sum of the first layout width and the second layout width is equal to the horizontal pixel value of the screen resolution, the first layout height and the second layout height are less than or equal to the vertical pixel value of the screen resolution, the ratio of the first layout width to the first layout height is equal to the first aspect ratio, and the ratio of the second layout width to the second layout height is equal to the second aspect ratio; The first display area is generated according to the first layout parameters, and the second display area is generated according to the second layout parameters.

5. The display device according to claim 4, wherein the at least one processor is specifically configured to execute computer instructions so that the display device controls the display to display the first projection window in the first display area in the following manner: Generate the layout position coordinates of the first projection window according to the first layout parameters; Recording the first layout parameter and the layout position coordinates; Draw the first projection window according to the first layout parameters and the layout position coordinates.

6. The display device according to claim 4, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Reading a first preset height of an edge area, wherein the edge area includes a display area of ​​the multi-channel screen projection interface except the first display window and the second guide window; The sum of the first layout height and the first preset height is set to be equal to the vertical pixel value of the screen resolution, so as to generate the first layout height based on the first preset height.

7. The display device according to claim 6, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After controlling the display to display the first projection window, in response to a focus event of the first projection window, controlling the display to display a playback control function control in the edge area; Reading a second preset height of the edge area, wherein the second preset height is greater than the first preset height; The sum of the first layout height and the second preset height is set to be equal to the vertical pixel value of the screen resolution, so as to generate the first layout height based on the second preset height.

8. The display device according to claim 4, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After controlling the display to display the first screen projection window, calculating the window aspect ratio of the first screen projection window according to the first layout width and the first layout height; comparing the window aspect ratio with the first aspect ratio; If the window aspect ratio is not equal to the first aspect ratio, scaling is performed on the first projection picture in the first projection window.

9. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Monitoring the change event of the first screen projection session; When a change event of the first resolution is monitored, detecting a changed resolution of the first projection screen, where the changed resolution is a resolution of the first projection screen after the change event is executed; Calculating the window display area according to the screen resolution, the changed resolution and the reference resolution, the window display area including a seventh display area and an eighth display area, the seventh display area and the eighth display area having the same height; The display is controlled to display the first projection window in the seventh display area, and the display is controlled to display the second guide window in the eighth display area.

10. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a screen projection connection instruction sent by a third terminal device, detecting a screen projection replacement mode; If the screen projection replacement mode is the first mode, the focus position of the focus frame is detected, and the first replacement screen projection window at the focus position is removed, and a third screen projection window is displayed according to the orientation mark of the first replacement screen projection window; the orientation mark represents the display orientation of the window, and the third screen projection window is used to display the third screen projection screen of the third terminal device, and the third screen projection window is the same as the root node of the multi-channel screen projection interface; If the screen projection replacement mode is the second mode, detecting a locking attribute of the screen projection window, removing the second replacement screen projection window according to the locking attribute, and displaying the third screen projection window according to the orientation mark of the second replacement screen projection window; The second replacement projection window is a projection window whose locking attribute is in an unlocked state.

11. The display device according to claim 10, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Obtain the lock property of the screen projection window; If the locking attributes of the projection window are all in the unlocked state, setting the replacement mode parameter to the first parameter, where the first parameter is used to characterize that the projection replacement mode is the first mode; If the locking attribute of the screen projection window includes a locking state, the replacement mode parameter is set to a second parameter, and the second parameter is used to characterize that the screen projection replacement mode is a second mode.

12. The display device according to claim 10, wherein the at least one processor is specifically configured to execute computer instructions so that the display device displays the third projection window according to the position indicator of the first replacement projection window in the following manner: Recording the first position identifier of the first replacement screen projection window, and creating the third screen projection window; Setting the position identifier of the third projection window to the first position identifier; Draw the third screen projection window according to the first orientation identifier.

13. The display device according to claim 10, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: If the screen projection replacement mode is the second mode, traverse the lock properties of the screen projection window; If the lock attribute of the projection window does not include an unlocked state, control the communication device to send a message to the third terminal The device sends a notification that the screen projection failed.

14. The display device according to claim 13, wherein the at least one processor is specifically configured to execute computer instructions so that the display device displays the third projection window according to the position identifier of the second replacement projection window in the following manner: Recording the second position identifier of the second replacement screen projection window, and creating the third screen projection window; Setting the position identifier of the third projection window to the second position identifier; Draw the third screen projection window according to the second orientation identifier.

15. The display device according to claim 10, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Controlling the display to display a target lock control, wherein the target lock control is associated with the third projection window; In response to a selection instruction input by the user based on the target lock control, the current value of the lock control and the lock attribute of the third projection window are refreshed.

16. The display device according to claim 10, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a termination screen projection instruction of a target screen projection session, deleting the target screen projection session; the target screen projection session is a communication session generated based on the third terminal device; Recording a third position identifier of the third projection window; Remove the third screen projection window, and call the guide window, where the guide window is used to display a guide page; the root node of the guide window is the same as the root node of the multi-channel screen projection interface; The display is controlled to display the guide window according to the third position mark.

17. The display device according to claim 16, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Detecting the screen resolution, the base resolution and the projection screen resolution, wherein the base resolution is the resolution of the guide page; The projection screen resolution is the resolution of the projection screen in the multi-channel projection screen interface; The window layout parameters of the guide window are calculated according to the screen resolution, the reference resolution and the projection screen resolution; the window layout parameters include a layout width and a layout height.

18. The display device according to claim 17, wherein the at least one processor is specifically configured to execute computer instructions so that the display device controls the display to display the guide window according to the third position mark in the following manner: Generating the layout position coordinates of the guide window according to the window layout parameters and the third position identifier; Recording the window layout parameters and the layout position coordinates; The guide window is drawn according to the window layout parameters and the layout position coordinates.

19. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a connection event of establishing a screen projection connection relationship between the display device and the fourth terminal device, acquiring a media asset type of the screen projection data; Extracting broadcast control data associated with the media asset type from the broadcast control list, and creating a broadcast control view according to the broadcast control data, wherein the broadcast control list includes broadcast control data of a preset broadcast control view, and the broadcast control view is a control in the projection interface for performing playback control on the projection data; The display position of the broadcast control view is set according to the display area of ​​the projection window, and the display is controlled in the The display position displays the play control view.

20. The display device according to claim 19, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Obtaining configuration information when the display device establishes a screen projection connection with the fourth terminal device; The media resource type of the screen projection data of the fourth terminal device is read in the configuration information.

21. The display device according to claim 20, wherein the at least one processor is specifically configured to execute computer instructions so that the display device extracts the broadcast control data associated with the media asset type from the broadcast control list in the following manner: Calling a play control list, wherein the play control data in the play control list includes a state identifier, a play control type, and a resource identifier, wherein the state identifier represents a functional state of the play control view, the play control type represents a processing operation associated with the play control view, and the resource identifier represents a functional module associated with the play control view; generating a broadcast control sub-list according to the broadcast control data associated with the media asset type in the broadcast control list; A play control view is created according to the play control sub-list.

22. The display device according to claim 21, wherein the at least one processor is specifically configured to execute computer instructions so that the display device generates a play control sub-list according to the play control data associated with the media asset type in the play control list in the following manner: Copying the broadcast control list, and deleting the broadcast control data not associated with the media asset type in the broadcast control list, so as to generate a broadcast control sub-list; Alternatively, a broadcast control sub-list is created, and the broadcast control data associated with the media asset type in the broadcast control list is added to the broadcast control sub-list.

23. The display device according to claim 19, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Obtaining the area size of the display area of ​​the projection window, and obtaining the preset view size of the broadcast control view; Calculating the display quantity of the playback view according to the area size and the view size; According to the display quantity, the broadcast control data associated with the media asset type is extracted from the broadcast control list.

24. The display device according to claim 23, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to the change notification of the display area, acquiring the changed display area; The display position of the play control view is reset according to the changed display area, and the display quantity of the play view is recalculated according to the area size of the changed display area.

25. The display device according to claim 23, wherein the at least one processor is specifically configured to execute computer instructions so that the display device extracts the broadcast control data associated with the media asset type in the broadcast control list according to the display quantity in the following manner: Traversing the weight values ​​of the broadcast control data in the broadcast control list; Sorting the broadcast control data in the broadcast control list in descending order of weight values; The aforementioned displayed number of broadcast control data associated with the media asset type are extracted from the sorted broadcast control list.

26. The display device according to claim 23, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Acquire the area width of the display area and the total width of the play control bar, wherein the play control bar is obtained by arranging the play control views associated with the media asset type at a preset spacing; If the area width is greater than or equal to the total width, extracting the broadcast control data associated with the media asset type in the broadcast control list; If the area width is smaller than the total width, the display quantity is calculated according to the area width and the total width, and the broadcast control data associated with the media asset type is extracted from the broadcast control list according to the display quantity.

27. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a click instruction of a target broadcast control view, acquiring a broadcast control type of the target broadcast control view in the broadcast control list; Calling the processing interface associated with the broadcast control type; Based on the processing interface, a processing operation associated with the target broadcast control view is performed on the projection data.

28. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a full screen instruction for displaying the first projection window in full screen, obtaining a screen resolution; Setting a first target display area of ​​the first projection window according to the screen resolution, and setting a second target display area of ​​the second projection window, wherein the number of pixels included in the second target display area is less than or equal to a preset pixel threshold; Control the display to display the first projection screen in the first target display area, and control the display to display the second projection screen in the second target display area.

29. The display device according to claim 27, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a multi-window instruction for exiting full-screen display of a first projection window, obtaining a first resolution of the first projection picture and a second resolution of the second projection picture; resetting the first target display area according to the first resolution, and resetting the second target display area according to the second resolution; Control the display to display the first projection screen in the first target display area after reset, and control the display to display the second projection screen in the second target display area after reset.

30. The display device according to claim 28, wherein the at least one processor is specifically configured to execute computer instructions so that the display device performs the following steps of resetting the first target display area according to the first resolution and resetting the second target display area according to the second resolution: Obtain a first window size and a second window size, wherein the first window size is the window size of the first projection window recorded by the display device when the multi-projection interface is displayed; the second window size is the window size of the second projection window recorded by the display device when the multi-projection interface is displayed; The first target display area is set according to the first window size, and the second target display area is set according to the second window size.

31. The display device according to claim 30, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a screen projection instruction for displaying a multi-screen projection interface, obtaining a first aspect ratio of the first projection screen and a second aspect ratio of the second projection screen; If the first aspect ratio is not equal to the second aspect ratio, setting a first window size of the first projection window according to the first aspect ratio, and setting a second window size of the second projection window according to the second aspect ratio, The window heights of the first window size and the second window size are equal, or the window widths of the first window size and the second window size are equal; Control the display to display the first projection window according to the first window size; and control the display to display the second projection window according to the second window size; record the first window size and the second window size.

32. The display device according to claim 28, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After controlling the display to display the first projection screen in the first target display area, recording the full-screen display time of the first projection screen; If the full-screen display duration is greater than or equal to a preset duration threshold, close the screen projection connection channel with the second terminal device; control the display to stop displaying the second projection screen in the second target display area.

33. The display device according to claim 28, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In the process of setting the first target display area of ​​the first projection window according to the screen resolution, obtaining the screen resolution of the first projection screen; Read the screen width and screen height from the screen resolution; Determine a reference size according to the screen resolution and the picture resolution, the reference size being the screen width or the screen height; Based on the reference size, calculating a scaling ratio; Scaling the first projection screen according to the scaling ratio.

34. The display device according to claim 28, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a full screen instruction for full screen displaying the first projection window, setting a conversion mode parameter of the first projection window to a first parameter value, wherein the first parameter value indicates that the projection window is displayed in full screen; Setting the conversion mode parameter of the second projection window to a second parameter value, wherein the second parameter value indicates hiding the projection window; And, in response to a multi-window instruction for exiting full-screen display of the first projection window, a conversion mode parameter of the first projection window and the second projection window is set to a third parameter value, and the third parameter value represents the multi-window display projection window.

35. The display device according to claim 28, wherein the at least one processor is specifically configured to execute computer instructions so that the display device sets the second target display area of ​​the second projection window in the following manner: Determine a boundary area according to the first target display area, wherein the boundary area is an area outside the first target display area in the entire target display area of ​​the display, or an area formed by edge pixels of the first target display area; According to the preset pixel threshold, setting the minimum number of pixels in the second target display area; The second target display area is determined in the boundary area according to the minimum number of pixels.

36. The display device according to claim 28, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: After setting the first target display area of ​​the first projection window according to the screen resolution, detecting the display content of the second projection window; If the display content of the second screen projection window is a guide screen, close the second screen projection window; If the display content of the second projection window is a projection screen, execute the step of setting a second target display area of ​​the second projection window.

37. The display device according to claim 1, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Control the audio output interface to play audio data of a terminal device whose audio parameter in the screen projection list is a first audio parameter; the audio parameter indicates whether the audio data is allowed to be played or not, the first audio parameter allows the audio data to be played, and the screen projection list is used to record the terminal devices that establish a screen projection connection with the display device, and the connection order of the terminal devices to establish a screen projection connection with the display device; In response to an exit event of disconnecting the screen projection connection relationship between the display device and the fifth terminal device, removing the fifth terminal device from the screen projection list, and recording the exit time of the exit event, the fifth terminal device being a terminal device in the screen projection list whose current audio parameter is the first audio parameter; If a connection event for establishing a screen projection connection relationship between the display device and the sixth terminal device is detected within a preset time threshold after the exit time, the sixth terminal device is added to the screen projection list, and the audio parameter of the sixth terminal device is set to the first audio parameter; If a connection event for establishing a screen projection connection relationship between the display device and the sixth terminal device is not detected within a preset time threshold after the exit time, the seventh terminal device is determined in the screen projection list according to the connection order, and the audio parameters of the seventh terminal device are set to the first audio parameters.

38. The display device according to claim 37, wherein the at least one processor is specifically configured to execute computer instructions so that the display device determines the seventh terminal device in the screen projection list according to the connection order in the following manner: Selecting, in the screen projection list, a terminal device that establishes a screen projection connection with the display device earliest according to the connection order as the seventh terminal device; Alternatively, the terminal device that establishes a screen projection connection with the display device the latest is selected from the screen projection list according to the connection order as the seventh terminal device.

39. The display device according to claim 37, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a connection event of establishing a screen projection connection relationship between the display device and the eighth terminal device, setting the audio parameter of the sixth terminal device to a second audio parameter, and setting the audio parameter of the eighth terminal device to a first audio parameter; the second audio parameter indicates that audio data is not allowed to be played; Alternatively, the audio parameter of the eighth terminal device is set to the second audio parameter.

40. The display device according to claim 37, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a connection event of establishing a screen projection connection relationship between the display device and the terminal device, creating a sound view, wherein the sound view is a control in the screen projection interface for performing playback control on audio data; Reading a status identifier of the sound view, where the status identifier is associated with an audio parameter of the terminal device; If the state identifier is a first identifier value, controlling the display to display a first icon of the sound view; The first icon represents a non-silent mode; If the state identifier is a second identifier value, controlling the display to display a second icon of the sound view; Place The second icon represents the silent mode.

41. The display device according to claim 40, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: Reading audio parameters of the terminal device; If the audio parameter is a first audio parameter, setting the state flag of the sound view to a first flag value; If the audio parameter is a second audio parameter, the state flag of the sound view is set to a second flag value.

42. The display device according to claim 37, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a full-screen instruction for full-screen display of a designated projection window, deleting other terminal devices other than the target terminal device in the projection list, and disconnecting the projection connection with the other terminal devices, wherein the target terminal device is a terminal device whose projection screen is displayed in the designated projection window; Reading audio parameters of the target terminal device; If the audio parameter is the second audio parameter, setting the audio parameter of the target terminal device to the first audio parameter, and displaying the designated screen projection window in full screen; If the audio parameter is the first audio parameter, the designated projection window is displayed in full screen.

43. The display device according to claim 37, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to a display instruction to display a projection screen of a target terminal device in a designated projection window, detecting a window state of the designated projection window; If the window state is a locked state, obtain the target audio parameters of the specified projection window, where the target audio parameters are audio parameters of the terminal device of the specified projection window when the projection screen recorded when the window state is set to a locked state is displayed; The audio parameters of the target terminal device are set as target audio parameters.

44. The display device according to claim 37, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to an exit event of disconnecting the screen projection connection relationship between the display device and the fifth terminal device, recording a first media asset type and a play control list of the screen projection data of the fifth terminal device, wherein the play control list includes play control data of a play control view associated with the first media asset type, and the play control view is a control in the screen projection interface for performing play control on the screen projection data; Within a preset time threshold after the exit time, when a connection event for establishing a screen projection connection relationship between the display device and the sixth terminal device is detected, the second media asset type of the screen projection data of the sixth terminal device is obtained; if the first media asset type is the same as the second media asset type, the broadcast control list is set for the sixth terminal device.

45. The display device according to claim 37, wherein the at least one processor is further configured to execute computer instructions to cause the display device to perform: In response to an exit event of disconnecting the screen projection connection relationship between the display device and the fifth terminal device, recording a display position parameter of a target screen projection window, wherein the target screen projection window is a screen projection window that displays a screen projection image of the fifth terminal device; If a connection event for establishing a screen projection connection relationship between the display device and the sixth terminal device is detected within a preset time threshold after the exit time, the display is controlled to display the projection screen of the sixth terminal device in the target screen projection window according to the display position parameters.

46. ​​A multi-channel screen projection control method, applied to the display device according to any one of claims 1 to 45, the method comprising: In response to the screen projection connection instruction sent by the first terminal device, a first screen projection window is created, the first guide window is released, and the display of the second guide window in the multi-way screen projection interface is retained; the first screen projection window is used to display the first screen projection screen of the first terminal device, and the first screen projection window is the same as the root node of the multi-way screen projection interface; the multi-way screen projection interface includes a first guide window and a second guide window, and the first guide window and the second guide window are used to display a guide page; the first guide window, the second guide window and the root node of the multi-way screen projection interface are the same; Obtain a first screen projection session sent by the first terminal device; The first screen projection session includes a first resolution of the first screen projection image; Detecting the screen resolution and the base resolution of the guide page; Calculate the window display area according to the screen resolution, the first resolution and the reference resolution; the window display area includes a first display area and a second display area, and the first display area and the second display area have the same height; The first projection window is displayed in the first display area, and the display is controlled to display the second guide window in the second display area.

Images