Display equipment and display method of projection screen picture

Abstract

Embodiments of the invention provide a display device and a projection screen display method, and the display device can establish a communication connection with a terminal device and receive projection screen data sent by the terminal device. Wherein the screen projection data comprises a picture frame used for forming a screen projection picture, and the picture frame comprises an effective picture area and an invalid picture area. According to the method, the invalid picture area of the picture frame can be detected, the target display parameter of the screen projection picture is calculated according to the invalid picture area, and then the display is controlled to display the screen projection picture according to the target display parameter. Wherein the target display parameter is used for setting the center point of the effective picture area in the picture frame on the center point of the user interface of the display equipment, and meanwhile, the proportion of the effective picture area of the picture frame in the user interface is maximized. According to the method, the invalid picture area displayed by the display equipment can be removed to the greatest extent, and meanwhile, the picture centering display effect of the effective picture area is kept, so that the watching experience of a user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a display method of a screen mirroring picture. BACKGROUND

[0002] A display device is an intelligent device capable of presenting a user interface and supporting user interaction. Taking a smart television as an example, the smart television is based on Internet application technology, has an open operating system and a chip, has an open application platform, and has a bidirectional man-machine interaction function, and integrates audio, entertainment, data and other functions into one, so as to meet the diversified and personalized needs of users. The display device can establish a communication connection with a terminal device such as a mobile phone, and based on a specific screen mirroring protocol, obtain screen mirroring data of the terminal device, so as to display the screen mirroring picture of the terminal device through the display device.

[0003] In the process of establishing screen mirroring between the display device and the terminal device, the terminal device can record the picture content displayed by itself to form screen mirroring data in the form of a video stream, and then send the screen mirroring data to the display device for display. However, in some scenarios, when the terminal device screens to the display device, the picture displayed in the terminal device can have an invalid picture area, i.e., a pure color area such as a black area without display content. For example, when the terminal device switches from a horizontal screen state to a vertical screen state, black areas can appear at the top and bottom of the terminal device screen. Or, the media played in the terminal device itself can also have an invalid picture area. In this way, when the display device displays the screen mirroring picture corresponding to the screen mirroring data, the invalid picture area included in the screen mirroring data will also be completely displayed in the display device, resulting in a waste of display space of the display device and affecting the user's viewing experience. SUMMARY

[0004] The present application provides a display device and a display method of a screen mirroring picture to solve the problem of low display space utilization when a terminal device screens to a display device.

[0005] In a first aspect, some embodiments of the present application provide a display device, comprising a display, a communicator and a controller. The display is configured to display a user interface, the user interface comprising a screen mirroring picture of a terminal device; the communicator is configured to establish a communication connection with the terminal device and receive screen mirroring data sent by the terminal device; the screen mirroring data comprises a picture frame used to form the screen mirroring picture, the picture frame comprising a valid picture area and an invalid picture area; and the controller is configured to perform the following program steps:

[0006] detecting the invalid picture area of the picture frame;

[0007] calculate a target display parameter of the projection picture according to the invalid picture area; the target display parameter is used to set a center point of the valid picture area on a reference center point of the user interface and to maximize a display proportion of the valid picture area on the user interface;

[0008] control the display to display the projection picture according to the target display parameter.

[0009] In a second aspect, some embodiments of the present application further provide a display method of a projection picture, applied to the display device provided in the first aspect, the display device comprising a display, a communicator and a controller; the display is configured to display a user interface, the user interface comprising a projection picture of a terminal device; the communicator is configured to establish a communication connection with the terminal device and receive projection data sent by the terminal device; the projection data comprises picture frames used to form the projection picture, the picture frames comprising a valid picture area and an invalid picture area; the method comprises:

[0010] detect the invalid picture area of the picture frames;

[0011] calculate a target display parameter of the projection picture according to the invalid picture area; the target display parameter is used to set a center point of the valid picture area on a reference center point of the user interface and to maximize a display proportion of the valid picture area on the user interface;

[0012] control the display to display the projection picture according to the target display parameter.

[0013] It can be known from the above technical solutions that the display device and the display method of the projection picture provided by some embodiments of the present application can establish a communication connection with a terminal device and receive projection data sent by the terminal device. The projection data comprises picture frames used to form a projection picture, and the picture frames comprise a valid picture area and an invalid picture area. The method is applied to the display device, the method can detect the invalid picture area of the picture frames, calculate a target display parameter of the projection picture according to the invalid picture area, and control the display to display the projection picture according to the target display parameter. The target display parameter is used to set a center point of the valid picture area in the picture frames on a center point of a user interface of the display device and maximize a proportion of the valid picture area of the picture frames on the user interface. The method can remove the invalid picture area displayed by the display device to the greatest extent while maintaining a picture center display effect of the valid picture area, thereby improving the viewing experience of a user. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1 This is a schematic diagram showing usage scenarios of the display device in some embodiments of the present application;

[0016] Figure 2 A hardware configuration block diagram of a control device provided in some embodiments of the present application;

[0017] Figure 3 A hardware configuration block diagram of a display device provided in some embodiments of the present application;

[0018] Figure 4 A schematic diagram of software configuration in a display device provided in some embodiments of the present application;

[0019] Figure 5 A schematic diagram of mirror screen projection provided in some embodiments of the present application;

[0020] Figure 6 Schematic diagram of push screen projection provided in some embodiments of this application;

[0021] Figure 7 Schematic diagram of the functional modes of a display device provided in some embodiments of the present application;

[0022] Figure 8 Schematic diagram of the effect of screen projection data frame and screen projection image during screen projection provided by some embodiments of the present application;

[0023] Figure 9a A schematic diagram of a screen projection display effect provided in some embodiments of the present application;

[0024] Figure 9b A schematic diagram of another screen projection display effect provided in some embodiments of the present application;

[0025] Figure 9c A schematic diagram of another screen projection display effect provided in some embodiments of the present application;

[0026] Figure 9d A schematic diagram of another screen projection display effect provided in some embodiments of the present application;

[0027] Figure 9e A schematic diagram of another screen projection display effect provided in some embodiments of the present application;

[0028] Figure 10A schematic diagram of an effect of optimizing screen projection provided in some embodiments of the present application;

[0029] Figure 11 A schematic diagram of a process for optimizing screen projection provided in some embodiments of the present application;

[0030] Figure 12 A flowchart of a screen projection display method provided in some embodiments of the present application;

[0031] Figure 13 A schematic diagram of the margins of the invalid screen area provided in some embodiments of the present application;

[0032] Figure 14 A schematic diagram of the display effect of the optimized projection screen provided in some embodiments of the present application;

[0033] Figure 15 A flowchart of another method for displaying a screen projection image provided in some embodiments of the present application;

[0034] Figure 16 A schematic diagram of another effect of optimizing screen projection provided in some embodiments of the present application;

[0035] Figure 17 A flowchart of another method for displaying a screen projection image provided in some embodiments of the present application;

[0036] Figure 18a A schematic diagram of another effect of optimizing screen projection provided in some embodiments of the present application;

[0037] Figure 18b Some embodiments of this application provide Figure 18a The display principle diagram of the screen projection process shown;

[0038] Figure 19a A schematic diagram of another effect of optimizing screen projection provided in some embodiments of the present application;

[0039] Figure 19b Some embodiments of this application provide Figure 19a The display principle diagram of the screen projection process shown;

[0040] Figure 20a A schematic diagram of another effect of optimizing screen projection provided in some embodiments of the present application;

[0041] Figure 20b Some embodiments of this application provide Figure 20a The display principle diagram of the screen projection process shown;

[0042] Figure 21a A schematic diagram of an optimized screen projection display effect provided in some embodiments of the present application;

[0043] Figure 21b A schematic diagram of another optimized screen projection display effect provided in some embodiments of the present application;

[0044] Figure 21c A schematic diagram of another optimized screen projection display effect provided in some embodiments of the present application;

[0045] Figure 21d A schematic diagram of another optimized screen projection display effect provided in some embodiments of the present application;

[0046] Figure 21e A schematic diagram of another optimized screen projection display effect provided in some embodiments of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the exemplary embodiments of the present application clearer, the technical solutions in the exemplary embodiments of the present application will be clearly and completely described below in combination with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0048] Based on the exemplary embodiments shown in this application, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application. In addition, although the disclosure in this application is presented based on one or several exemplary examples, it should be understood that each aspect of the disclosure can independently constitute a complete technical solution.

[0049] It should be understood that the terms "first," "second," "third," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, for example, enabling implementation in an order other than that shown or described in the embodiments of this application.

[0050] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0051] The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a television, a smart TV, a laser projection device, a monitor, an electronic whiteboard (electronic bulletin board), an electronic table (electronic table), etc.Figure 1 and Figure 3 This is a specific implementation of the display device of the present application.

[0052] Figure 1 FIG is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100 .

[0053] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0054] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the smart device.

[0055] In some embodiments, the display device 200 may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0056] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0057] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0058] Figure 2 FIG. 1 exemplarily shows a configuration block diagram of the control device 100 according to an exemplary embodiment. Figure 2As shown, the control device 100 comprises a device controller 110, a communication interface 130, a user input / output interface 140, a memory, a power supply. The control device 100 can receive the input operation instruction of the user, and convert the operation instruction into an instruction that can be recognized and responded by the display device 200, and play a role of an intermediary in the interaction between the user and the display device 200.

[0059] As shown, the display device 200 comprises at least one of a tuner and demodulator 210, a communicator 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a communication device interface 280. Figure 3

[0060] In some embodiments, the controller comprises a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an n-th interface for input / output.

[0061] The display 260 comprises a display screen component for presenting a picture, and a driving component for driving the image display, a component for receiving the image signal from the output of the controller, and a component for displaying the video content, the image content, and the menu operation interface, and a user operation UI interface.

[0062] The display 260 can be a liquid crystal display, an OLED display, and a projection display, and can also be a projection device and a projection screen.

[0063] In some embodiments, the communicator 220 is a component for communicating with the external device or the server 400 according to various communication protocol types. The display device 200 can be provided with multiple communicators 220 according to the different communication modes supported. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communicator 220 containing WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communicator 220 containing Bluetooth function.

[0064] The communicator 220 can make the display device 200 and the external device or the server 400 in communication connection through wireless or wired connection. The wired connection can connect the display device 200 and the external device through data lines, interfaces, and other components. The wireless connection can connect the display device 200 and the external device through wireless signals or wireless networks. The display device 200 can directly establish a connection relationship with the external device, or indirectly establish a connection relationship through a gateway, a router, a connection device, and the like.

[0065] ​In some embodiments, the display device 200 can also support establishing communication connections with multiple external devices at the same time. Multiple external devices can be connected through the same. For example, the external device is a terminal device 500 such as a mobile phone or tablet computer, which can establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices can also be connected to the display device 200 through different types of connection methods. For example, a first terminal device is connected to the display device 200 via a wireless local area network; a second terminal device is connected to the display device 200 via Bluetooth.

[0066] To achieve communication between the display device 200 and the terminal device 500, the display device 200 and the terminal device 500 are each equipped with a communicator supporting the same type of communication. For example, if the display device 200 is equipped with a communicator 220 including a WiFi module, the terminal device 500 should also be equipped with a communicator including a WiFi module. Furthermore, while the same type of communicator is provided, a specific communication transmission protocol is required for data transmission between the display device 200 and the terminal device 500. Examples include WiFi, Bluetooth, ZigBee, and NFC protocols.

[0067] The communication device interface 280 may be used to receive control signals from the control device 100 (eg, an infrared remote controller, etc.).

[0068] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0069] Device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0070] The tuner-demodulator 210 receives broadcast television signals via a wired or wireless reception method, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0071] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0072] The controller 250 controls the operation of the display device and responds to the user's operation by storing various software control programs in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object displayed on the display 260, the controller 250 can perform an operation related to the object selected by the user command.

[0073] In some embodiments, the controller 250 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a random access memory (RAM), a read-only memory (ROM), a first interface to an n-th interface for input / output, a communication bus, and the like.

[0074] The user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0075] The "user interface" is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form that the user can accept. The commonly used form of the user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an icon, a window, a control, and the like interface elements displayed in the display screen of an electronic device, wherein the control can include an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a widget, and the like visible interface elements.

[0076] For ease of description, some embodiments of the present application are described by taking the display device 200 configured with an Android system as an example, but are not limited thereto. The display device 200 provided by the embodiments of the present application can also be configured with other systems.

[0077] For example, as shown in FIG. 2, the display device 200 can include a display 260, a user input interface, a communication interface, a storage, a controller, and the like. Figure 4As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0078] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0079] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0080] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0081] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0082] In some embodiments, the system runtime layer provides support for the upper layer, i.e., the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime layer to implement the functions implemented by the framework layer.

[0083] In some embodiments, the kernel layer is a layer between hardware and software. As shown in the figure, the kernel layer at least contains at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc. Figure 4

[0084] The display device 200 can establish a screen projection connection channel and transmit screen projection data based on the screen projection connection after a communication connection relationship is established with the terminal device 500, so that the display device 200 can display the screen projection picture of the terminal device 500. In some embodiments, in order to establish a screen projection connection relationship, a user can send a screen projection connection request through the terminal device 500, and send the screen projection connection request to the display device 200 through a WiFi network. The display device 200 completes the transmission protocol configuration according to the screen projection connection request, thereby establishing a transmission channel of screen projection data between the display device 200 and the terminal device 500.

[0085] The display device 200 can also realize screen projection connection with the terminal device 500 through different connection modes. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, the screen projection connection can be established based on the WiFi network. For another example, when the display device 200 and the terminal device 500 are both provided with NFC (Near Field Communication) components, the screen projection connection relationship can be established through the NFC components. Obviously, the display device 200 and the terminal device 500 can also use other wired or wireless connection modes to establish a screen projection connection relationship, such as RF (Radio Frequency) connection, infrared connection, cellular network, and other connection modes that can be thought of by those skilled in the art on the basis of the connection modes provided in the above embodiments.

[0086] After the screen projection connection is established, the display device 200 can receive screen projection related data of the terminal device 500 through the screen projection data channel, so as to play the screen projection related content in the display device 200. Under different screen projection data transmission protocols, the form of the screen projection data obtained by the display device 200 is different. For example, the display device 200 and the terminal device 500 can establish a screen projection connection through a DLNA (Digital Living Network Alliance) protocol, a Miracast protocol, an AirPlay push protocol, a WiDi screen projection, an NFC mirror screen projection, a Yebao screen projection, etc.​

[0087] The DLNA protocol is a protocol used to share media resources between devices. This protocol uses network technology (wired or wireless) to interconnect various devices and exchange data through standard protocols, allowing media resources to be shared between devices. The DLNA protocol is based on the UPnP protocol, enabling devices to discover and control each other, achieving seamless transmission and sharing of media resources. The DLNA protocol supports a variety of devices, including TVs, speakers, mobile phones, computers, and more, allowing users to seamlessly switch and share media content between different devices.

[0088] The Miracast protocol is a wireless display standard developed by the Wi-Fi Alliance and based on Wi-Fi Direct. The Miracast screen projection protocol is compatible with Android devices, allowing Android devices to share video images through Miracast. Smart devices running Android 5.0 and above have the Miracast protocol built-in, allowing direct wireless screen projection without installing any software, using the Wi-Fi P2P function between the projection device and the receiving device's Wi-Fi network card. Miracast also supports simultaneous audio and video transmission.

[0089] AirPlay is a wireless playback technology introduced by Apple, suitable for wireless screen projection on iOS and Mac systems. Apple devices have built-in AirPlay projection functionality. When used with a screen projector that supports the AirPlay server service, the display is clear and stable. The protocol also includes a built-in projection code function. However, the Apple device and screen projector must be on the same local area network (LAN segment) and cannot be used across different network segments or VLANs.

[0090] WiDi is an Intel-developed wireless streaming technology based on Wi-Fi Direct. Windows 8 / 10 / 11 laptops natively support WiDi, allowing you to cast your screen directly without installing any software. It also supports USB reverse control.

[0091] In some embodiments, after the display device 200 establishes a screen projection connection with the terminal device 500, the projection data can be transmitted through the mirror mode and the push mode. Among them, the mirror mode means that the display device 200 synchronously displays the screen content displayed by the terminal device 500. For example, Figure 5As shown, the display device 200 displays the main interface of the terminal device 500, and when the user performs an interactive operation on the terminal device 500, causing the main interface content to change, the content displayed by the display device 200 also changes accordingly. To this end, the terminal device 500 can generate screen projection data in the form of a video stream based on the display content, and send the screen projection data to the display device 200 for display.

[0092] The push mode refers to that the display device 200 accesses a network link pushed by the terminal device 500, and obtains a corresponding screen projection picture. For example, as shown in FIG. 4, when the terminal device 500 plays a video asset, a screen projection control is provided in the upper right corner of the play interface. After the user clicks the screen projection control and selects the display device 200 as a device for displaying the screen projection picture, the terminal device 500 can send the URL address of the currently played video asset to the display device 200. After receiving the URL address, the display device 200 accesses the URL address to obtain the video asset data, and plays the video asset. Figure 6

[0093] Among the screen projection protocols supported by the display device 200 and the terminal device 500, some protocols are based on the mirror mode, and some protocols are based on the push mode. For example, under the DLNA protocol, the display device 200 can obtain the link address of a resource in the terminal device 500 through a screen projection data channel, and obtain the asset content data by accessing the link address, so as to play the asset data. At the same time, the display device 200 can also obtain the control instructions sent by the terminal device 500 through the screen projection data channel when playing the asset data, such as fast forward, pause, stop screen projection, etc. For example, after establishing a connection through a communication protocol, the terminal device 500 such as a mobile phone and the display device 200 will negotiate the device functions and network conditions to select a suitable video and audio transmission format. After establishing a session for video and audio streaming, a series of real-time streaming protocol (RTSP) control commands are used to control the play and termination of the video and audio streaming.

[0094] ​In other screen projection connection protocols such as Miracast screen projection, Airplay screen projection, NFC mirror screen projection, and Leplay screen projection, the display device 200 can obtain a video stream, and at this time, the video stream is screen projection data. The video stream can be video data corresponding to a final display screen of the display unit of the terminal device 500, that is, the video stream includes media content screens, UI interfaces, and other video screen contents that can be displayed in the display unit in the terminal device 500. The video stream can also be only video data that can be obtained in the terminal device 500, such as a media video being played. At this time, the terminal device 500 can be a signal source of the display device 200, the display device 200 can continuously obtain video data through a screen projection data channel, and display the video data in the display 260 in real time, to achieve a synchronous display effect with the terminal device 500. Alternatively, the video stream can also be video data obtained from the cloud, that is, the terminal device 500 can send a URL address of a video media to the display device 200, and the display device 200 accesses the URL address based on a screen projection protocol to obtain screen projection data after receiving the URL address.

[0095] It should be noted that the screen projection protocol described above can not only obtain a video stream in the terminal device 500, but also obtain an audio data stream in the terminal device 500, so that the video data and the audio data can be synchronously sent to the display device 200, to enable the display device 200 to synchronously play the video and the audio.

[0096] The terminal device 500 can obtain different screen projection data according to different screen projection protocols. For example, for system applications or direct screen projection at a system level, the terminal device 500 can establish a screen projection data channel by using a Miracast screen projection protocol, an Airplay screen projection protocol, or a WiDi screen projection protocol. Since the Miracast screen projection protocol and the Airplay screen projection protocol require system-level application permissions, the Miracast screen projection protocol and the Airplay screen projection protocol are only applicable to system applications, or the display device 200 and the terminal device 500 of the same manufacturer directly establish a screen projection connection through an operating system built-in rule.

[0097] The display device 200 can establish a screen projection connection with the terminal device 500 based on any of the screen projection modes provided in the embodiments described above. In some embodiments, the display device 200 can also set a layout of a screen projection window. To this end, as shown in FIG. 2, the display device 200 can internally include multiple function modules in the same application, such as a screen projection application. The screen projection application includes a data management module, a message processing module, a service management module, an algorithm processing module, a View management module, an application interaction processing module, a location management module, and a screen projection management module. Figure 7

[0098] ​Among them, the data management module mainly manages all the message data involved in the current template, and synchronously updates the data information. The message processing module mainly transmits related messages between different modules, such as transmitting data changes to the View layer for refreshing display, or synchronizing View changes to the data management module for change processing of broadcast data, etc. The service management module is responsible for starting and closing the services of the entire application and some customized processing, such as starting specific services in specific scenarios.

[0099] The algorithm processing module is responsible for all algorithm processing of the entire application. The View management module is mainly responsible for all UI display of the entire template application, receiving data to refresh the display of the View. The application interaction processing module is responsible for the processing of interaction with external applications, such as any entry starting the current template application needs to interact with this module, and different customized management is performed according to the difference of the starting entry. The screen projection management module is responsible for screen projection protocol interaction, client list management, and state information processing of window display content.

[0100] It should be noted that the screen projection application, as an application that can display multiple protocol multi-path mirror projection and push content, is an upper layer application inside the display device 200. The Framework layer of the display device 200 can provide Java interfaces for hardware docking, and the multi-path screen projection viewing application realizes docking with the lower layer and information acquisition through the Java interface of the Framework layer. The other Nativie, Runtime layer, HAL layer, Linux kernel are all standard layers of Android architecture, and each layer interacts and processes according to the business process.

[0101] The display device 200 first starts the application through the service management module, and does some initialization preparation work, such as starting to listen to the mirror or push the corresponding protocol service; the View management module initializes the View displayed by the application layer; the data management module creates some data processing module data; the algorithm processing module performs algorithm layout calculation on the screen projection content, etc.

[0102] During the process of establishing a screen projection connection, the service management module can start the screen projection application and register the Session; Usession Connection Listener, Usession Prepare Listener, USession DisplayListener, and Usession Event Listener. Among them, Session represents a conversation between the client and the server, which can save the client's information. Usession Connection Listener is used to monitor the connection and disconnection events of the Session. Usession Prepare Listener interacts with the client's screen projection preparation. Usession DisplayListener monitors changes in playback information and status as well as changes in the client's status. Usession Event Listener processes information such as client control.

[0103] After the View is initialized, the display device 200 will display the default boot page when entering the application. The boot page will display some device information, such as the device name of the current display device 200 and the corresponding network information. The boot page is then displayed through the control system standard TextView and ImageView controls and the corresponding Layout.

[0104] In order to facilitate the management of View and Session, in some embodiments, a management object MultiCastPlayView can be set, wherein the management object MultiCastPlayView can be associated with information such as Session, SurfaceView / TextureView, mViewPosition, etc. For example, for terminal device A, the data corresponding to its Client can be represented as: Session: SessionClient, View: SurfaceView / TextureView.

[0105] When the display device 200 prepares the View as the server, it can notify the terminal device 500 as the mobile terminal to record or send the video stream (i.e., the projection data) through UsesionPrepareListener. When the terminal device 500 is ready to send the video information, the display device 200 knows that the projection playback is to be performed through onFirstFrame() of UsesionDisplayListener. Therefore, it can obtain the current Client video width and height from the Session, determine the direction of the current video stream based on the width and height, and then use the algorithm to calculate the current position and size to be displayed, i.e., the original position and size.

[0106] For example, when the resolution of the terminal device 500 is 1920×1080 and the screen resolution of the display device 200 is 1920×1080, the projection data sent by the terminal device 500 to the display device 200 is as follows: Figure 8 As shown on the left side. The original position and size of the projection screen obtained by the algorithm processing module are: width 1920, height 1080, and distance from the upper left corner position (0, 0). The data management module of the display device 200 can organize and store the data obtained by the algorithm processing module, and then transmit the data, including: Session: SessionClient, View: SurfaceView / TextureView, Position: (0, 0), Width: 1920, Height: 1080.

[0107] It should be noted that the data associated with the management objects and the parameters of the algorithm processing provided in the above embodiment are only exemplary descriptions and are not limitations thereto. Other data objects may also be used in the embodiments of the present application.

[0108] After receiving the data information, the View management module of the display device 200 sets the display parameters through LayoutParams, including the position (i.e., coordinates) from the upper and left boundaries, as well as the corresponding display width and height, and then displays the position through the setLayoutParams method. That is:

[0109] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width, height, 0); lp1.width=1920; 1p1.height=1080; lp1.leftMargin=0; 1p1.topMargin=0; mPlayerView.set LayoutParams(lp1).

[0110] By setting the display position of the projection SurfaceView / TextureView, the display device 200 can display a projection screen as shown in the right part of the figure. Figure 8

[0111] In some embodiments, the projection data sent by the terminal device 500 to the display device 200 includes a picture frame used to form a projection screen, and the picture frame includes a valid picture area and an invalid picture area. In terms of visual angle, the valid picture area can be used to present picture content for a user to view. That is, the valid picture area can be a region in the picture frame that has picture content for a user to view, and the invalid picture area can be a region in the picture frame other than the valid picture area, for example, the invalid picture area can be located at the periphery of the valid picture area and has no substantial effect. In other words, the invalid picture area can be a region in the picture frame that has no actual meaning and is a region that a user does not pay attention to. In some implementations, the invalid picture area can present a single color effect, such as a black, gray, or other solid color region.

[0112] In some embodiments, the display device 200 can scan all pixel points in the picture frame, which can include a row-by-row scanning manner and a column-by-column scanning manner, such as first scanning all pixel points in the picture frame row by row and then scanning all pixel points in the picture frame column by column, or first scanning all pixel points in the picture frame column by column and then scanning all pixel points in the picture frame row by row. Of course, the above scanning order is not unique. The region in which the color values of all row and array pixel points are target values is marked as an invalid picture area. The color value is a characteristic numerical value representing the color of a pixel point, such as an RGB value.

[0113] For example, a pixel point with an RGB value of (0, 0, 0) is a pixel point without display content, and a region in the picture frame composed of pixel points with an RGB value of (0, 0, 0) is an invalid picture area, such as the region a shown in Figure 8 A region composed of other color value pixel points is a valid picture area, such as the region b shown in Figure 8 Since the invalid picture area does not set the RGB value of the pixel point, it can present a display effect of black, gray, or other solid colors.

[0114] ​In some embodiments, when the target value is the color value corresponding to black, the pixel points of the entire row of the picture frame whose color value is the target value are marked as an invalid picture area, and the pixel points of the entire column of the picture frame whose color value is the target value are marked as an invalid picture area. At this time, the display effect presented by the invalid picture area is black, and the invalid picture area can also be referred to as a black edge or a black edge area. Similarly, when the target value is the color value corresponding to gray, the display effect presented by the invalid picture area is gray, and the invalid picture area can also be referred to as a gray edge or a gray edge area. It is understandable that the principles for other colors are the same as above and will not be elaborated here.

[0115] For example, Figure 9a As shown, when the terminal device 500 is in a vertical screen state, the terminal device 500 displays the terminal interface 910; at this time, the terminal device 500 starts recording and sends the projection data to the display device 200. The picture frame 920 in the projection data includes a valid picture area 921 and an invalid picture area 922. The display device 200 displays the following based on the projection data: Figure 9a The projection screen 930 is shown.

[0116] like Figure 9b As shown, when the terminal device 500 is in a horizontal screen state, the media played by the terminal device 500 has an invalid screen area, and the terminal device 500 displays the terminal interface 910; at this time, the terminal device 500 starts recording and sends the projection data to the display device 200. The picture frame 920 in the projection data includes an invalid screen area 922 and a valid screen area 921. The display device 200 displays the following based on the projection data: Figure 9b The projection screen 930 is shown.

[0117] like Figure 9c As shown, when the terminal device 500 is Figure 9b When the horizontal screen state is switched to the vertical screen state, the media played by the terminal device 500 still has an invalid screen area 922, and the terminal device 500 displays the terminal interface 910; at this time, the terminal device 500 starts recording and sends the projection data to the display device 200. The picture frame 920 in the projection data includes the invalid screen area 922 and the valid screen area 921. The display device 200 displays the following based on the projection data: Figure 9c The projection screen 930 is shown.

[0118] like Figure 9d As shown, when the terminal device 500 is in a horizontal screen state, the user adjusts the playback window of the terminal device 500 to the size of area d, and the terminal device 500 displays the terminal interface 910; at this time, the terminal device 500 starts recording and sends the projection data to the display device 200. The picture frame 920 in the projection data includes an invalid picture area 922 and a valid picture area 921. The display device 200 displays the following based on the projection data:Figure 9d The projection screen 930 is shown.

[0119] like Figure 9e As shown, when the terminal device 500 is in a vertical screen state, the user adjusts the playback window of the terminal device 500 to the size of area e, and the terminal device 500 displays the terminal interface 910; at this time, the terminal device 500 starts recording and sends the projection data to the display device 200. The picture frame 920 in the projection data includes an invalid picture area 922 and a valid picture area 921. The display device 200 displays the following based on the projection data: Figure 9e The projection screen 930 is shown.

[0120] Obviously, by Figures 9a-9e As can be seen from the screen projection process shown, when the picture frame of the screen projection data sent by the terminal device 500 includes an invalid picture area, the display device 200 will also display the invalid picture area completely in the user interface, and the invalid picture area will occupy a certain display space of the display device 200.

[0121] like Figure 11 As shown, in some embodiments, the display device 200 detects the invalid screen area of ​​the picture frame and calculates the pre-processed width of the picture frame in the first direction and the pre-processed height of the picture frame in the second direction based on the width and height of the invalid screen area (S1101). The pre-processed width is the width of the valid screen area 921, and the pre-processed height is the height of the valid screen area 922. The first direction is perpendicular to the second direction, such as the x direction and the y direction; the pre-processed width is the difference between the picture frame width and the width of the invalid screen area in the first direction, and the pre-processed height is the difference between the picture frame height and the height of the invalid screen area in the second direction. Then, a first ratio of the picture frame width to the pre-processed width is calculated, and a second ratio of the picture frame height to the pre-processed height is calculated (S1102). The first ratio is set as the scaling ratio of the projected screen in the first direction, and the second ratio is set as the scaling ratio of the projected screen in the second direction (S1103). Finally, the display device 200 scales the width of the projected screen according to the first ratio and the height of the projected screen according to the second ratio (S1104). That is, the scaling ratios of the first direction and the second direction are calculated respectively, and the projection image is scaled according to the scaling ratios.

[0122] For example, the resolution of the terminal device 500 is 1920×1080, and the screen resolution of the display device 200 is 1920×1080. Figure 10 As shown, the first direction is the x direction, the second direction is the y direction, the terminal device 500 records its own terminal interface based on the screen projection protocol, forms screen projection data, and sends the screen projection data to the display device 200. Among them, the screen frame 920 of the screen projection data and the invalid screen area 922 and the invalid screen area 922, as shown Figure 10The size of the invalid picture area 922 is (128, 72), and the algorithm processing module of the display device 200 calculates the pre-processing height by subtracting 72 pixels from both sides in the y direction, and the pre-processing height is 1080-2*72=936; similarly, the display device 200 calculates the pre-processing width by subtracting 128 pixels from both sides in the x direction, and the pre-processing width is 1920-2*128=1664.

[0123] After the display device 200 calculates the pre-processing width and the pre-processing height, the ratio of the original size of the picture frame to the pre-processing height and the pre-processing width is calculated, the first ratio is 1920 / 1664, and the second ratio is 1080 / 936. That is, scaleX=(float)1920 / (float)(1920-2*128)=1.15f, and scaleY=(float)1080 / (float)(1080-2*72)=1.15f. Then, the display device 200 sets the position of the View through the View management module. The scaleX is set as the scaling ratio of the x direction of the projection picture (View: Surface View / Texture View), and the scaleY is set as the scaling ratio of the y direction of the projection picture. In this way, when the display device 200 displays the picture frame of the projection data, the projection picture 930 shown on the right is presented, which can reduce the invalid picture area in the user interface of the display device 200. Figure 10

[0124] However, as shown in the projection process Figures 9a-9e , the position and size of the invalid picture area 922 in the picture frame 920 are uncertain. In the case that the invalid picture area 922 is not uniformly distributed or the invalid picture area 922 is not symmetrical in the x direction and the y direction, the projection picture 930 is scaled by the algorithm processing module by subtracting the invalid picture area in the x direction and the y direction, and the projection picture 930 may be offset in the display device 200.

[0125] Based on the above application scenarios, in some embodiments, as shown in Figure 12 , the display device 200 can include a display 260, a communicator 220, and a controller 250. The display 260 is configured to display a user interface 261, and the user interface 261 includes a projection picture of a terminal device 500. The communicator 200 is configured to establish a communication connection with the terminal device 500 and receive projection data sent by the terminal device 500, wherein the projection data includes a picture frame 920 used to form a projection picture, and the picture frame 920 includes a valid picture area 921 and an invalid picture area 922. The controller 250 is configured to perform the following program steps:

[0126] ​S1201: Detect an invalid picture area of the picture frame.

[0127] The display device 200 establishes a screen projection connection with the terminal device 500 through the communicator 220, receives the screen projection data sent by the terminal device 500, and controls the display 260 to display the picture frame of the screen projection data based on the full-screen display of the screen projection window. At the same time, in the process of screen projection, the display device 200 also detects the invalid picture area 922 of the picture frame 920 in real time.

[0128] In some embodiments, the display device 200 can detect the color value (such as the RGB value) of each picture frame 920 of the screen projection data to determine the invalid picture area 922 in the picture frame 920. That is, the display device 200 reads the picture frame of the screen projection data and detects the color value of the picture frame. Then, the pixel points with the target value as the color value are marked as the pixel points of the invalid picture area to obtain the invalid picture area of the picture frame 920. The target value represents the color value of the pixel point without display content. For example, taking the RGB value as the color value, the target value can be (0, 0, 0).

[0129] S1202: Calculate a target display parameter of the screen projection picture according to the invalid picture area; the target display parameter is used to set the center point of the valid picture area at the reference center point of the user interface and make the display ratio of the valid picture area in the user interface maximum.

[0130] After detecting the invalid picture area of the picture frame 920, the display device 200 calculates the target display parameter of the screen projection picture according to the size of the invalid picture area 922. The display parameter is a parameter used to set the screen projection picture, such as the layout set by the View management module. The target display parameter of the present application is used to set the valid picture area 921 of the picture frame 920 at the reference center point of the user interface and make the display ratio of the valid picture area 921 in the user interface maximum. That is, the screen projection picture displayed according to the target display parameter can reduce the invalid picture area 922 in the user interface to the maximum extent, and the center of the valid picture area 921 can be located at the center position (screen center point) of the user interface, so that the valid picture area 921 can be displayed centrally. When the display device 200 displays the screen projection picture, the display parameter of the screen projection picture is calculated according to the above principle to adjust the display effect of the screen projection picture.

[0131] For example, when the invalid picture area 922 is a black border area, the screen projection picture displayed according to the target display parameter can reduce the black border area in the user interface to the maximum extent, so that the valid picture area 921 can be displayed in the user interface to the maximum extent, and the valid picture area 921 can be displayed centrally in the user interface.

[0132] S1203: Control the display to display the projection image according to the target display parameters.

[0133] After the display device 200 calculates the target display parameters according to the above principles, it controls the display 260 to display the projection screen according to the target display parameters, thereby minimizing the invalid screen area of ​​the user interface while allowing the effective screen area 921 to be displayed symmetrically based on the center position of the user interface. For example, after the display device 200 calculates the target display parameters through the algorithm processing module of the projection application, it stores the target display parameters through the data management module and transmits the target display parameters to the View management module. The View management module then sets the position of the projection screen based on the target display parameters, so that the display effect corresponding to the target display parameters is presented in the user interface of the display 260.

[0134] In some embodiments, when calculating the target display parameters, the display device 200 can use the reference center point as the center of symmetry to compare the invalid screen areas in the first direction and the second direction to determine whether the invalid screen area 922 is symmetrical. When determining whether the invalid screen area 922 is symmetrical, the display device 200 can use an image processing algorithm to determine whether the invalid screen area 922 is centrally symmetrical based on the reference center point; or, the display device 200 can also use a geometric symmetry detection algorithm to determine whether the invalid screen area 922 is centrally symmetrical based on the reference center point; or, the display device 200 can also obtain the margins of the invalid screen area 922 from the upper, lower, left, and right boundaries of the user interface (or projection window) to determine whether the invalid screen area 922 is centrally symmetrical based on the reference center point. For ease of description, in the embodiment of the present application, the margins of the invalid screen area 922 from the upper, lower, left, and right boundaries of the user interface are represented as top, bottom, left, and right, respectively.

[0135] It is understood that the present embodiment exemplifies two methods for determining whether the invalid screen area is symmetrical, but is not intended to be limiting. The method for determining whether the invalid screen area 922 is symmetrical may also be implemented based on other methods, such as using a third-party algorithm library or service framework.

[0136] In some embodiments, as Figure 13As shown, the display device 200 detects the first inactive picture area 922a and the second inactive picture area 922b in the first direction, and detects the third inactive picture area 922c and the fourth inactive picture area 922d in the second direction. The first margin (left) of the first inactive picture area 922a in the first direction, the second margin (right) of the second inactive picture area 922b in the first direction, the third margin (top) of the third inactive picture area 922c in the second direction, and the fourth margin (bottom) of the fourth inactive picture area 922d in the second direction are calculated. That is, the margin corresponding to the first inactive picture area 922a is left, the margin corresponding to the second inactive picture area 922b is right, the margin corresponding to the third inactive picture area 922c is top, and the margin corresponding to the fourth inactive picture area 922d is bottom.

[0137] After the margins of the inactive picture area 922 in each direction are calculated, if the first margin is equal to the second margin, and the third margin is equal to the fourth margin, it is marked that the inactive picture area 922 is centrosymmetric at the reference center point; otherwise, if the first margin is not equal to the second margin, and / or the third margin is not equal to the fourth margin, it is marked that the inactive picture area 922 is not centrosymmetric at the reference center point. That is, when the two margins in the x direction (left and right) are not equal, or the two margins in the y direction (top and bottom) are not equal, or the two margins in the x direction are not equal and the two margins in the y direction are not equal, the display device 200 marks that the inactive picture area 922 is not centrosymmetric at the reference center.

[0138] It should be noted that in the embodiments of the present application, one or more of the first inactive picture area, the second inactive picture area, the third inactive picture area, and the fourth inactive picture area can not exist, that is, the value of any one of top, bottom, left, and right can be 0.

[0139] For ease of description, the following embodiments are described with respect to several scenarios of content-free display areas, taking the first direction as the x direction and the second direction as the y direction as an example.

[0140] Scenario one: the inactive picture area is centrosymmetric at the reference center point, and the first ratio and the second ratio are equal

[0141] In some embodiments, if the invalid image area 922 is centrally symmetrical about the reference center point, the display device 200 calculates the pre-processed width of the image frame 920 in a first direction and the pre-processed height of the image frame 920 in a second direction. Then, a first ratio of the image frame width to the pre-processed width and a second ratio of the image frame height to the pre-processed height are calculated. Since the first ratio and the second ratio are equal at this point, the display device 200 can simply set one of the ratios as the zoom ratio of the projected image.

[0142] That is, when the first ratio is equal to the second ratio, the display device 200 sets the first ratio as the scaling ratio of the projection screen in the first direction to the second direction, or sets the second ratio as the scaling ratio of the projection screen in the first direction to the second direction to generate the target display parameters. Figure 14 The projection screen 930 is shown.

[0143] In some embodiments, when the first ratio is set as the scaling ratio of the projected image 930 in the first direction to the second direction, the display device 200 scales the width of the projected image 930 according to the first ratio, and scales the height of the projected image 930 according to the first ratio. Similarly, when the second ratio is set as the scaling ratio of the projected image 930 in the first direction to the second direction, the display device 200 scales the width of the projected image 930 according to the second ratio, and scales the height of the projected image 930 according to the second ratio.

[0144] For example, the resolution of the terminal device 500 is 1920×1080, and the screen resolution of the display device 200 is 1920×1080. Figure 10 As shown, the display device 200 calculates that the invalid screen area is centrally symmetrical about the reference center point, and the size of the invalid screen area is (128, 72). The display device 200 calculates the pre-processed height as 1080-2*72=936, and calculates the pre-processed width as 1920-2*128=1664. Then, the first ratio and the second ratio are calculated, that is, scaleX=(float)1920 / (float)(1920-2*128)=1.15f and scaleY=(float)1080 / (float)(1080-2*72)=1.15f are calculated. Since the first ratio is equal to the second ratio, the position of the View is set through the View management module, and scaleX or scaleY is set to the scaling ratio of the projection screen (View: SurfaceView / TextureView) in the x-direction and y-direction. In this way, when the display device 200 displays the screen frame of the projection data, it presents Figure 10 The projection screen 930 shown on the right can reduce the invalid screen area in the user interface of the display device 200.

[0145] Scenario two: the invalid picture area is centrosymmetric at the reference center point, and the first ratio and the second ratio are not equal

[0146] As shown in FIG. 9, in some embodiments, if the invalid picture area 922 is centrosymmetric at the reference center point, the display device 200 calculates a pre-processing width of the picture frame 920 in the first direction and a pre-processing height of the picture frame in the second direction (S1501). Then, a first ratio of the picture frame width to the pre-processing width is calculated, and a second ratio of the picture frame height to the pre-processing height is calculated (S1502). Figure 15 At this time, since the overall distribution of the invalid picture area in the x direction and the y direction is not uniform, the values of the first ratio and the second ratio are not equal. In order to ensure that the projection picture can be displayed in the original ratio of the valid picture area 921 after scaling, when the first ratio and the second ratio are not equal, the display device 200 obtains the minimum value of the first ratio and the second ratio (S1503), and sets the minimum value as the scaling ratio of the projection picture in the first direction, and sets the minimum value as the scaling ratio of the projection picture in the second direction to generate the target display parameter (S1504).

[0147] In some embodiments, if the invalid picture area 922 is centrosymmetric at the reference center point, but the first ratio and the second ratio are not equal, when the display device 200 displays the projection picture according to the target display parameter, the width of the projection picture 930 is scaled according to the minimum value, and the height of the projection picture 930 is scaled according to the minimum value (S1505).

[0148] For example, the resolution of the terminal device 500 is 1920x1080, and the screen resolution of the display device 200 is 1920x1080. As shown in FIG. 10, in some embodiments, if the invalid picture area 922 is centrosymmetric at the reference center point, the display device 200 calculates a pre-processing width of the picture frame 920 in the first direction and a pre-processing height of the picture frame in the second direction (S1501). Then, a first ratio of the picture frame width to the pre-processing width is calculated, and a second ratio of the picture frame height to the pre-processing height is calculated (S1502).

[0149] Figure 16 ​As shown, the display device 200 calculates that the invalid screen area is centrally symmetrical about the reference center point, and the size of the invalid screen area is (100, 50). The display device 200 calculates the preprocessing height as 1080-2*50=980, and calculates the preprocessing width as 1920-2*100=1720. Then, the first ratio in the x direction and the second ratio in the y direction are calculated, that is, scaleX=(float)1920 / (float)(1920-2*100)=1.12f, scaleY=(float)1080 / (float)(1080-2*50)=1.10f. Since the first ratio is not equal to the second ratio, the display device 200 obtains the minimum value of 1.10f between the first ratio and the second ratio. The View management module is then used to set the position of the View according to the ratio of 1.10f. That is, the projection screen is scaled through View.animate().scaleX(scaleY).scaleY(scaleY). In this way, when the display device 200 displays the picture frame of the projection data, it presents Figure 16 The projection screen 930 shown on the right can reduce the invalid screen area in the user interface of the display device 200.

[0150] Scenario 3: The invalid image area is not centrally symmetrical about the reference center point

[0151] like Figure 17 As shown, in some embodiments, if the invalid screen area 922 is not centrally symmetrical at the reference center point, that is, the two invalid screen areas in the x direction are asymmetrical with each other, or the two invalid screen areas in the y direction are asymmetrical with each other, or the two invalid screen areas in the x direction are asymmetrical and the two invalid screen areas in the y direction are also asymmetrical, the display device 200 calculates the pre-processed width of the screen frame 920 in the first direction and the pre-processed height of the screen frame in the second direction (S1701), and then calculates the target center point of the screen frame 920 under the pre-processed width and pre-processed height (S1702), that is, the center point of the valid screen area. Then, the center offset is calculated based on the target center point and the reference center point (S1703), and the position parameters of the projection screen are set based on the center offset (S1704). Then, the first ratio of the screen frame width to the pre-processed width is calculated, and the second ratio of the screen frame height to the pre-processed height is calculated (S1705).

[0152] At the same time, in order to enable the projection screen to be displayed according to the original ratio of the effective screen area 921, the minimum value of the first ratio and the second ratio is obtained (S1706), and the minimum value is set as the scaling ratio of the projection screen in the first direction, and the minimum value is set as the scaling ratio of the projection screen in the second direction to generate the target display parameters (S1707).

[0153] In some embodiments, if the invalid screen area 922 is not centrally symmetrical at the reference center point, the display device 200 moves the position of the projection screen in the user interface according to the position parameters (S1708), and then scales the width of the projection screen according to the minimum value, and scales the height of the projection screen according to the minimum value (S1709).

[0154] For example, the following three examples are used to illustrate the three situations of scenario three.

[0155] Example 1: The two invalid screen areas in the x-direction are asymmetric, but the two invalid screen areas in the y-direction are symmetric

[0156] The resolution of the terminal device 500 is 1920×1080, and the screen resolution of the display device 200 is 1920×1080. Then the reference center point of the user interface is (960, 540). Figure 18a As shown, the display device 200 detects that the invalid image area is not centrally symmetrical, and detects that the four margins of the invalid image area are left = 60, right = 100, top = 65, and bottom = 65. The algorithm processing module of the display device 200 calculates the pre-processed height as 1080-65-65 = 950, and calculates the pre-processed width as 1920-100-60 = 1760. Then, the target center point can be obtained as (940, 540) based on the pre-processed height and pre-processed width. Figure 18b As shown, after the display device 200 subtracts the various margins, the center point of the remaining picture frame area (effective picture area) is offset. In order to display the remaining picture frame content in the center, the display device 200 needs to move it 20 pixels to the right before it can be scaled. Therefore, the algorithm processing module of the display device 200 can calculate the offset as 960-940=20 based on the target center point (940, 540) and the reference center point (960, 540), that is, a rightward shift of 20. Then, the display device 200 calculates the first ratio and the second ratio based on the preprocessed height and preprocessed width, that is, scaleX=(float)1920 / (float)(1920-100-60)=1.09f, scaleY=(float)1080 / (float)(1080-2*65)=1.14f.

[0157] The algorithm processing module selects the minimum value 1.09f from the first ratio and the second ratio, transmits it to the View management module as a scaling ratio, and transmits 20 to the right as a position parameter to the View management module. Finally, the View management module sets the position of the projection screen to right-shift the projection screen by 20 pixels and scale the width and height of the frame according to the ratio of 1.09f. In this way, when the display device 200 displays the frame of the projection data, it presents Figure 18a The right side of the projection screen 930 can reduce the invalid screen area in the user interface of the display device 200, and the effective screen area 921 of the frame can be displayed in the center of the user interface.

[0158] In some embodiments, when the display device 200 sets the position parameters of the projection screen based on the center offset, it calculates the first difference and the second difference between the target center point and the reference center point. The first difference is the difference between the target center point and the reference center point in the first direction, and the second difference is the difference between the target center point and the reference center in the second direction. Then, according to the first difference, the distance between the projection screen and the left edge of the projection window is set, and according to the second difference, the distance between the projection screen and the top edge of the projection window is set.

[0159] For example, when the projection screen needs to be right-shifted by 20 pixels and scaled by 1.09f, the target display parameters set by the View management module can include: FrameLayout.LayoutParams lp1 = new FrameLayout.LayoutParams(width, height, 0); lp1.width = 1920; lp1.height = 1080; lp1.leftMargin = 20; lp1.topMargin = 0; mPlayerView.setLayoutParams(lp1); mPlayerView.animate().scaleX(scaleX).scaleY(scaleX).

[0160] Example two: The two invalid screen areas in the x direction are symmetrical, but the two invalid screen areas in the y direction are asymmetrical

[0161] The resolution of the terminal device 500 is 1920x1080, and the screen resolution of the display device 200 is 1920x1080, so the reference center point of the user interface is (960, 540). As shown in Figure 19aAs shown, the display device 200 detects that the invalid image area is not centrally symmetrical, and detects that the four margins of the invalid image area are left = 100, right = 100, top = 0, and bottom = 66. The algorithm processing module of the display device 200 calculates the pre-processed height as 1080-66-0 = 1014, and the pre-processed width as 1920-100-100 = 1720. Then, the target center point can be obtained as (960, 507) based on the pre-processed height and pre-processed width. Figure 19b As shown, after the display device 200 subtracts the various margins, the center point of the remaining picture frame area (effective picture area) is offset. In order to display the remaining picture frame content in the center, the display device 200 needs to move it downward by 33 pixels before it can be scaled. Therefore, the algorithm processing module of the display device 200 can calculate the offset of 540-507=33 based on the target center point (960, 507) and the reference center point (960, 540), that is, a downward translation of 33. Then, the display device 200 calculates the first ratio and the second ratio based on the preprocessing height and the preprocessing width, that is, scaleX=(float)1920 / (float)(1920-100-100)=1.12f, scaleY=(float)1080 / (float)(1080-65)=1.07f.

[0162] The algorithm processing module selects the smallest ratio of 1.07f between the first ratio and the second ratio, transmits it to the View management module as the scaling ratio, and transmits the downward translation of 33 as the position parameter to the View management module. Finally, the position of the projection screen is set by the View management module to translate the projection screen downward by 33 pixels and scale the width and height of the picture frame according to the ratio of 1.07f. Among them, the target display parameters set by the View management module may include: FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.widt h=1920;lp1.height=1080;lp1.leftMargin=20;lp1.topMargin=0;mPlayerView.setLay outParams(lp1);mPlayerView.animate().scaleX(scaleY).scaleY(scaleY). In this way, when the display device 200 displays the picture frame of the projection data, it presents Figure 19aThe projection screen 930 shown on the right can reduce the invalid screen area in the user interface of the display device 200, while allowing the valid screen area 921 of the screen frame to be displayed in the center of the user interface.

[0163] Example 3: The two invalid screen areas in the x-direction are asymmetric, and the two invalid screen areas in the y-direction are asymmetric.

[0164] The resolution of the terminal device 500 is 1920×1080, and the screen resolution of the display device 200 is 1920×1080. Then the reference center point of the user interface is (960, 540). Figure 20a As shown, the display device 200 detects that the invalid image area is not centrally symmetrical, and detects that the four margins of the invalid image area are lett = 100, right = 80, top = 30, and bottom = 100. The algorithm processing module of the display device 200 calculates the pre-processed height as 1080-30-100 = 950, and calculates the pre-processed width as 1920-100-80 = 1740. Then, the target center point can be obtained as (970, 475) based on the pre-processed height and pre-processed width. Figure 20b As shown, after the display device 200 subtracts the various margins, the center point of the remaining picture frame area (effective picture area) is offset. In order to display the remaining picture frame content in the center, the display device 200 needs to move it 10 pixels to the right and 65 pixels downward before it can be scaled. Therefore, the algorithm processing module of the display device 200 can calculate the offsets including 10 and 65 based on the target center point (970, 475) and the reference center point (960, 540), that is, a translation of 10 to the right and a translation of 65 downward. Then, the display device 200 calculates the first ratio and the second ratio based on the preprocessed height and the preprocessed width, that is, scaleX = (float) 1920 / (float) (1920-100-80) = 1.10f, scaleY = (float) 1080 / (float) (1080-30-100) = 1.14f

[0165] The algorithm processing module selects the minimum value 1.10f from the first ratio and the second ratio, transmits it to the View management module as a scaling ratio, and transmits the right translation of 10 and the downward translation of 65 to the View management module as position parameters. Finally, the View management module sets the position of the projection screen to right translate the projection screen by 10 pixels and downward translate it by 65 pixels, and scales the width and height of the picture frame by a ratio of 1.10f. The target display parameters set by the View management module can include: FrameLayout.LayoutParams lp1 = new FrameLayout.LayoutParams(width, height, 0); lp1.width = 1920; lp1.height = 1080; lp1.leftMargin = 10; lp1.topMargin = -65; mPlayerView.setLayoutParams(lp1); mPlayerView.animate().scaleX(scaleX).scaleY(scaleX). Thus, when the display device 200 displays the picture frame of the projection data, it presents Figure 20a the projection screen 930 shown on the right, which can reduce the invalid picture area in the user interface of the display device 200, and at the same time, the effective picture area 921 of the picture frame can be displayed in the center of the user interface.

[0166] Based on the above display device 200, some embodiments of the present application also provide a display method for the projection screen, which is applied to the display device 200 described in the above embodiments. The display device 200 can include a display 260, a communicator 220, and a controller 250. The display 260 is configured to display a user interface 261, and the user interface 261 includes a projection screen of a terminal device 500. The communicator 200 is configured to establish a communication connection with the terminal device 500 and receive projection data sent by the terminal device 500. The projection data includes a picture frame used to form a projection screen, and the picture frame includes an effective picture area and an invalid picture area. As shown in the above embodiments, the method includes the following steps: Figure 12

[0167] S1201: detecting the invalid picture area of the picture frame;

[0168] S1202: calculating target display parameters of the projection screen according to the invalid picture area; the target display parameters are used to set the center point of the effective picture area to the reference center point of the user interface, and make the effective picture area have the maximum display ratio in the user interface;

[0169] ​S1203: controlling the display to display the screen mirroring picture according to the target display parameter.

[0170] Based on the above embodiments, for the screen mirroring process shown in FIG. 1A, the display device 200 can perform scaling and moving on the screen mirroring picture by the method for displaying a screen mirroring picture provided in the embodiments of the present application, and display the screen mirroring effect shown in FIG. 1B. Figures 9a-9b Figures 21a-21e After the display device 200 performs steps S1201-S1203, the screen mirroring process shown in FIG. 2A presents the screen mirroring effect shown in FIG. 2B. Figure 9a Figure 21a After the display device 200 performs steps S1201-S1203, the screen mirroring process shown in FIG. 3A presents the screen mirroring effect shown in FIG. 3B. Figure 9b Figure 21b After the display device 200 performs steps S1201-S1203, the screen mirroring process shown in FIG. 4A presents the screen mirroring effect shown in FIG. 4B. Figure 9c Figure 21c After the display device 200 performs steps S1201-S1203, the screen mirroring process shown in FIG. 5A presents the screen mirroring effect shown in FIG. 5B. Figure 9d Figure 21d After the display device 200 performs steps S1201-S1203, the screen mirroring process shown in FIG. 6A presents the screen mirroring effect shown in FIG. 6B. Figure 9e Figure 21e After the display device 200 performs steps S1201-S1203, the screen mirroring process shown in FIG. 7A presents the screen mirroring effect shown in FIG. 7B.

[0171] As can be seen from the above technical solutions, the display device and the method for displaying a screen mirroring picture provided in some embodiments of the present application can establish a communication connection with the terminal device 500, and receive screen mirroring data sent by the terminal device 500. The screen mirroring data includes picture frames used to form a screen mirroring picture, and the picture frames include valid picture areas and invalid picture areas. The method is applied in a display device, and the method can detect the invalid picture areas of the picture frames, calculate target display parameters of the screen mirroring picture according to the invalid picture areas, and control the display 260 to display the screen mirroring picture according to the target display parameters. The target display parameters are used to set the center point of the valid picture areas in the picture frames on the center point of the user interface of the display device, and at the same time, make the valid picture areas of the picture frames have the maximum proportion in the user interface. The method can remove the invalid picture areas displayed by the display device 200 to the greatest extent, while maintaining the picture center display effect of the valid picture areas, thereby improving the viewing experience of the user.

[0172] The same or similar parts among various embodiments in the present specification can be referred to each other, which will not be described herein again.

[0173] ​​​​​​Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present application can be implemented by means of software plus necessary universal hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to cause a computer device (which can be a personal computer, a server, or a network device, and the like) to execute the methods of the various embodiments or some parts of the embodiments of the present application.

[0174] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0175] For the convenience of explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: A display configured to display a user interface, wherein the user interface includes a projection screen of the terminal device; A communicator configured to establish a communication connection with a terminal device and receive projection data sent by the terminal device; the projection data includes a picture frame for forming the projection picture, and the picture frame includes a valid picture area and an invalid picture area; The controller is configured as: Detecting an invalid picture area of ​​the picture frame; Calculating target display parameters of the projection screen based on the invalid screen area; the target display parameters are used to set the center point of the valid screen area to the reference center point of the user interface, and to maximize the display proportion of the valid screen area in the user interface; Control the display to display the projection screen according to the target display parameters.

2. The display device according to claim 1, wherein The controller calculates the target display parameters of the projection image and is configured to: Taking the reference center point as the center of symmetry, comparing the invalid screen areas in a first direction and a second direction, where the first direction is perpendicular to the second direction; If the invalid picture area is centrosymmetric about the reference center point, calculating a pre-processed width of the picture frame in the first direction and a pre-processed height of the picture frame in the second direction; The pre-processing width is the difference between the picture frame width and the invalid picture area width in the first direction, and the pre-processing height is the difference between the picture frame height and the invalid picture area height in the second direction; calculating a first ratio of the picture frame width to the preprocessing width, and calculating a second ratio of the picture frame height to the preprocessing height; When the first ratio is equal to the second ratio, the first ratio is set as the scaling ratio of the projection screen in the first direction and the second direction, or the second ratio is set as the scaling ratio of the projection screen in the first direction and the second direction to generate the target display parameters.

3. The display device according to claim 2, wherein The controller controls the display to display the projection image according to the target display parameters, and is configured to: When the first ratio is set as the scaling ratio of the projection screen in the first direction to the second direction, the width of the projection screen is scaled according to the first ratio, and the height of the projection screen is scaled according to the first ratio; When the second ratio is set as the scaling ratio of the projection screen in the first direction and the second direction, the width of the projection screen is scaled according to the second ratio, and the height of the projection screen is scaled according to the second ratio.

4. The display device according to claim 2, wherein: The controller calculates the target display parameters of the projection image and is configured to: When the first ratio is not equal to the second ratio, obtaining a minimum value of the first ratio and the second ratio; The minimum value is set as the scaling ratio of the projection screen in the first direction, and the minimum value is set as the scaling ratio of the projection screen in the second direction to generate the target display parameters.

5. The display device according to claim 4, wherein The controller controls the display to display the projection image according to the target display parameters, and is configured to: Scale the width of the projection screen according to the minimum value, and scale the height of the projection screen according to the minimum value.

6. The display device according to claim 2, wherein: The controller is configured to: If the invalid picture area is not centrosymmetric about the reference center point, calculating a preprocessing width of the picture frame in the first direction and a preprocessing height of the picture frame in the second direction; The pre-processing width is the difference between the picture frame width and the invalid picture area width in the first direction, and the pre-processing height is the difference between the picture frame height and the invalid picture area height in the second direction; Calculating a target center point of the picture frame under the preprocessing width and the preprocessing height; Calculating a center offset according to the target center point and the reference center point; Setting the position parameters of the projection image based on the center offset; calculating a first ratio of the picture frame width to the preprocessing width, and calculating a second ratio of the picture frame height to the preprocessing height; Obtaining a minimum value of the first ratio and the second ratio; The minimum value is set as the scaling ratio of the projection screen in the first direction, and the minimum value is set as the scaling ratio of the projection screen in the second direction to generate the target display parameters.

7. The display device according to claim 6, wherein: The controller controls the display to display the projection image according to the target display parameters, and is configured to: Moving the projection image to a position on the user interface according to the position parameter; Scale the width of the projection screen according to the minimum value, and scale the height of the projection screen according to the minimum value.

8. The display device according to claim 6, wherein: The projection screen is displayed in the user interface through the projection window; the controller sets the position parameters of the projection screen based on the center offset, and is configured to: Calculating a first difference and a second difference between the target center point and the reference center point, the first difference being the difference between the target center point and the reference center point in the first direction, and the second difference being the difference between the target center point and the reference center point in the second direction; The distance between the left edge of the projection screen and the projection window is set according to the first difference, and the distance between the upper edge of the projection screen and the projection window is set according to the second difference.

9. The display device according to claim 2, wherein: The controller compares the invalid screen areas in the first direction and the second direction, and is configured to: detecting a first invalid screen area and a second invalid screen area in the first direction, and detecting a third invalid screen area and a fourth invalid screen area in the second direction; Calculating a first margin of the first invalid screen area in the first direction, a second margin of the second invalid screen area in the first direction, a third margin of the third invalid screen area in the second direction, and a fourth margin of the fourth invalid screen area in the second direction; If the first margin is equal to the second margin and the third margin is equal to the fourth margin, marking the invalid screen area as being centrally symmetrical about the reference center point; If the first margin is not equal to the second margin, and / or the third margin is not equal to the fourth margin, it is marked that the invalid screen area is not centrally symmetrical about the reference center point.

10. A method for displaying a projection screen, characterized in that: Applied to a display device, the display device includes a display, a communicator, and a controller, the display is configured to display a user interface, the user interface includes a projection screen of a terminal device; the communicator is configured to establish a communication connection with the terminal device and receive projection data sent by the terminal device, the projection data includes a picture frame for forming the projection screen; the method includes: Detecting an invalid picture area of ​​the picture frame; Calculating target display parameters of the projection screen based on the invalid screen area; the target display parameters are used to set the center point of the valid screen area to the reference center point of the user interface, and to maximize the display proportion of the valid screen area in the user interface; Control the display to display the projection screen according to the target display parameters.

Images