Display equipment and picture display method based on screen projection data

Abstract

Some embodiments of the invention provide a screen projection data-based picture display method, which can receive screen projection data sent by a terminal device, and detect feature information of the screen projection data in a preset time period. If the feature information is changed in the preset time period, maintaining display parameters of the projection screen; otherwise, calculating a target display parameter of the screen projection picture according to the invalid picture area of the current picture frame, and controlling the display to display the screen projection picture according to the target display parameter. Wherein the target display parameter is used for setting the center point of the effective picture area in the current picture frame at the reference center point of the user interface, and meanwhile, the display proportion of the effective picture area in the user interface is made to be maximum. According to the method, when the projection screen data is in the stable state, the invalid picture area displayed by the display equipment can be removed to the greatest extent, and the picture centering display effect of the valid picture area is kept, so that the display effect of the projection screen picture is improved while the system resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of display devices and in particular to a display device and a method for displaying images based on projection data. Background Art

[0002] Display devices are intelligent devices that can present user interfaces and support user interaction. For example, smart TVs are based on Internet application technologies, feature open operating systems and chips, and boast an open application platform. They enable two-way human-computer interaction and integrate multiple functions, including audio, video, entertainment, and data, to meet diverse and personalized user needs. Display devices can establish communication connections with mobile phones and other terminal devices. Based on specific projection protocols, they can obtain projection data from terminal devices and display the projection image on the display device.

[0003] During the process of establishing screen projection between the display device and the terminal device, the terminal device can record the screen content displayed by itself to form projection data in the form of a video stream, and then send the projection data to the display device for display. However, in some scenarios, when the terminal device projects the screen to the display device, the screen displayed in the terminal device may have invalid screen areas, that is, solid color areas such as black with no display content. For example, when the terminal device switches from landscape mode to portrait mode, black areas may appear at the top and bottom of the terminal device screen. Alternatively, the media played in the terminal device itself may also have invalid screen areas. In this way, when the display device displays the projection screen corresponding to the projection data, the invalid screen area included in the projection data will also be fully displayed in the display device, resulting in a waste of display space of the display device, affecting the user's viewing experience. Summary of the Invention

[0004] The present application provides a display device and a screen display method based on projection data to solve the problem of low display space utilization when a terminal device projects a screen to a display device.

[0005] In a first aspect, some embodiments of the present application provide a display device comprising a display, a communicator, and a controller. The display is configured to display a user interface, wherein the user interface comprises a projection screen of a terminal device; the communicator is configured to establish a communication connection with the terminal device and receive projection data sent by the terminal device; the projection data comprises a frame for forming the projection screen, wherein the frame comprises an effective screen area and an invalid screen area; and the controller is configured to execute the following program steps:

[0006] Detecting the feature information of the projection data within a preset time period;

[0007] If the characteristic information changes, maintaining the display parameters of the projection screen;

[0008] If the characteristic information has not changed, the target display parameters of the projected screen are calculated based on the invalid screen area of ​​the current screen frame, and the display is controlled to display the projected screen according to the target display parameters; the target display parameters are used to set the center point of the valid screen area in the current screen frame to the reference center point of the user interface, and to maximize the display proportion of the valid screen area in the user interface.

[0009] In a second aspect, some embodiments of the present application further provide a screen display method based on projection data, which is applied to the display device provided in the first aspect, wherein the display device includes a display, a communicator, and a controller; wherein the display is configured to display a user interface, and the user interface includes a projection screen of the terminal device; the communicator is configured to establish a communication connection with the terminal device, and receive the projection data sent by the terminal device; the projection data includes a screen frame for forming the projection screen, and the screen frame includes a valid screen area and an invalid screen area; the method includes the following steps:

[0010] Detecting characteristic information of the projection data within a preset time period;

[0011] If the characteristic information changes, maintaining the display parameters of the projection screen;

[0012] If the characteristic information has not changed, the target display parameters of the projected screen are calculated based on the invalid screen area of ​​the current screen frame, and the display is controlled to display the projected screen according to the target display parameters; the target display parameters are used to set the center point of the valid screen area in the current screen frame to the reference center point of the user interface, and to maximize the display proportion of the valid screen area in the user interface.

[0013] It can be seen from the above technical solutions that some embodiments of the present application provide a screen display method based on projection data, which can receive the projection data sent by the terminal device and detect the characteristic information of the projection data in a preset time period. If the characteristic information changes within the preset time period, the display parameters of the projection screen are maintained; otherwise, the target display parameters of the projection screen are calculated based on the invalid screen area of ​​the current screen frame, and the display is controlled to display the projection screen according to the target display parameters. The target display parameters are used to set the center point of the effective screen area in the current screen frame to the reference center point of the user interface, and to maximize the display proportion of the effective screen area in the user interface. The method can remove the invalid screen area displayed by the display device to the greatest extent when the projection data is in a stable state, and maintain the centering display effect of the effective screen area, thereby improving the display effect of the projection screen while reducing system resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1 This is a schematic diagram showing usage scenarios of the display device in some embodiments of the present application;

[0016] Figure 2 A hardware configuration block diagram of a control device provided in some embodiments of the present application;

[0017] Figure 3 A hardware configuration block diagram of a display device provided in some embodiments of the present application;

[0018] Figure 4 A schematic diagram of software configuration in a display device provided in some embodiments of the present application;

[0019] Figure 5 A schematic diagram of mirror screen projection provided in some embodiments of the present application;

[0020] Figure 6 Schematic diagram of push screen projection provided in some embodiments of this application;

[0021] Figure 7 Schematic diagram of the functional modes of a display device provided in some embodiments of the present application;

[0022] Figure 8 Schematic diagram of the effect of screen projection data frame and screen projection image during screen projection provided by some embodiments of the present application;

[0023] Figure 9a A schematic diagram of a screen projection display effect provided in some embodiments of the present application;

[0024] Figure 9b A schematic diagram of another screen projection display effect provided in some embodiments of the present application;

[0025] Figure 10 A schematic diagram of an effect of optimizing screen projection provided in some embodiments of the present application;

[0026] Figure 11 A schematic diagram of a process for optimizing screen projection provided in some embodiments of the present application;

[0027] Figure 12a This is an example diagram of the effect of a screen mirroring process provided in some embodiments of the present application;

[0028] Figure 12b This is an example diagram of another mirroring screen projection process provided in some embodiments of the present application;

[0029] Figure 13 A schematic diagram of a flow chart of a method for displaying a screen based on projection data provided in some embodiments of the present application;

[0030] Figure 14a A schematic diagram of a first picture frame set provided in some embodiments of the present application;

[0031] Figure 14b A schematic diagram of a second picture frame set provided in some embodiments of the present application;

[0032] Figure 15 A schematic diagram of a third picture frame set provided in some embodiments of the present application;

[0033] Figure 16a This is a schematic diagram of the effect of the screen projection in Example 1 provided in some embodiments of the present application;

[0034] Figure 16b This is a schematic diagram of the effect of the screen projection in Example 1 provided in some embodiments of the present application;

[0035] Figure 17 A schematic diagram of the margins of the invalid screen area provided in some embodiments of the present application;

[0036] Figure 18 A flowchart of a screen projection display method provided in some embodiments of the present application;

[0037] Figure 19 A flowchart of another method for displaying a screen projection image provided in some embodiments of the present application;

[0038] Figure 20a This is a schematic diagram of the effect of the screen projection in Example 2 provided in some embodiments of the present application;

[0039] Figure 20b This is a schematic diagram of the effect of the screen projection in Example 2 provided in some embodiments of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the exemplary embodiments of the present application clearer, the technical solutions in the exemplary embodiments of the present application will be clearly and completely described below in combination with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0041] Based on the exemplary embodiments shown in this application, all other embodiments obtained by persons of ordinary skill in the art without inventive effort are within the scope of protection of this application. In addition, although the disclosure in this application is presented based on one or several exemplary examples, it should be understood that each aspect of the disclosure can independently constitute a complete technical solution.

[0042] It should be understood that the terms "first," "second," "third," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, for example, enabling implementation in an order other than that shown or described in the embodiments of this application.

[0043] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0044] The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a television, a smart TV, a laser projection device, a monitor, an electronic whiteboard (electronic bulletin board), an electronic table (electronic table), etc. Figure 1 and Figure 3 This is a specific implementation of the display device of the present application.

[0045] Figure 1 FIG is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100 .

[0046] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0047] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 can be controlled using an application running on the smart device.

[0048] In some embodiments, the display device 200 may not use the aforementioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0049] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0050] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0051] Figure 2 FIG. 1 exemplarily shows a configuration block diagram of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a device controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert the commands into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0052] like Figure 3 As shown, the display device 200 includes at least one of a tuner and demodulator 210 , a communicator 220 , a detector 230 , a device interface 240 , a controller 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a communication device interface 280 .

[0053] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0054] The display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a controller, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0055] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0056] In some embodiments, the communicator 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with multiple communicators 220 depending on the supported communication methods. For example, if the display device 200 supports wireless network communication, the display device 200 may be provided with a communicator 220 including WiFi functionality. If the display device 200 supports Bluetooth connection communication, the display device 200 may also be provided with a communicator 220 including Bluetooth functionality.

[0057] The communicator 220 can establish a communication connection between the display device 200 and an external device or server 400 via a wireless or wired connection. A wired connection can connect the display device 200 to an external device via a data cable, an interface, or other components. A wireless connection can connect the display device 200 to an external device via a wireless signal or wireless network. The display device 200 can establish a connection with an external device directly or indirectly through a gateway, router, or connection device.

[0058] In some embodiments, the display device 200 can also support establishing communication connections with multiple external devices at the same time. Multiple external devices can be connected through the same. For example, the external device is a terminal device 500 such as a mobile phone or tablet computer, which can establish a communication connection with the display device 200 by accessing the wireless local area network where the display device 200 is located. Multiple external devices can also be connected to the display device 200 through different types of connection methods. For example, a first terminal device is connected to the display device 200 via a wireless local area network; a second terminal device is connected to the display device 200 via Bluetooth.

[0059] To achieve communication between the display device 200 and the terminal device 500, the display device 200 and the terminal device 500 are each equipped with a communicator supporting the same type of communication. For example, if the display device 200 is equipped with a communicator 220 including a WiFi module, the terminal device 500 should also be equipped with a communicator including a WiFi module. Furthermore, while the same type of communicator is provided, a specific communication transmission protocol is required for data transmission between the display device 200 and the terminal device 500. Examples include WiFi, Bluetooth, ZigBee, and NFC protocols.

[0060] The communication device interface 280 may be used to receive control signals from the control device 100 (eg, an infrared remote controller, etc.).

[0061] Detector 230 is used to acquire signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image capturer, such as a camera, for capturing the external environment, user attributes, or user interaction gestures; or, detector 230 includes a sound capturer, such as a microphone, for receiving external sounds.

[0062] Device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0063] The tuner-demodulator 210 receives broadcast television signals via a wired or wireless reception method, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0064] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0065] Controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Controller 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object for display on display 260, controller 250 may perform operations related to the object selected by the user command.

[0066] In some embodiments, the controller 250 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus, etc.

[0067] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0068] A user interface is the medium for interaction and information exchange between an application or operating system and the user. It converts information between its internal form and a user-friendly format. A common user interface is the graphical user interface (GUI), which refers to a graphical user interface related to computer operations. It can be an icon, window, control, or other interface element displayed on an electronic device's display. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0069] For ease of description, some embodiments of the present application are described using the display device 200 configured with the Android system as an example, but this is not intended to limit it. The display device 200 provided in the embodiments of the present application can also be configured with other systems.

[0070] like Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0071] In some embodiments, at least one application program runs in the application layer, which can be a Window program, a system setting program, a clock program, etc. provided by the operating system, or an application program developed by a third party developer. In a specific implementation, the application programs in the application layer are not limited to the above examples.

[0072] The framework layer provides an application programming interface (API) and a programming framework for the application programs. The application framework layer includes some pre-defined functions. The application framework layer is equivalent to a processing center that decides which application program in the application layer to act. The application program can access the resources in the system and obtain the services of the system through the API interface in the execution.

[0073] As shown in Figure 4 In the embodiment of the present application, the application framework layer includes managers (Managers) and a content provider (Content Provider), wherein the managers include at least one of the following modules: an activity manager (Activity Manager) for interacting with all activities running in the system; a location manager (Location Manager) for providing access to the system location service for the system service or the application; a package manager (Package Manager) for retrieving various information related to the application package currently installed on the device; a notification manager (Notification Manager) for controlling the display and clearing of the notification message; and a window manager (Window Manager) for managing the icons, windows, toolbars, wallpapers and desktop components on the user interface.

[0074] In some embodiments, the activity manager is used to manage the life cycle of each application program and the general navigation back function, such as controlling the exit, opening, back, etc. of the application program. The window manager is used to manage all window programs, such as obtaining the size of the display screen, judging whether there is a status bar, locking the screen, intercepting the screen, controlling the display window change (such as reducing the display window, shaking the display, twisting the display, etc.), etc.

[0075] In some embodiments, the system runtime library layer provides support for the upper layer, i.e. the framework layer. When the framework layer is used, the Android operating system runs the C / C++ library included in the system runtime library layer to realize the functions of the framework layer.

[0076] In some embodiments, the kernel layer is a layer between hardware and software. As Figure 4As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0077] After establishing a communication connection with the terminal device 500, the display device 200 can establish a screen projection connection channel and transmit projection data based on the projection connection so that the display device 200 can display the projection image of the terminal device 500. In some embodiments, to establish a screen projection connection relationship, the user can issue a screen projection connection request through the terminal device 500 and send the screen projection connection request to the display device 200 via the WiFi network. The display device 200 then completes the transmission protocol configuration based on the screen projection connection request, thereby establishing a transmission channel for projection data between the display device 200 and the terminal device 500.

[0078] The display device 200 can also achieve a screen projection connection with the terminal device 500 through different connection methods. For example, when the display device 200 and the terminal device 500 are connected to the same wireless local area network, a screen projection connection can be established based on the WiFi network. For another example, when both the display device 200 and the terminal device 500 are provided with NFC (Near Field Communication) components, a screen projection connection relationship can be established through the NFC component. Obviously, other wired or wireless connection methods can also be used between the display device 200 and the terminal device 500 to establish a screen projection connection relationship, such as RF radio frequency connection, infrared connection, cellular network and other connection methods that can be associated with those skilled in the art based on the connection methods provided in the above embodiments.

[0079] After establishing the screen projection connection, the display device 200 can receive the screen projection related data from the terminal device 500 through the screen projection data channel, thereby playing the screen projection related content on the display device 200. Under different screen projection data transmission protocols, the screen projection data obtained by the display device 200 is in different forms. For example, the display device 200 and the terminal device 500 can establish a screen projection connection through the Digital Living Network Alliance (DLNA) protocol, Miracast protocol, AirPlay push protocol, WiDi screen projection, NFC mirror screen projection, LeTV screen projection, and other protocols.

[0080] The DLNA protocol is a protocol used to share media resources between devices. This protocol uses network technology (wired or wireless) to interconnect various devices and exchange data through standard protocols, allowing media resources to be shared between devices. The DLNA protocol is based on the UPnP protocol, enabling devices to discover and control each other, achieving seamless transmission and sharing of media resources. The DLNA protocol supports a variety of devices, including TVs, speakers, mobile phones, computers, and more, allowing users to seamlessly switch and share media content between different devices.

[0081] The Miracast protocol is a wireless display standard developed by the Wi-Fi Alliance and based on Wi-Fi Direct. The Miracast screen projection protocol is compatible with Android devices, allowing Android devices to share video images through Miracast. Smart devices running Android 5.0 and above have the Miracast protocol built-in, allowing direct wireless screen projection without installing any software, using the Wi-Fi P2P function between the projection device and the receiving device's Wi-Fi network card. Miracast also supports simultaneous audio and video transmission.

[0082] AirPlay is a wireless playback technology introduced by Apple, suitable for wireless screen projection on iOS and Mac systems. Apple devices have built-in AirPlay projection functionality. When used with a screen projector that supports the AirPlay server service, the display is clear and stable. The protocol also includes a built-in projection code function. However, the Apple device and screen projector must be on the same local area network (LAN segment) and cannot be used across different network segments or VLANs.

[0083] WiDi is an Intel-developed wireless streaming technology based on Wi-Fi Direct. Windows 8 / 10 / 11 laptops natively support WiDi, allowing you to cast your screen directly without installing any software. It also supports USB reverse control.

[0084] In some embodiments, after the display device 200 establishes a screen projection connection with the terminal device 500, the projection data can be transmitted through the mirror mode and the push mode. Among them, the mirror mode means that the display device 200 synchronously displays the screen content displayed by the terminal device 500. For example, Figure 5 As shown, the display device 200 displays the main interface of the terminal device 500, and when the user performs an interactive operation on the terminal device 500, resulting in a change in the content of the main interface, the content displayed by the display device 200 also changes. To this end, the terminal device 500 can generate screen projection data in the form of a video stream based on the displayed content, and send the screen projection data to the display device 200 for display.

[0085] Push mode means that the display device 200 accesses the network link pushed by the terminal device 500 and obtains the corresponding projection screen. Figure 6 As shown, when the terminal device 500 plays video media, a screen projection control is provided in the upper right corner of the playback interface. After the user clicks the screen projection control and selects the display device 200 as the device for displaying the screen projection, the terminal device 500 can send the URL address of the currently playing video media to the display device 200. After receiving the URL address, the display device 200 accesses the URL address to obtain the video media data and plays the video media.

[0086] Among the screen projection protocols supported by the display device 200 and the terminal device 500, some protocols are based on the mirror mode, and some protocols are based on the push mode. For example, under the DLNA protocol, the display device 200 can obtain the link address of a resource in the terminal device 500 through the screen projection data channel, and obtain the media content data by accessing the link address, thereby playing the media data. At the same time, the display device 200 can also obtain the control instructions sent by the terminal device 500 through the screen projection data channel when playing the media data, such as fast forward, pause, stop screen projection, etc. For example, after establishing a connection through the communication protocol, the terminal device 500 such as a mobile phone and the display device 200 will negotiate the device functions and network conditions to select a suitable audio and video transmission format. After establishing the online protocol (Session) of the audio and video stream, a series of real-time streaming protocol (RTSP) control commands are used to control the playback and termination of the audio and video stream.

[0087] Under other screen projection connection protocols such as Miracast, Airplay, NFC mirroring, and LeTV, the display device 200 can obtain a video stream, which serves as projection data. The video stream can be the video data corresponding to the final display screen of the display unit of the terminal device 500, that is, the video stream includes the media content screen, UI interface, and other video screen content that can be displayed in the display unit of the terminal device 500. The video stream can also be only the video data that can be obtained in the terminal device 500, such as the media video data being played. In this case, the terminal device 500 can serve as a signal source for the display device 200, and the display device 200 can continuously obtain video data through the projection data channel and display it in real time on the display 260, achieving a synchronized display effect with the terminal device 500. Alternatively, the video stream can also be video data obtained from the cloud, that is, the terminal device 500 can send the URL address of the video media to the display device 200. After receiving the URL address, the display device 200 accesses the URL address based on the projection protocol to obtain the projection data.

[0088] It should be noted that the above-mentioned screen projection protocol can not only obtain the video stream in the terminal device 500, but also obtain the audio data stream in the terminal device 500, so that the video data and audio data can be sent to the display device 200 synchronously, so that the display device 200 can play the video and audio synchronously.

[0089] The terminal device 500 can obtain different projection data according to different projection protocols. For example, for system applications or direct projection at the system level, the terminal device 500 can use Miracast projection, Airplay projection or WiDi projection protocol to establish a projection data channel. Since the above-mentioned Miracast projection protocol and Airplay projection protocol require system-level application permissions, the Miracast projection protocol and Airplay projection protocol are only applicable to system applications, or the display device 200 and the terminal device 500 of the same manufacturer directly establish a projection connection through the built-in rules of the operating system.

[0090] The display device 200 can establish a screen projection connection with the terminal device 500 based on any of the screen projection methods provided in the above embodiments. In some embodiments, the display device 200 can also set the layout of the screen projection window. Figure 7 As shown, the display device 200 can have multiple functional modules built into the same application, such as a screen projection application. The screen projection application includes: a data management module, a message processing module, a service management module, an algorithm processing module, a view management module, an application interaction processing module, a location management module, and a screen projection management module.

[0091] The data management module primarily manages all message data involved in the current template and synchronizes data updates. The message processing module primarily transmits related messages between modules, such as transmitting data changes to the View layer for refresh display, or synchronizing View changes to the data management module for broadcast control data change processing. The service management module is responsible for enabling and disabling services across the entire application, as well as performing customized processing, such as launching specific services in specific scenarios.

[0092] The algorithm processing module is responsible for all algorithm processing of the entire application. The View management module is mainly responsible for all UI displays of the entire template application, receiving data to refresh the View display. The application interaction processing module is responsible for processing interactions with external applications. For example, any entry to start the current template application needs to interact with this module, and different customized management is performed according to the differences in the startup entry. The screen projection management module is responsible for screen projection protocol interaction, client list management, and status information processing of window display content.

[0093] It should be noted that the screen projection application, as an application that can display multi-protocol multi-channel mirror projection and push content, is an upper-layer application within the display device 200. The framework layer of the display device 200 can provide a Java interface for docking with hardware, etc. The multi-channel screen projection viewing application uses the Java interface of the Framework layer to achieve docking with lower layers and obtain information. Other layers such as the National, Runtime, HAL, and Linux kernel are all standard layers of the Android architecture, and each layer interacts according to the business process.

[0094] The display device 200 first starts the application through the service management module and performs some initial preparations, such as starting the monitoring mirror or pushing the corresponding protocol service; the View management module initializes the View displayed by the application layer; the data management module creates some data for the data processing module; the algorithm processing module performs algorithm layout calculations on the projection content, etc.

[0095] During the process of establishing a screen projection connection, the service management module can start the screen projection application and register the Session; Usession Connection Listener, Usession Prepare Listener, Usession DisplayListener, and Usession Event Listener. Among them, Session represents a conversation between the client and the server, which can save the client's information. Usession Connection Listener is used to monitor the connection and disconnection events of the Session. Usession Prepare Listener interacts with the client's screen projection preparation. UsessionDisplay Listener monitors changes in playback information and status as well as changes in the client's status. Usession EventListener processes information such as client control.

[0096] After the View is initialized, the display device 200 will display the default boot page when entering the application. The boot page will display some device information, such as the device name of the current display device 200 and the corresponding network information. The boot page is then displayed through the control system standard TextView and ImageView controls and the corresponding Layout.

[0097] In order to facilitate the management of View and Session, in some embodiments, a management object MultiCastPlayView can be set, where the management object MultiCastPlayView can be associated with information such as Session, SurfaceView / TextureView, mViewPosition, etc. For example, for terminal device A, the data corresponding to its Client can be represented as: Session:SessionClient, View:SurfaceView / TextureView.

[0098] When the display device 200 prepares the View as the server, it can notify the terminal device 500 as the mobile terminal to record or send the video stream (i.e., the projection data) through UsesionPrepareListener. When the terminal device 500 is ready to send the video information, the display device 200 knows that the projection is to be played through onFirstFrame() of UsesionDisplayListener. Therefore, it can obtain the current client video width and height from the Session, determine the direction of the current video stream based on the width and height, and then use the algorithm to calculate the current position and size to be displayed, i.e., the initial position and size.

[0099] That is, in some embodiments, the display device 200 obtains a screen projection session (Session) sent by the terminal device 500. The screen projection session includes the resolution of the picture frame. The screen resolution of the display device is then detected, and the initial display parameters of the projection screen are calculated based on the resolution of the picture frame and the screen resolution. The initial display parameters are used to set the center point of the picture frame to the center point of the user interface, to make the aspect ratio of the projection screen equal to the aspect ratio of the picture frame, and to maximize the proportion of the picture frame in the user interface. Then, the display 260 is controlled to display the projection screen according to the initial display parameters.

[0100] For example, when the resolution of the terminal device 500 is 1920×1080 and the screen resolution of the display device 200 is 1920×1080, the projection data sent by the terminal device 500 to the display device 200 is as follows: Figure 8As shown in the left part. The algorithm processing module obtains the screen resolution of the screen frame in the projection data from the projection session (Session) as 1920×1080, and then calculates the screen display area of ​​the projection screen as the full-screen area (i.e., the entire user interface) through the screen resolution 1920×1080 and the picture resolution 1920×1080, that is, the projection screen: width 1920, height 1080, distance from the upper left corner position (0, 0). The data management module of the display device 200 can sort and store the data obtained by the algorithm processing module, and then transmit the data, including the initial display parameters: Session: SessionClient, View: SurfaceView / TextureView, Position: (0, 0), Width: 1920, Height: 1080.

[0101] After receiving the data information, the View management module of the display device 200 sets the display parameters through LayoutParams, including the position (i.e., coordinates) from the top and left boundaries, as well as the corresponding display width and height, and then displays the position through the setLayout outParams method. That is:

[0102] FrameLayout.LayoutParams lp1=new FrameLayout.LayoutParams(width,height,0); lp1.width=1920; lp1.height=1080; lp1.leftMargin=0; lp1.topMargin=0; mPlayerView.set LayoutParams(lp1).

[0103] By setting the display position of the projection SurfaceView / TextureView, the display device 200 can display Figure 8 The projection screen is shown on the right.

[0104] It should be noted that the data associated with the management objects and the parameters of the algorithm processing provided in the above embodiment are only exemplary descriptions and are not limitations thereto. Other data objects may also be used in the embodiments of the present application.

[0105] In some embodiments, the projection data sent by the terminal device 500 to the display device 200 includes a picture frame for forming a projection screen, and the picture frame includes a valid picture area and an invalid picture area. Among them, from a visual perspective, the valid picture area can be used to present picture content for users to watch. That is, the valid picture area can be an area in the picture frame with picture content for users to watch, and the invalid picture area can be an area in the picture frame other than the valid picture area. For example, the invalid picture area can be located on the periphery of the valid picture area and has no substantial effect. In other words, the invalid picture area can be an area in the picture frame that has no practical meaning and is an area that the user does not pay attention to. In some embodiments, the invalid picture area will present a single color effect, such as the invalid picture area can be a solid color area such as black or gray.

[0106] In some embodiments, the display device 200 can scan all pixels in the picture frame, which can include row-by-row scanning and column-by-column scanning, such as first scanning each pixel of the picture frame row by row and then scanning each pixel of the picture frame column by column, or first scanning each pixel of the picture frame column by column and then scanning each pixel of the picture frame row by row. Of course, the above scanning order is not unique. Then, the area where the color values ​​of the entire row and array pixels are all the target values ​​is marked as an invalid picture area. The color value is a characteristic value that characterizes the color of the pixel, such as an RGB value.

[0107] For example, a pixel with an RGB value of (0, 0, 0) is a pixel with no display content, and the area in the picture frame composed of pixels with an RGB value of (0, 0, 0) is an invalid picture area. Figure 8 The area a shown in the figure is the effective image area; the area composed of pixels of other color values ​​is the effective image area, such as Figure 8 As shown in area b, since the RGB values ​​of the pixels in the invalid screen area are not set, a pure color such as black may be displayed.

[0108] In some embodiments, when the target value is the color value corresponding to black, the pixel points of the entire row of the picture frame whose color value is the target value are marked as an invalid picture area, and the pixel points of the entire column of the picture frame whose color value is the target value are marked as an invalid picture area. At this time, the display effect presented by the invalid picture area is black, and the invalid picture area can also be referred to as a black edge or a black edge area. Similarly, when the target value is the color value corresponding to gray, the display effect presented by the invalid picture area is gray, and the invalid picture area can also be referred to as a gray edge or a gray edge area. It is understandable that the principles for other colors are the same as above and will not be elaborated here.

[0109] For example, Figure 9aAs shown, when the terminal device 500 is in a horizontal screen state, the media played by the terminal device 500 has an invalid screen area, and the terminal device 500 displays the terminal interface 910; at this time, the terminal device 500 starts recording and sends the projection data to the display device 200. The picture frame 920 in the projection data includes an invalid screen area 922 and a valid screen area 921. The display device 200 displays the following based on the projection data: Figure 9a The projection screen 930 is shown.

[0110] like Figure 9b As shown, when the terminal device 500 is Figure 9a When the horizontal screen state is switched to the vertical screen state, the media played by the terminal device 500 still has an invalid screen area 922, and the terminal device 500 displays the terminal interface 910; at this time, the terminal device 500 starts recording and sends the projection data to the display device 200. The picture frame 920 in the projection data includes the invalid screen area 922 and the valid screen area 921. The display device 200 displays the following based on the projection data: Figure 9b The projection screen 930 is shown.

[0111] Obviously, by Figures 9a-9b As can be seen from the screen projection process shown, when the picture frame of the screen projection data sent by the terminal device 500 includes an invalid picture area, the display device 200 will also completely display the invalid picture area 922 in the user interface, and the invalid picture area will occupy a certain display space of the display device 200.

[0112] Therefore, if Figure 11 As shown, in some embodiments, the display device 200 detects the invalid screen area of ​​the picture frame and calculates the pre-processed width of the picture frame in the first direction and the pre-processed height of the picture frame in the second direction based on the invalid screen area (S1101). The pre-processed width is the width of the valid screen area 921, and the pre-processed height is the height of the valid screen area 922. The first direction is perpendicular to the second direction, such as the x direction and the y direction; the pre-processed width is the difference between the picture frame width and the width of the invalid screen area in the first direction, and the pre-processed height is the difference between the picture frame height and the height without content in the second direction. Then, the first ratio of the picture frame width to the pre-processed width is calculated, and the second ratio of the picture frame height to the pre-processed height is calculated (S1102). The first ratio is set as the scaling ratio of the projection screen in the first direction, and the second ratio is set as the scaling ratio of the projection screen in the second direction (S1103). Finally, the display device 200 scales the width of the projection screen according to the first ratio and the height of the projection screen according to the second ratio (S1104). That is, the scaling ratios of the first direction and the second direction are calculated respectively, and the projection image is scaled according to the scaling ratios.

[0113] For example, the resolution of the terminal device 500 is 1920×1080, and the screen resolution of the display device 200 is 1920×1080. Figure 10 As shown, the first direction is the x direction, the second direction is the y direction, the terminal device 500 records its own terminal interface based on the screen projection protocol, forms screen projection data, and sends the screen projection data to the display device 200. Among them, the screen frame 920 of the screen projection data and the invalid screen area 922 and the invalid screen area 922, as shown Figure 10 , the invalid screen area 922 is (128, 72), so when the algorithm processing module of the display device 200 calculates the preprocessing height, 72 pixels are subtracted from both sides in the y direction, and the preprocessing height is 1080-2*72=936; similarly, when the display device 200 calculates the preprocessing width, 128 pixels are subtracted from both sides in the x direction, and the preprocessing width is 1920-2*128=1664.

[0114] After the display device 200 calculates the pre-processed width and pre-processed height, it calculates the ratio of the width and height of the original size of the picture frame to the pre-processed height and pre-processed width. The first ratio is 1920 / 1664, and the second ratio is 1080 / 936. That is, scaleX = (float) 1920 / (float)(1920-2*128) = 1.15f, scaleY = (float) 1080 / (float)(1080-2*72) = 1.15f. Then, the display device 200 sets the position of the View through the View management module. Set scaleX to the scaling ratio of the projection screen (View: Surface View / TextureView) in the x direction, and set scaleY to the scaling ratio of the projection screen in the y direction. In this way, when the display device 200 displays the picture frame of the projection data, it will be presented. Figure 10 The projection screen 930 shown on the right can reduce the invalid screen area in the user interface of the display device 200.

[0115] During the screen projection process between the terminal device 500 and the display device 200, if the user manipulates the terminal device 500 to change the data information of each frame 920 in the projection data, the projection screen 930 displayed on the display device 200 may also change accordingly. Figure 12a In the mirror mode projection process shown, the user opens the gallery of the terminal device 500 and quickly switches between different photos. The invalid screen area displayed in the terminal interface is constantly changing. Correspondingly, in the process of the user switching photos, the projection data recorded by the terminal device 500 and the invalid screen area 922 of the picture frame 930 are constantly changing. The display device 200 scales the projection screen 930 according to the invalid screen area 922, and the projection screen 930 appears as follows. Figure 12a The display effect is shown on the right.

[0116] For example, Figure 12b In the mirror mode projection process shown, the user switches the terminal device 500 from the vertical screen state to the horizontal screen state, and the invalid screen area 922 in the terminal interface of the terminal device 500 is constantly changing; accordingly, in the projection data recorded by the terminal device 500, the invalid screen area 922 of each frame 930 will also change; the display device 200 scales the projection screen 930 according to the invalid screen area 922, and the following will be displayed. Figure 12b The display effect is shown on the right.

[0117] Depend on Figure 12a-12b As can be seen from the process shown, when the invalid image area 922 of the screen frame in the projection data changes frequently, the display device 200 needs to continuously calculate the scaling ratio based on the changed invalid image area 922 and scale the projection image according to the calculated scaling ratio. This not only increases the resource consumption of the display device 200, but also causes the effective image area of ​​the projection image 930 to appear to be larger and smaller in a short period of time, affecting the user's viewing experience.

[0118] Based on the above application scenarios, in some embodiments, such as Figure 13 As shown, the display device 200 may include a display 260, a communicator 220, and a controller 250. The display 260 is configured to display a user interface 261, which includes a projection screen of the terminal device 500; the communicator 200 is configured to establish a communication connection with the terminal device 500 and receive projection data sent by the terminal device 500, wherein the projection data includes a picture frame 920 for forming a projection screen, and the picture frame 920 includes an effective picture area 921 and an invalid picture area 922; the controller 250 is configured to execute the following program steps:

[0119] S1301: Detect characteristic information of the projection data within a preset time period.

[0120] The display device 200 establishes a communication connection with the terminal device 500 through the communicator 220, and receives the projection data generated by the terminal device 500. Then, within a preset time period, the characteristic information in the projection data is detected. Among them, the characteristic information may include various data parameters of the picture frame 930, which is used to determine whether the projection data is stable within the preset time period. For example, the preset time period may be 500ms, and the display device 200 detects the characteristic information of the projection data within 500ms. Within the preset time period, when the characteristic information changes, it can be determined that the state of the projection data in the preset time period is unstable; when the characteristic information does not change, it can be determined that the state of the projection data in the preset time period is stable.

[0121] When detecting feature information, the display device 200 can detect it according to one or more of the following principles: 1. Whether the valid picture area or invalid picture area in two or more consecutive picture frames 930 in the projection data has changed; 2. Whether the resolution of the projection data has changed; 3. Whether the size of the projection data has changed; 4. Whether the key frame of the projection data has changed; 5. Whether the projection data has changed, etc.

[0122] In some embodiments, the feature information may include at least one of the following: active screen area 921, inactive screen area 922, resolution, timestamp, keyframe, frame size, and video identifier. That is, the feature information may include one or more combinations of the above, all of which may affect the display quality of the projection screen 930.

[0123] The following describes the above-mentioned data information respectively.

[0124] For the valid screen area 921 and the invalid screen area 922: Since the valid screen area 921 and the invalid screen area 922 are two relative data parameters, when one of them changes, the other must also change. Therefore, the characteristic information may include only the valid screen area 921, may include only the invalid screen area 922, or may include both the valid screen area 921 and the invalid screen area. For example, during the screen projection process in mirror mode, when the user switches the screen displayed in the terminal device 500 or changes the placement status of the terminal device 500, the invalid screen area and the valid screen area in the terminal interface 910 change, and the invalid screen area 922 and the valid screen area 921 of each screen frame 930 in the projection data also change. When the valid screen area 921 or the invalid screen area 922 changes, it means that the user may be operating, and the display device 200 can determine that the status of the current projection data is unstable.

[0125] In some embodiments, the display device 200 can detect the effective screen area 921 of the image frame and obtain first area information of the effective screen area 921 in the image frame. The first area information includes the width and height of the effective screen area and the position of the effective screen area in the image frame, such as the coordinates of the upper left corner or the center point. If the first area information of the image frame is inconsistent, the effective screen area 921 is marked as changed; if the first area information of the image frame is consistent, the effective screen area 921 is marked as unchanged.

[0126] In some embodiments, the display device 200 may further detect the invalid screen area 922 of the picture frame and obtain second region information of the invalid screen area 922 in the picture frame. The second region information includes the width and height of the invalid screen area 922 and the position of the invalid screen area 922 in the picture frame 920. If the second region information of the picture frames is inconsistent, the invalid screen area 922 is marked as changed; if the second region information of the picture frames is consistent, the invalid screen area 922 is marked as unchanged.

[0127] It should be noted that, in some embodiments, the display device 200 may also obtain the first region information and the second region information simultaneously, which is not limited in this application.

[0128] Regarding resolution and screen frame size. For example, during the screen projection process, the user can modify the resolution of the projection data in the projection settings or projection application of the terminal device 500; and some projection protocols also allow users to customize the resolution and size of the projection screen. When the resolution and screen frame size change, it means that the user may be operating, and the display device 200 can determine that the state of the projection data is unstable.

[0129] For the timestamp, key frame and video identifier (the unique identifier of the projection data, which may be different in different devices, applications and projection protocols), for example, during the projection process, when the timestamp, key frame and video identifier of the projection data change, it means that the projection data sent by the terminal device 500 has changed, and the display device 200 may also determine that the status of the projection data is unstable.

[0130] It should be noted that the feature information is described above through several data information, but it is not limited thereto. The feature information provided in the embodiment of the present application can be other data information of the projection data. In some embodiments, the display device 200 can also set the feature information according to the projection screen protocol established with the terminal device 500.

[0131] In some embodiments, as Figure 14a As shown, when the time between the current frame 1502 and the initial frame is less than or equal to the preset time period, the display device 200 uses the time of the initial frame 1401 as the timing starting point to obtain the first frame set of the preset time period. The initial frame is the earliest frame in the projection data (the first frame). Then, the feature information of the frames in the first frame set is parsed.

[0132] When the feature information of any two frames in the first frame set is inconsistent, the display device 200 determines that the feature information has changed within a preset time period; otherwise, the display device 200 determines that the feature information has not changed within the preset time period. That is, if the feature information is inconsistent, the display device 200 marks that the feature information of the frames in the first frame set has changed within the preset time period; if the feature information is consistent, the display device 200 marks that the feature information of the frames in the first frame set has not changed within the preset time period.

[0133] For example, the preset time period is 500ms, and the characteristic information includes the invalid screen area 922 and the screen frame size. After receiving the first frame of the projection data, the display device 200 turns on the timing function. Taking the time of the first frame as the starting point, each frame of the projection data within 500ms is obtained to generate a first frame set, and the characteristic information of each frame obtained is stored. Then compare the characteristic information of each frame in the first frame set. If the invalid screen area of ​​any two frames in the first frame set, or the frame size of any two frames is inconsistent, the marked characteristic information changes within 500ms; otherwise, the marked characteristic information does not change within 500ms.

[0134] In some embodiments, when the time between the current frame and the initial frame is greater than a preset time period, such as Figure 14b As shown, the display device 200 uses the time of the current frame 1402 as the timing end point and obtains a second frame set of a preset time period. Then, feature information of the frames in the second frame set is parsed. If the feature information of the frames in the second frame set is inconsistent, the feature information is marked as having changed within the preset time period. If the feature information of the frames in the second frame set is consistent, the feature information is marked as having not changed within the preset time period.

[0135] For example, the preset time period is 500ms, and the characteristic information includes the invalid screen area 922 and the screen frame size. After the display device 200 receives the initial screen frame time for 500ms, the display device 200 uses the time of the current screen frame as the timing end point, obtains each screen frame of the projection data within 500ms to generate a second screen frame set, and stores the characteristic information of each screen frame obtained. Then compare the characteristic information of each screen frame in the second screen frame set. If the invalid screen area of ​​any two screen frames in the second screen frame, or the screen frame size of any two screen frames is inconsistent, the marked characteristic information changes within 500ms; otherwise, the marked characteristic information does not change within 500ms.

[0136] Or, as Figure 15As shown, in some embodiments, the display device 200 sets an acquisition cycle when receiving the initial picture frame 1401 of the projection data. The initial picture frame is the earliest picture frame in the projection data, and the acquisition cycle is equal to the preset time period. Then, the picture frames within the preset time period are acquired according to the acquisition cycle to generate a third picture frame set. The feature information of the picture frames in the third picture frame set is then parsed. If the feature information of the picture frames in the third picture frame set is inconsistent, the feature information is marked as having changed within the preset time period; if the feature information of the picture frames in the third picture frame set is consistent, the feature information is marked as having not changed within the preset time period.

[0137] In some embodiments, when setting the acquisition cycle, the display device 200 can perform polling checks using Handler and Runnable, using the postDelayed method to execute a Runnable task every 500ms. Alternatively, the display device 200 can use the Timer and TimerTask classes to create a Timer object and use the schedule method of the Timer object to set the execution time period.

[0138] In some embodiments, the preset time period can be set based on the projection data sent by the terminal device 500. For example, the display device 200 can set the preset time period based on the frame rate of the projection data; the display device 200 presets the number of detected picture frames, and then sets the value of the preset time period based on the preset number of picture frames and the frame rate of the projection data.

[0139] S1302: If the characteristic information changes, maintain the display parameters of the projection image.

[0140] The display device 200 detects the characteristic information of the projection data. When it is detected that the characteristic information has changed within a preset time period, it indicates that the state of the projection data is unstable. The display device 200 maintains the current display parameters of the projection screen and does not perform scaling or other processing on it, so that the projection screen maintains the original display effect.

[0141] For example, the resolution of the terminal device 500 is 1920×1080, the screen resolution of the display device 200 is 1920×1080, the characteristic information is the invalid screen area, and the preset time period is 500ms. Figure 16a As shown, the terminal device 500 and the display device 200 establish a screen projection connection based on the mirror mode, and the user switches pictures in the gallery in the terminal device 500. Figure 16aAs shown in the picture frames a-c, when the display device 200 detects that the invalid picture area of the screen mirroring data changes within 500 ms, the display parameter of the screen mirroring picture is maintained, and no processing is performed. In this way, the consumption of system resources of the display device 200 can be reduced, and the problem of the effective picture area 921 being too large or too small in the user interface can be improved.

[0142] S1303: If the feature information does not change, the target display parameter of the screen mirroring picture is calculated according to the invalid picture area of the current picture frame, and the display is controlled to display the screen mirroring picture according to the target display parameter; the target display parameter is used to set the center point of the effective picture area in the current picture frame to the reference center point of the user interface, and to maximize the display proportion of the effective picture area in the user interface.

[0143] In a preset time period, the display device 200 detects the feature information of the screen mirroring data. When it is detected that the feature information does not change, it indicates that the state of the screen mirroring data has stabilized, and the display device 200 can calculate the target display parameter of the screen mirroring picture 930 according to the invalid picture area 922 in the current picture frame. The target display parameter is used to set the center point of the effective picture area 921 to the reference center point of the user interface, reduce the invalid picture area 922 displayed in the user interface, and maximize the display proportion of the effective picture area 921 in the user interface.

[0144] That is, when the state of the screen mirroring data is stable, the display device 200 calculates the target display parameter of the screen mirroring picture according to the above principle. The screen mirroring picture 930 corresponding to the target display parameter can maximize the reduction of the invalid picture area 922 in the user interface, and can display the center of the effective picture area 921 at the center of the user interface (screen center point), and can display the effective picture area 921 in the center.

[0145] For example, when the invalid picture area 922 is a black border area, the screen mirroring picture displayed by the display device 200 according to the target display parameter can maximize the reduction of the black border area in the user interface, maximize the display of the effective picture area 921 in the user interface, and can keep the effective picture area 921 displayed in the center of the user interface.

[0146] In some embodiments, when calculating the target display parameters, the display device 200 can use the reference center point as the center of symmetry to compare the invalid screen area 922 in the first direction and the second direction in the current screen frame to determine whether the invalid screen area 922 is symmetrical. When determining whether the invalid screen area 922 is symmetrical, the display device 200 can use an image processing algorithm to determine whether the invalid screen area 922 is centrally symmetrical based on the reference center point; or, the display device 200 can also use a geometric symmetry detection algorithm to determine whether the invalid screen area 922 is centrally symmetrical based on the reference center point; or, the display device 200 can also obtain the margins of the invalid screen area 922 from the upper, lower, left, and right boundaries of the user interface (or projection window) to determine whether the invalid screen area 922 is centrally symmetrical based on the reference center point. For ease of description, in the embodiment of the present application, the margins of the invalid screen area 922 from the upper, lower, left, and right boundaries of the user interface are represented as top, bottom, left, and right, respectively.

[0147] It is understood that the present embodiment exemplifies two methods for determining whether the invalid screen area is symmetrical, but is not intended to be limiting. The method for determining whether the invalid screen area 922 is symmetrical may also be implemented based on other methods, such as using a third-party algorithm library or service framework.

[0148] In some embodiments, as Figure 17 As shown, the display device 200 detects the first invalid screen area 922a and the second invalid screen area 922b of the screen frame in the first direction, and detects the third invalid screen area 922c and the fourth invalid screen area 922d in the second direction. Then calculate the first margin (left) of the first invalid screen area 922a in the first direction, the second margin (right) of the second invalid screen area 922b in the first direction, the third margin (top) of the third invalid screen area 922c in the second direction, and the fourth margin (bottom) of the fourth invalid screen area in the second direction. That is, the margin corresponding to the first invalid screen area 922a is left, the margin corresponding to the second invalid screen area 922b is right, the margin corresponding to the third invalid screen area 922c is top, and the margin corresponding to the fourth invalid screen area 922d is bottom.

[0149] After calculating the margins of the invalid screen area 922 in each direction, if the first margin is equal to the second margin, and the third margin is equal to the fourth margin, then the invalid screen area 922 is marked as being centrally symmetrical about the reference center point; conversely, if the first margin is not equal to the second margin, and / or the third margin is not equal to the fourth margin, then the invalid screen area 922 is marked as not being centrally symmetrical about the reference center point. That is, when the two margins in the x-direction (left and right) are not equal; or the two margins in the y-direction (top and bottom) are not equal; or the two margins in the x-direction are not equal and the two margins in the y-direction are also not equal, the display device 200 marks the invalid screen area 922 as not being centrally symmetrical about the reference center point.

[0150] It should be noted that the picture frame of the embodiment of the present application may not have one or more of the first invalid picture area, the second invalid picture area, the third invalid picture area and the fourth invalid picture area, that is, the value of any one of the values ​​of top, bottom, left and right can be 0.

[0151] In some embodiments, as Figure 18 As shown, the display device 200 detects the invalid screen area 922 of the current screen frame, and then compares the invalid screen area 922 in the first direction and the second direction with the reference center point as the center of symmetry. If the invalid screen area 922 is centrally symmetrical with the reference center point, the pre-processed width of the screen frame 920 in the first direction and the pre-processed height of the screen frame in the second direction are calculated (S1801). Wherein, the pre-processed width is the difference between the screen frame width and the invalid screen area width in the first direction, and the pre-processed height is the difference between the screen frame height and the invalid screen area height in the second direction. Then, the first ratio of the screen frame width to the pre-processed width is calculated, and the second ratio of the screen frame height to the pre-processed height is calculated (S1802).

[0152] When the display device 200 calculates that the first ratio and the second ratio are equal, one of the ratios is set as the scaling ratio of the projected image in the first direction and the second direction. That is, when the first ratio and the second ratio are equal, the display device 200 sets the first ratio as the scaling ratio of the projected image in the first direction and the second direction, or sets the second ratio as the scaling ratio of the projected image in the first direction and the second direction, to generate the target display parameters corresponding to the current frame (S1803).

[0153] Accordingly, in some embodiments, when the first ratio is set as the scaling ratio of the projected image 930 in the first direction to the second direction, the display device 200 scales the width of the projected image 930 according to the first ratio, and scales the height of the projected image 930 according to the first ratio. Similarly, when the second ratio is set as the scaling ratio of the projected image 930 in the first direction to the second direction, the display device 200 scales the width of the projected image 930 according to the second ratio, and scales the height of the projected image 930 according to the second ratio.

[0154] In some embodiments, when the first ratio is not equal to the second ratio, in order to ensure that the projected screen can be displayed according to the original ratio of the effective screen area 921 after scaling, the display device 200 obtains the minimum value of the first ratio and the second ratio (S1804), and then sets the minimum value to the scaling ratio of the projected screen in the first direction, and sets the minimum value to the scaling ratio of the projected screen in the second direction to generate the target display parameters corresponding to the current screen frame (S1805).

[0155] Accordingly, in some embodiments, when the display device 200 displays the projection screen 930 according to the target, the width of the projection screen 930 is scaled according to the minimum value, and the height of the projection screen 930 is scaled according to the minimum value.

[0156] For example, the resolution of terminal device 500 is 1920×1080, and the screen resolution of display device 200 is 1920×1080. Display device 200 calculates that the invalid screen area is centrosymmetrical about the reference center point, and the left and right heights of invalid screen area 922 are 100, and the top and bottom are 50. Display device 200 calculates the preprocessing height as 1080-2*50=980, and the preprocessing width as 1920-2*100=1720. Then, the first scale in the x-direction and the second scale in the y-direction are calculated, namely, scaleX=(float)1920 / (float)(1920-2*100)=1.12f, and scaleY=(float)1080 / (float)(1080-2*50)=1.10f. The display device 200 obtains the minimum value 1.10f between the first ratio and the second ratio as the scaling ratio of the projection image in the x direction and the y direction.

[0157] like Figure 19As shown, in some embodiments, if the invalid picture area 922 of the current picture frame is not centrally symmetrical about the reference center point, that is, the two invalid picture areas 922 in the x-direction are asymmetrical with each other, or the two invalid picture areas 922 in the y-direction are asymmetrical with each other, or the two invalid picture areas 922 in the x-direction are asymmetrical and the two invalid picture areas 922 in the y-direction are also asymmetrical, the display device 200 calculates the pre-processed width of the picture frame 920 in the first direction and the pre-processed height of the picture frame in the second direction (S1901).

[0158] After calculating the preprocessing width and the preprocessing height, the display device 200 calculates the target center point of the picture frame 920 under the preprocessing width and the preprocessing height (S1902), that is, the center point of the effective picture area 921. Then, the center offset is calculated according to the target center point and the reference center point (S1903), and the position parameters of the projection screen are set based on the center offset (S1904). Then, the first ratio of the picture frame width to the preprocessing width is calculated, and the second ratio of the picture frame height to the preprocessing height is calculated (S1905). Similarly, in order to enable the projection screen to be displayed according to the original ratio of the first target area (effective picture area 921), the minimum value of the first ratio and the second ratio is obtained (S1906), and the minimum value is set as the scaling ratio of the projection screen in the first direction, and the minimum value is set as the scaling ratio of the projection screen in the second direction to generate the target display parameters corresponding to the current screen frame (S1907).

[0159] Correspondingly, in some embodiments, when the display device 200 displays the projection screen 930 according to the target display parameters, the position of the projection screen in the user interface is moved according to the position parameters (S1908), and then the width of the projection screen is scaled according to the minimum value, and the height of the projection screen is scaled according to the minimum value (S1909).

[0160] The following describes two examples of the display device 200 executing steps S1301-S1305. In the examples, the resolution of the terminal device 500 is 1920×1080, the screen resolution of the display device 200 is 1920×1080, and the preset time period is 500 ms.

[0161] Example 1: The characteristic information is the effective screen area 921 and the resolution. The user operates the terminal device 500 to switch the picture content displayed in the terminal interface. The terminal interface of the terminal device 500 is as follows: Figure 16aThe display device 200 detects the feature information of the projection data within 500ms and finds that the resolution of the projection data remains unchanged at 1920×1080 within 500ms, but the effective image area 921 increases from frame a to frame b and decreases from frame b to frame c. At this time, the display device 200 maintains the display parameters of the projection image 930 and does not perform any processing.

[0162] When the user stops operating the terminal device 500, Figure 16b As shown, the display device 200 detects the characteristic information of the projection data within 500ms, and detects that the resolution of the projection data remains unchanged at 1920×1080 within 500ms, and the effective picture area 921 from picture frame d to picture frame f has not changed. At this time, the display device 200 calculates the target display parameters of picture frame f, and displays the projection picture 930 according to the calculated target display parameters. That is, the algorithm processing module detects that left and right are 656.5, and top and bottom are 154. The preprocessing height is 1080-2*154=772, and the preprocessing width is calculated to be 1080-2*656.5=607. Then, the first scale in the x direction and the second scale in the y direction are calculated, that is, scaleX = (float)1920 / (float)(1920-2*656.5) = 3.16f, scaleY = (float)1080 / (float)(1080-2*154) = 1.40f. Then, the minimum value 1.40f of the first scale and the second scale is obtained as the scaling ratio of the projection screen, so that the View management module scales the projection screen 930 according to 1.40f, and displays as follows Figure 16b The right part shows the display effect of the projection screen f.

[0163] Example 2: The feature information is the effective screen area 921, resolution and video identifier. The user switches the terminal device 500 from the vertical screen state to the horizontal screen state, and the terminal interface of the terminal device 500 is displayed as follows: Figure 20a The display device 200 detects the feature information of the projection data within 500ms and finds that the resolution of the projection data remains unchanged at 1920×1080 within 500ms, and the video identifier does not change. However, the effective image area 921 from frame g to frame h decreases, while the effective image area from frame h to frame i increases. At this time, the display device 200 maintains the display parameters of the projection image and does not perform any processing.

[0164] When the terminal device 500 is stabilized in the horizontal screen state, Figure 20bAs shown, the display device 200 detects the characteristic information of the projection data within 500ms, and detects that the resolution of the projection data remains unchanged at 1920×1080 within 500ms, and the effective screen area 921 from frame j to frame l has not changed. At this time, the display device 200 calculates the target display parameters of frame l and displays the projection screen 930 according to the calculated target display parameters. That is, the algorithm processing module detects that the left and right are 90, and the top and bottom are 50. The preprocessing height is 1080-2*50=980, and the preprocessing width is calculated to be 1920-2*90=1740. Then, the first ratio in the x direction and the second ratio in the y direction are calculated, that is, scaleX=(float)1920 / (float)(1920-2*90)=1.10f, scaleY=(float)1080 / (float)(1080-2*50)=1.10f. Then, 1.10f is used as the scaling ratio of the projection screen, so that the View management module scales the projection screen according to 1.1f, and the display is as follows: Figure 20b The right part shows the display effect of the projection screen l.

[0165] Based on the above-mentioned display device 200, some embodiments of the present application also provide a method for detecting the area of ​​screen projection, which is applied to the display device 200 described in the above-mentioned embodiment, and the display device 200 may include a display 260, a communicator 220 and a controller 250. The display 260 is configured to display a user interface 261, and the user interface 261 includes the projection screen of the terminal device 500; the communicator 200 is configured to establish a communication connection with the terminal device 500, and receive the projection data sent by the terminal device 500, wherein the projection data includes a picture frame for forming a projection screen, and the picture frame includes a valid picture area and an invalid picture area. Figure 13 As shown, the method includes the following procedural steps:

[0166] S1301: Detecting feature information of projection data within a preset time period;

[0167] S1302: If the characteristic information is changed, maintain the display parameters of the projection image;

[0168] S1303: If the characteristic information has not changed, calculate the target display parameters of the projected screen based on the invalid screen area of ​​the current screen frame, and control the display to display the projected screen according to the target display parameters; the target display parameters are used to set the center point of the valid screen area in the current screen frame to the reference center point of the user interface, and to maximize the display proportion of the valid screen area in the user interface.

[0169] It should be noted that the above examples are only described for a few scenarios, but the screen display method based on projection data provided in the embodiments of the present application is not limited to the above scenarios, and the screen display method for other scenarios also needs to follow the above principles, which will not be elaborated here.

[0170] It can be seen from the above technical solutions that some embodiments of the present application provide a screen display method based on projection data, which can receive the projection data sent by the terminal device and detect the characteristic information of the projection data in a preset time period. If the characteristic information changes within the preset time period, the display parameters of the projection screen are maintained; otherwise, the target display parameters of the projection screen are calculated based on the invalid screen area of ​​the current screen frame, and the display is controlled to display the projection screen according to the target display parameters. The target display parameters are used to set the center point of the effective screen area in the current screen frame to the reference center point of the user interface, and to maximize the display proportion of the effective screen area in the user interface. The method can remove the invalid screen area displayed by the display device to the greatest extent when the projection data is in a stable state, and maintain the centering display effect of the effective screen area, thereby improving the display effect of the projection screen while reducing system resource consumption.

[0171] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be repeated here.

[0172] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention or certain portions of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0174] For the sake of explanation, the foregoing descriptions have been presented in terms of specific embodiments. However, it is to be appreciated that specific embodiments described herein are not intended to limit the scope of the present application, which is defined with reference to the following claims. Various modifications and changes can be made thereto by those skilled in the art which fall within the scope of the present application as defined by the following claims. The embodiments were chosen and described in order to explain the principles of the application and the practical application and to enable others skilled in the art to understand for implementing various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A display device, characterized in that: include: A display configured to display a user interface, wherein the user interface includes a projection screen of the terminal device; a communicator configured to establish a communication connection with a terminal device and receive projection data sent by the terminal device, wherein the projection data includes a picture frame for forming the projection picture, and the picture frame includes a valid picture area and an invalid picture area; The controller is configured as: Detecting characteristic information of the projection data within a preset time period; If the characteristic information changes, maintaining the display parameters of the projection screen; If the characteristic information has not changed, calculating the target display parameters of the projection image according to the invalid image area of ​​the current image frame, and controlling the display to display the projection image according to the target display parameters; The target display parameter is used to set the center point of the effective picture area in the current picture frame to the reference center point of the user interface, and to maximize the display proportion of the effective picture area in the user interface.

2. The display device according to claim 1, wherein The feature information includes at least one of the valid picture area, the invalid picture area, resolution, timestamp, key frame, picture frame size, and video identifier.

3. The display device according to claim 1, wherein When the time length between the current picture frame and the initial picture frame is less than or equal to the preset time period, the controller detects characteristic information of the projection data in the preset time period, and is configured to: Taking the time of the initial picture frame as the timing starting point, obtaining the first picture frame set of the preset time period, wherein the initial picture frame is the earliest picture frame in the projection data; parsing feature information of the picture frames in the first picture frame set; If the feature information of the picture frames in the first picture frame set is inconsistent, marking that the feature information has changed within the preset time period; If the feature information of the frames in the first frame set is consistent, it is marked that the feature information has not changed within the preset time period.

4. The display device according to claim 3, wherein When the time between the current picture frame and the initial picture frame is greater than the preset time period, the controller detects characteristic information of the projection data in the preset time period, and is configured to: Taking the time of the current picture frame as the timing end point, obtaining a second picture frame set of the preset time period; parsing feature information of the picture frames in the second picture frame set; If the feature information of the picture frames in the second picture frame set is inconsistent, marking that the feature information has changed within the preset time period; If the feature information of the frames in the second frame set is consistent, it is marked that the feature information has not changed within the preset time period.

5. The display device according to claim 1, wherein: The controller is further configured to: When receiving the initial picture frame of the projection data, setting an acquisition period; the initial picture frame is the earliest picture frame in the projection data, and the acquisition period is equal to the preset time period; Acquire picture frames within the preset time period according to the acquisition cycle to generate a third picture frame set; parsing feature information of the picture frames in the third picture frame set; If the feature information of the picture frames in the third picture frame set is inconsistent, marking that the feature information has changed within the preset time period; If the feature information of the frames in the third frame set is consistent, it is marked that the feature information has not changed within the preset time period.

6. The display device according to claim 1, wherein The controller is further configured to: Obtaining a screen projection session sent by the terminal device, where the screen projection session includes a resolution of the picture frame; Detect the screen resolution of the display device; Calculating initial display parameters of the projection image according to the resolution of the image frame and the screen resolution; The initial display parameters are used to set the center point of the picture frame at the center point of the user interface, make the aspect ratio of the projected picture equal to the aspect ratio of the picture frame, and maximize the proportion of the picture frame in the user interface; Control the display to display the projection screen according to the initial display parameters.

7. The display device according to claim 1, wherein The controller calculates the target display parameters of the projection screen according to the invalid screen area of ​​the current screen frame, and is configured to: Taking the reference center point as the center of symmetry, comparing the invalid image areas of the current image frame in a first direction and a second direction, where the first direction is perpendicular to the second direction; If the invalid picture area is centrally symmetric with respect to the reference center point, calculating a pre-processed width of the picture frame in the first direction and a pre-processed height of the picture frame in the second direction; The pre-processing width is the difference between the picture frame width and the invalid picture area width in the first direction, and the pre-processing height is the difference between the picture frame height and the invalid picture area height in the second direction; calculating a first ratio of the picture frame width to the preprocessing width, and calculating a second ratio of the picture frame height to the preprocessing height; When the first ratio is equal to the second ratio, the first ratio is set as the scaling ratio of the projection screen in the first direction and the second direction, or the second ratio is set as the scaling ratio of the projection screen in the first direction and the second direction to generate the target display parameters.

8. The display device according to claim 7, wherein: The controller is further configured to: When the first ratio is not equal to the second ratio, obtaining a minimum value of the first ratio and the second ratio; The minimum value is set as the scaling ratio of the projection screen in the first direction, and the minimum value is set as the scaling ratio of the projection screen in the second direction to generate the target display parameters.

9. The display device according to claim 7, wherein: The controller is further configured to: If the invalid picture area is not centrally symmetrical about the reference center point, calculating a pre-processed width of the picture frame in the first direction and a pre-processed height of the picture frame in the second direction; The pre-processing width is the difference between the picture frame width and the invalid picture area width in the first direction, and the pre-processing height is the difference between the picture frame height and the invalid picture area height in the second direction; Calculating a target center point of the picture frame under the preprocessing width and the preprocessing height; Calculating a center offset according to the target center point and the reference center point; Setting a position parameter of the projection screen based on the center offset, wherein the position parameter is used to set a display position of the projection screen; calculating a first ratio of the picture frame width to the preprocessing width, and calculating a second ratio of the picture frame height to the preprocessing height; Obtaining a minimum value of the first ratio and the second ratio; The minimum value is set as the scaling ratio of the projection screen in the first direction, and the minimum value is set as the scaling ratio of the projection screen in the second direction to generate the target display parameters.

10. A screen display method based on projection data, characterized in that: The method is applied to a display device, wherein the display device includes a display, a communicator, and a controller. The display is configured to display a user interface, wherein the user interface includes a projection screen of a terminal device. The communicator is configured to establish a communication connection with the terminal device and receive projection data sent by the terminal device. The projection data includes a picture frame for forming the projection screen, and the picture frame includes a valid picture area and an invalid picture area. The method includes: Detecting characteristic information of the projection data within a preset time period; If the characteristic information changes, maintaining the display parameters of the projection screen; If the characteristic information has not changed, the target display parameters of the projected screen are calculated based on the invalid screen area of ​​the current screen frame, and the display is controlled to display the projected screen according to the target display parameters; the target display parameters are used to set the center point of the valid screen area in the current screen frame to the reference center point of the user interface, and to maximize the display proportion of the valid screen area in the user interface.

Images