A display device and a method for displaying screen mirroring data

Abstract

Some embodiments of the present application disclose a display device and a method for displaying screen mirroring data. The method decodes the first image frame in the second screen mirroring data obtained by demultiplexing to obtain a second image frame. If the second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, the second image frame is discarded; it is detected whether the undiscarded second image frame is a black screen image; if it is not a black screen image, the undiscarded second image frame is rendered to obtain a rendered second image frame; the rendered second image frame is played by a player, and the display is controlled to stop displaying a prompt message. This can ensure that when there are error frames such as black screen images, colored screen or green screen, the second image frame is not rendered, and the prompt message is always displayed on the display device, improving the user's viewing experience.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular, to a display device and a method for displaying screen mirroring data. Background Art

[0002] With the rapid development of display devices, the functions that display devices can provide for users are becoming increasingly rich. Currently, display devices include televisions, set-top boxes, and products with display screens, etc. Taking a television as an example, the scenarios of a television are increasing. It is not only used as a device for watching TV programs at home, but also can be used for screen mirroring display, etc.

[0003] When the television and the mobile phone are on the same network, the user can control the mobile phone to transmit the data displayed on the interface of the mobile phone to the television and use the television to display the data, that is, screen mirroring display. However, it is often found that problems such as a scrambled screen or a green screen occur when the television plays the data, seriously affecting the user's viewing experience.

[0004] Therefore, how to avoid the problems of a scrambled screen or a green screen in the television in the screen mirroring scenario and improve the user's viewing experience has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] Some embodiments of the present application provide a display device and a method for displaying screen mirroring data. The method decodes the first image frame in the second screen mirroring data obtained by demultiplexing to obtain a second image frame. If the second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, that is, the second image frame is an error frame such as a green screen or a scrambled screen, the second image frame is discarded. When the second image frame is discarded, the second image frame obtained by decoding other first image frames in the picture group where the target image frame is located is a black screen image; it is detected whether the undiscarded second image frame is a black screen image; if it is not a black screen image, the undiscarded second image frame is rendered to obtain a rendered second image frame; the rendered second image frame is played by the player, and the display is controlled to stop displaying a prompt message. In this way, it can be ensured that when an error frame such as a black screen image, a scrambled screen or a green screen appears, the second image frame is not rendered, and a prompt message is always displayed on the display device, improving the user's viewing experience.

[0006] In a first aspect, some embodiments of the present application provide a display device, characterized by including:

[0007] A display;

[0008] A controller, configured to:

[0009] When receiving the first screen mirroring data sent by the terminal device, control the display to show a prompt message and create a player; wherein, the prompt message is used to prompt the user that screen mirroring is in progress currently;

[0010] Demultiplex the first screen mirroring data to obtain second screen mirroring data, where the second screen mirroring data includes first image frames arranged in sequence;

[0011] Decode the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames;

[0012] If the second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, discard the second image frame, where the target image frame is the first image frame at the beginning of the second screen mirroring data; when discarding the second image frame, the second image frames decoded from other first image frames in the picture group where the target image frame is located are black screen images;

[0013] Detect whether the undiscarded second image frame is a black screen image; if the undiscarded second image frame is not a black screen image, render the undiscarded second image frame to obtain a rendered second image frame; play the rendered second image frame using the player and control the display to stop showing the prompt message.

[0014] In some embodiments, the controller is further configured to:

[0015] If the second image frame is not decoded from the target image frame, and / or the pixel points of the second image frame do not conform to the preset rule, do not discard the second image frame.

[0016] In some embodiments, the controller is further configured to:

[0017] If it is a black screen image, do not render the undiscarded second image frame.

[0018] In some embodiments, when the controller executes decoding the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames, it is further configured to:

[0019] Arrange the first image frames in the second screen mirroring data in sequence in a decoding queue;

[0020] Decode the first image frames in the decoding queue in sequence to obtain corresponding second image frames.

[0021] In some embodiments, the second screen mirroring data further includes audio data; the controller is further configured to:

[0022] Synchronize the undiscarded second image frames with the audio data in the second screen mirroring data, and render the synchronized undiscarded second image frames and audio data to obtain synchronized and rendered second image frames and audio data;

[0023] Use the player to play the synchronized and rendered second image frames and audio data, and control the display to stop displaying the prompt message.

[0024] In some embodiments, the controller includes an application layer, a Java interface layer, a native implementation layer, a framework layer, and a hardware implementation layer; the controller is configured to:

[0025] When the application layer receives the first screen mirroring data sent by the terminal device, it controls the display to display a prompt message and sends a first notification to the Java interface layer; wherein, the prompt message is used to prompt the user that screen mirroring is in progress;

[0026] When the Java interface layer receives the first notification, it creates a player so that the player sends the first screen mirroring data to the native implementation layer;

[0027] The native implementation layer demultiplexes the first screen mirroring data to obtain second screen mirroring data, and sends the first image frames in the second screen mirroring data to the framework layer in sequence;

[0028] The framework layer decodes the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames; if the second image frame is decoded from the target image frame and the pixel points of the second image frame conform to a preset rule, then discard the second image frame, where the target image frame is the first image frame in the second screen mirroring data; when discarding the second image frame, the second image frames decoded from other first image frames in the picture group where the target image frame is located are black screen images; send the second image frames to the native implementation layer;

[0029] The native implementation layer detects whether the undiscarded second image frame is a black screen image; if it is not a black screen image, it renders the undiscarded second image frame to obtain a rendered second image frame, so that the player plays the rendered second image frame; after rendering, send a second notification to the application layer so that the application layer stops displaying the prompt message.

[0030] In some embodiments, the controller is further configured to:

[0031] If the framework layer detects that the second image frame is not decoded from the target image frame, and / or the pixel points of the second image frame do not conform to the preset rules, the second image frame is not discarded.

[0032] In some embodiments, the controller is further configured to:

[0033] If the native implementation layer detects that the undiscarded second image frame is a black screen image, the undiscarded second image frame is not rendered.

[0034] In some embodiments, before the native implementation layer renders the second image frame, the controller is further configured to:

[0035] The native implementation layer synchronizes the undiscarded second image frame with the audio data in the second screen mirroring data, and renders the synchronized undiscarded second image frame and audio data, so that the player plays the synchronized and rendered second image frame and audio data; after rendering, a second notification is sent to the application layer to cause the application layer to stop displaying the prompt message.

[0036] In a second aspect, a method for displaying screen mirroring data is provided, including:

[0037] When receiving the first screen mirroring data sent by the terminal device, controlling the display to display a prompt message and creating a player; wherein, the prompt message is used to prompt the user that screen mirroring is currently in progress;

[0038] Demultiplexing the first screen mirroring data to obtain second screen mirroring data, where the second screen mirroring data includes first image frames arranged in sequence;

[0039] Decoding the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames;

[0040] If the second image frame is decoded from the target image frame and the pixel points of the second image frame conform to the preset rules, the second image frame is discarded, where the target image frame is the first first image frame in the second screen mirroring data; when discarding the second image frame, the second image frame decoded from other first image frames in the picture group where the target image frame is located is a black screen image;

[0041] Detecting whether the undiscarded second image frame is a black screen image; if the undiscarded second image frame is not a black screen image, rendering the undiscarded second image frame to obtain a rendered second image frame; playing the rendered second image frame using the player, and controlling the display to stop displaying the prompt message.

[0042] Some embodiments of the present application provide a display device and a method for displaying screen mirroring data. The method decodes the first image frame in the second screen mirroring data obtained by demultiplexing to obtain a second image frame. If the second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, that is, the second image frame is an error frame such as a green screen or a mosaic screen, the second image frame is discarded. When the second image frame is discarded, the second image frame obtained by decoding other first image frames in the picture group where the target image frame is located is a black screen image; it is detected whether the undiscarded second image frame is a black screen image; if it is not a black screen image, the undiscarded second image frame is rendered to obtain a rendered second image frame; the rendered second image frame is played by the player, and the display is controlled to stop displaying the prompt message. This can ensure that when there are error frames such as black screen images, mosaic screens or green screens, the second image frame is not rendered, and the prompt message is always displayed on the display device, improving the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Shows an operation scenario between a display device and a control device according to some embodiments;

[0044] Figure 2 Shows a hardware configuration block diagram of a control device according to some embodiments;

[0045] Figure 3 Shows a hardware configuration block diagram of a display device according to some embodiments;

[0046] Figure 4 Exemplarily shows a flowchart of a method for displaying screen mirroring data according to some embodiments;

[0047] Figure 5 Exemplarily shows a schematic diagram of an interface of a terminal device according to some embodiments;

[0048] Figure 6 Exemplarily shows a schematic diagram of an interface of another terminal device according to some embodiments;

[0049] Figure 7 Exemplarily shows a schematic diagram of a user interface of a display device according to some embodiments;

[0050] Figure 8 Exemplarily shows a schematic diagram of an interface of another terminal device according to some embodiments;

[0051] Figure 9 Exemplarily shows a schematic diagram of another user interface of a display device according to some embodiments;

[0052] Figure 10Exemplarily shown is a software configuration diagram in a display device provided according to some embodiments;

[0053] Figure 11 Exemplarily, another method for displaying screen mirroring data provided according to some embodiments. Detailed implementation manners

[0054] To make the objectives and implementation manners of the present application clearer, the following will clearly and completely describe the exemplary implementation manners of the present application with reference to the accompanying drawings in the exemplary embodiments of the present application. Apparently, the described exemplary embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0055] It should be noted that the brief description of the terms in the present application is only for facilitating the understanding of the subsequent described implementation manners, rather than intending to limit the implementation manners of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0056] The terms "first", "second", "third", etc. in the specification, claims and the above-mentioned drawings of the present application are used to distinguish similar or like objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0057] The term "comprising" and any variations thereof are intended to cover but not be exclusive of inclusion. For example, a product or device comprising a series of components does not necessarily have to be limited to all the clearly listed components, but may include other components not clearly listed or inherent to these products or devices.

[0058] The display device provided by the implementation manner of the present application can have various implementation forms. For example, it can be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is a specific implementation manner of the display device of the present application.

[0059] Figure 1 This is a schematic diagram of the operation scenario between the display device and the control device according to the embodiment. As Figure 1 shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0060] In some embodiments, the control device 100 may be a remote controller. The communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short - range communication methods, and controls the display device 200 wirelessly or wiredly. The user can input user instructions through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200.

[0061] In some embodiments, an intelligent device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop, etc.) may also be used to control the display device 200. For example, an application program running on the intelligent device is used to control the display device 200.

[0062] In some embodiments, the display device may receive user control not by using the above - mentioned intelligent device or control device, but by touch, gesture, etc.

[0063] In some embodiments, the display device 200 may also be controlled in a manner other than the control device 100 and the intelligent device 300. For example, it can directly receive user voice instructions for control through a module for obtaining voice instructions configured inside the display device 200, or receive user voice instructions for control through a voice control device set outside the display device 200.

[0064] In some embodiments, the display device 200 also conducts data communication with the server 400. The display device 200 is allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 can provide various contents and interactions to the display device 200. The server 400 can be a cluster or multiple clusters, and can include one type or multiple types of servers.

[0065] Figure 2 An exemplary configuration block diagram of the control device 100 according to an exemplary embodiment is shown. As Figure 2 shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from the user, and convert the operation instructions into instructions recognizable and responsive by the display device 200, acting as an interaction intermediary between the user and the display device 200.

[0066] As Figure 3 , the display device 200 includes at least one of a tuner - demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0067] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output.

[0068] The display 260 includes a display screen component for presenting a picture, and a driving component for driving image display, a component for receiving an image signal from the output of the controller, and for displaying video content, image content, a menu control interface, and a user control UI interface.

[0069] The display 260 can be a liquid crystal display, an OLED display, and a projection display, and can also be a projection device and a projection screen.

[0070] The display 260 further includes a touch screen for receiving control instructions input by actions such as a user's finger sliding or clicking on the touch screen.

[0071] The communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module and other network communication protocol chips or a near field communication protocol chip, and an infrared receiver. The display device 200 can establish the sending and receiving of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0072] The user interface can be used to receive control signals of the control device 100 (such as an infrared remote controller, etc.).

[0073] The detector 230 is used to collect signals of the external environment or for external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environmental scenes, user attributes or user interaction gestures, or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0074] The external device interface 240 can include but is not limited to the following: a high-definition multimedia interface (HDMI), an analog or data high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc., any one or more interfaces. It can also be a composite input / output interface formed by the above multiple interfaces.

[0075] The tuner demodulator 210 receives broadcast television signals by wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as EPG data signals.

[0076] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices. That is, the tuner-demodulator 210 may also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0077] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0078] In some embodiments, the controller includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), a Random Access Memory (RAM), a Read-Only Memory (ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0079] The user can input a user command on the graphical user interface (GUI) that can be displayed on the display 260, and then the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and then the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0080] The "user interface" is a media interface for interaction and information exchange between an application program or an operating system and the user, and it realizes the conversion between the internal form of information and the form acceptable to the user. The common manifestation form of the user interface is the graphical user interface (GUI), which refers to the user interface related to computer operations displayed in a graphical manner. It can be an interface element such as an icon, a window, a control, etc. displayed on the display screen of an electronic device, where the control can include visible interface elements such as an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc.

[0081] When the TV and the mobile phone are on the same network, the user can control the mobile phone to transmit the data displayed on the mobile phone interface to the TV, and use the TV to display the data, that is, screen projection. However, it is often found that when the TV plays the data, problems such as distorted screen or green screen will appear, which seriously affects the user's viewing experience. Therefore, how to avoid the problem of distorted screen or green screen on the TV in the screen projection scenario and improve the user's viewing experience has become a problem that technicians in this field need to solve urgently.

[0082] In order to solve the above technical problems, the embodiments of the present application provide a display device and a method for displaying projection data.

[0083] The method decodes the first image frame in the second projection screen data obtained by demultiplexing to obtain a second image frame. If the second image frame is obtained by decoding the target image frame, and the pixel points of the second image frame meet the preset rules, that is, the second image frame is an error frame such as a green screen or a flowery screen, then the second image frame is discarded. When the second image frame is discarded, the second image frame obtained by decoding other first image frames in the picture group where the target image frame is located is a black screen image; and detect whether the second image frame that has not been discarded is a black screen image; if it is not a black screen image, the second image frame that has not been discarded is rendered to obtain a rendered second image frame; the player is used to play the rendered second image frame, and the display is controlled to stop displaying a prompt message. In this way, it can be ensured that when an error frame such as a black screen image, a flowery screen or a green screen appears, the second image frame is not rendered, and the prompt message is always displayed on the display device to improve the user's viewing experience.

[0084] Figure 4 The flowchart of a method for displaying screen data according to some embodiments is exemplarily shown. The steps include S100-S1100.

[0085] S100. When the first screen projection data sent by the terminal device is received, the display is controlled to display a prompt message and a player is created; wherein the prompt message is used to prompt the user that the screen is currently being projected.

[0086] In some embodiments, the terminal device may be a mobile phone, a tablet computer, or the like that can establish a screen projection connection with a display device and send first screen projection data.

[0087] For example, the terminal device may be a touch screen mobile phone. The user may call up the screen projection control on the terminal device. For example, Figure 5 The following is a schematic diagram of an interface of a terminal device provided according to some embodiments. Figure 5 The screen projection control 501 is displayed in the middle. The user can touch the screen projection control 501, and the screen projection device that can be selected is displayed on the interface.

[0088] For example,Figure 6 Exemplarily shown is a schematic diagram of the interface of another terminal device provided according to some embodiments. In Figure 6 device controls corresponding to the screen mirroring devices that can be selected are displayed. Specifically, it includes a device control 601 corresponding to the screen mirroring device A, a device control 602 corresponding to the screen mirroring device B, and a device control 603 corresponding to the screen mirroring device C. The user can touch the device control of the device on which the display content on the interface of the terminal device is desired to be displayed. When the user touches a certain device control, the first screen mirroring data including the interface of the terminal device is sent to the screen mirroring device corresponding to the device control, and this screen mirroring device can be a display device.

[0089] In some embodiments, when the terminal device sends the first screen mirroring data to the display device, protocols such as mircast screen mirroring, airplay screen mirroring, or rtp (Real-time Transport Protocol) + udp (User Datagram Protocol) screen mirroring can be used.

[0090] In some embodiments, the prompt message can be content in the form of pictures, texts, animations, or videos, etc. This prompt message can be provided by a preset application. Exemplarily, Figure 7 Exemplarily shown is a schematic diagram of the user interface of a display device provided according to some embodiments. In Figure 7 a prompt message provided by the preset application is displayed, and this prompt message is "Screen mirroring in progress".

[0091] S200. Demultiplex (Demux) the first screen mirroring data to obtain second screen mirroring data, and the second screen mirroring data includes first image frames arranged in sequence.

[0092] Demultiplexing means parsing the encapsulation format of the first screen mirroring data to obtain the second screen mirroring data containing the original audio and video code streams.

[0093] In some embodiments, the order of the first image frames in the second screen mirroring data is determined by the frame number. Exemplarily, the frame numbers of the first image frames in the second screen mirroring data are respectively 1 - 10, there are 10 first image frames in the second screen mirroring data, and the first image frames are arranged in ascending order of the frame number. The arranged first image frames are in the same order as the actual playback order of the interface content of the terminal device.

[0094] S300. Decode the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames.

[0095] In the embodiments of the present application, the number of first image frames in the second screen mirroring data may be multiple. The display device decodes the first image frames in the second screen mirroring data one by one in sequence to obtain second image frames corresponding to the first image frames.

[0096] In some embodiments, the step of decoding the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames includes: arranging the first image frames in the second screen mirroring data in sequence in a queue to be decoded; decoding the first image frames in the queue to be decoded in sequence to obtain corresponding second image frames.

[0097] In the embodiments of the present application, the first image frames are put into the queue to be decoded in sequence. When decoding, the first image frames are read one by one in sequence from the queue to be decoded, and the read first image frames are decoded to obtain second image frames.

[0098] Exemplarily, the first image frames A - D are put into the queue to be decoded one by one. When decoding, the first image frame A is first read from the queue to be decoded, and then the first image frame A is decoded to obtain the second image frame A. Then the first image frame B is read, and then the first image frame B is decoded to obtain the second image frame B, and so on. All the first image frames in the queue to be decoded are read and decoded to obtain corresponding second image frames. Additionally, the next first image frame can also be read during the decoding process of a first image frame. Since the data volume differences between different first image frames are not significant, reading the next first image frame during the decoding process of a first image frame will not cause the next first image frame to be decoded to obtain the corresponding second image frame first, while the previous first image frame is decoded to obtain the corresponding second image frame later.

[0099] S400. Detect whether the second image frame is decoded from a target image frame, where the target image frame is the first first image frame in the second screen mirroring data.

[0100] In the embodiments of the present application, since after the display device establishes a screen mirroring connection, users are more sensitive to the video content of the first frame when starting to play. If there are error frames such as a garbled screen or a green screen, the impact on the user's viewing experience is relatively large. Therefore, in the embodiments of the present application, it is detected whether the second image frame is decoded from the target image frame in the second screen mirroring data.

[0101] In some embodiments, the order of the first image frames in the second screen mirroring data is determined by the frame sequence number, and the first image frames are arranged in ascending order of the frame sequence number, with the frame sequence number starting from 1. Therefore, the first first image frame can be the first image frame with the frame sequence number 1, that is, the target image frame is the first image frame with the frame sequence number 1. When it is detected that the second image frame is decoded from the first image frame with the frame sequence number 1, it is determined that the second image frame is decoded from the target image frame.

[0102] In other embodiments, since the decoding of the first image frames in the second screen mirroring data is performed in sequence, the first second image frame obtained is decoded from the first first image frame in the second screen mirroring data, that is, decoded from the target image frame. Therefore, when the first decoded second image frame is decoded from the target image frame.

[0103] S500. If the second image frame is decoded from the target image frame, then detect whether the pixel points of the second image frame conform to a preset rule.

[0104] In the embodiments of the present application, the preset rule refers to the rule that the pixel points of error frames such as green screen or mosaic screen have. In the embodiments of the present application, the specific judgment method for the green screen or mosaic screen is not limited, so any method that can detect whether the pixel points of the second image frame conform to the preset rule can be accepted. Exemplarily, first filter the second image frame with a Laplacian operator, and then obtain the variance of the resulting image. If the variance is less than a certain value, the second image frame is regarded as a mosaic screen.

[0105] S600. If the pixel points of the second image frame conform to the preset rule, then discard the second image frame. When discarding the second image frame, the second image frames decoded from other first image frames in the group of pictures (GOP) where the target image frame is located are black screen images.

[0106] In the embodiments of the present application, if the pixel points of the second image frame conform to the preset rule, that is, the second image frame is an error frame such as a green screen or a mosaic screen, then the second image frame is discarded, and the second image frame is not rendered and displayed. In the embodiments of the present application, in order to improve the user experience, the error frame is directly discarded, so that the user will not see problem images such as a mosaic screen or a green screen at the beginning stage of screen mirroring.

[0107] In addition, for the first image frames belonging to the same group of pictures as the target image frame, since decoding depends on the second image frame corresponding to the target image frame during decoding, if the second image frame corresponding to the target image frame is discarded, then the other first image frames in the same group of pictures (GOP) will become black screen images after decoding and normal image frames cannot be obtained.

[0108] S700. If the pixel points of the second image frame do not conform to the preset rules, the second image frame is not discarded.

[0109] In the embodiments of the present application, if the pixel points of the second image frame do not conform to the preset rules, the second image frame is not a wrong frame such as a flower screen or a green screen. At this time, the second image frame can be retained. If it is determined in subsequent steps that the second image frame is not a black screen image either, the second image frame can be rendered and played.

[0110] S800. If the second image frame is not decoded from the target image frame, the second image frame is not discarded.

[0111] In the embodiments of the present application, if the second image frame is not decoded from the target image frame, that is, the first first image frame, the second image frame is retained. If it is determined in subsequent steps that the second image frame is not a black screen image either, the second image frame can be rendered and played.

[0112] In the embodiments of the present application, discarding the second image frame corresponding to the target image frame will cause the second image frames corresponding to other first image frames in the same picture group as the target image frame to be black screen images. To avoid the black screen image being displayed on the display device, in the embodiments of the present application, S900. Detect whether the undiscarded second image frame is a black screen image.

[0113] In the embodiments of the present application, black screen image detection is performed on all undiscarded second image frames. It should be noted that in the embodiments of the present application, the discarded second image frames are only the second image frames decoded from the target image frame and whose pixel points conform to the preset rules, and the rest of the second image frames are not discarded. Detect whether the undiscarded second image frame is a black screen image.

[0114] S1000. If the undiscarded second image frame is a black screen image, the undiscarded second image frame is not rendered.

[0115] In the embodiments of the present application, if it is a black screen image, the black screen graphic is not rendered, and the prompt message is continuously displayed. In this way, the user will not view the image rendered from the black screen image on the display device, and the user can see the intact interface content of the terminal device, that is, before there is no flower screen, green screen or black image, the prompt message can always be seen on the display device, improving the user experience.

[0116] S1100. If it is not a black screen image, the undiscarded second image frame is rendered to obtain a rendered second image frame; the rendered second image frame is played by the player, and the display is controlled to stop displaying the prompt message. Exemplarily, Figure 8An exemplary schematic diagram of the interface of another terminal device provided according to a writing embodiment is shown. Figure 9 An exemplary schematic diagram of the user interface of another display device provided according to some embodiments is shown. In Figure 9 it shows the same content as the terminal device interface in Figure 8 to achieve screen mirroring between the display device and the terminal device.

[0117] In the embodiments of the present application, when the undropped second image frame is not a mosaic screen, a green screen, or a black screen image, it will be rendered and played, and the display is controlled to stop displaying the prompt message. In this way, it is ensured that the interface content of the terminal device played on the display device after screen mirroring connection seen by the user is normal, without mosaic screens, green screens, or black screen images.

[0118] In some embodiments, the second screen mirroring data further includes audio data; the method further includes:

[0119] Synchronize the undropped second image frame with the audio data in the second screen mirroring data, and render the synchronized undropped second image frame and audio data to obtain synchronized rendered second image frame and audio data; play the synchronized rendered second image frame and audio data using the player, and control the display to stop displaying the prompt message.

[0120] In the embodiments of the present application, the rendered undropped second image frame and audio data are synchronized. In this way, the user can view the synchronized playback of the rendered second image frame and audio data on the display device, improving the user experience.

[0121] In the embodiments of the present application, the method decodes the first image frame in the second screen mirroring data obtained by demultiplexing to obtain a second image frame. If the second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, that is, the second image frame is an error frame such as a green screen or a mosaic screen, then the second image frame is discarded; if it is not a black screen image, then the undropped second image frame is rendered to obtain a rendered second image frame; the rendered second image frame is played using the player, and the display is controlled to stop displaying the prompt message. In this way, it can be ensured that when there are error frames such as black screen images, mosaic screens, or green screens, the second image frame is not rendered, and the prompt message is always displayed on the display device, improving the user's viewing experience.

[0122] In some embodiments, taking the case where the display device and the terminal device transmit the first screen mirroring data using the mircast screen mirroring protocol as an example to introduce the screen mirroring data display method.

[0123] The mircast screen mirroring protocol is a kind of encapsulated rtsp protocol. Figure 10An exemplary software configuration diagram of a display device provided according to some embodiments is shown. In some embodiments, the controller includes an application layer, a Java interface layer, a native implementation layer, a framework layer, and a hardware implementation layer.

[0124] Based on Figure 10 the structure in the software configuration diagram, an embodiment of the present application discloses a method for displaying screen mirroring data. Figure 11 Exemplarily, in addition to another method for displaying screen mirroring data provided according to some embodiments, the method includes S1101 - S1112:

[0125] S1101. When the application layer receives the first screen mirroring data sent by the terminal device, it controls the display to display a prompt message, and S1102. sends a first notification to the Java interface layer; wherein, the prompt message is used to prompt the user that screen mirroring is in progress; the first notification instructs the Java interface layer to create a player.

[0126] In the embodiments of the present application, the system - built - in application programs and non - system - level application programs all belong to the application layer. The application layer is responsible for direct interaction with the user. The application layer may include multiple application programs, such as a settings application program, a media center application program, etc.

[0127] S1103. When the Java interface layer receives the first notification, it creates a player. S1104. The player sends the first screen mirroring data to the native implementation layer.

[0128] In some embodiments, after the Java interface layer creates the player, it also performs Android native operations such as setting a window (surface), setting a path (setMediaDataSource), and preparation (prepare).

[0129] In some embodiments, before the player sends the first screen mirroring data to the native implementation layer, the method further includes: when the application layer obtains the first screen mirroring data, it stores the first screen mirroring data in a cache module; the player in the Java interface layer reads the first screen mirroring data from the cache module.

[0130] S1105. The native implementation layer demultiplexes the first screen mirroring data to obtain second screen mirroring data. S1106. The native implementation layer sends the first image frame in the second screen mirroring data to the framework layer in sequence.

[0131] In some embodiments, before the native implementation layer sends the first image frame in the second screen mirroring data to the framework layer in sequence, the first image frame in the second screen mirroring data is also arranged in the decoding queue in sequence, and the first image frame in the decoding queue is sent to the framework layer in sequence.

[0132] S1107. The framework layer decodes the first image frame in the second screen mirroring data in sequence to obtain the corresponding second image frame. It is detected whether the second image frame is decoded from the target image frame, where the target image frame is the first image frame in the second screen mirroring data. If the second image frame is decoded from the target image frame, it is detected whether the pixel points of the second image frame conform to a preset rule. S1108. If the pixel points of the second image frame conform to the preset rule and the second image frame is decoded from the target image frame, the second image frame is discarded. When the second image frame is discarded, the second image frames decoded from other first image frames in the group of pictures (GOP) where the target image frame is located are black screen images.

[0133] In some embodiments, if the framework layer detects that the second image frame is not decoded from the target image frame, and / or the pixel points of the second image frame do not conform to the preset rule, the second image frame is not discarded.

[0134] S1109. The framework layer sends the undiscarded second image frame to the native implementation layer.

[0135] S1110. The native implementation layer detects whether the undiscarded second image frame is a black screen image; S1111. If it is not a black screen image, the undiscarded second image frame is rendered to obtain the rendered second image frame, so that the player plays the rendered second image frame. S1112. After rendering, a second notification is sent to the application layer so that the application layer stops displaying the prompt message.

[0136] In some embodiments, if it is a black screen image, the undiscarded second image frame is not rendered.

[0137] In some embodiments, before the native implementation layer renders the second image frame, the method further includes:

[0138] The native implementation layer synchronizes the undropped second image frames with the audio data in the second screen mirroring data, and renders the synchronized undropped second image frames and audio data, so that the player plays the synchronized and rendered second image frames and audio data; after rendering, a second notification is sent to the application layer to cause the application layer to stop displaying the prompt message.

[0139] In some embodiments, since the decoder cannot detect and drop frames for the second image frames according to the set frame dropping attribute, the native implementation layer is used to perform the steps of detecting and dropping frames for the second image frames.

[0140] In some embodiments, the display device and the terminal device control and transmit the first screen mirroring data through the RTSP protocol. The framework layer includes a decoder.

[0141] In some embodiments, when the display device receives the first screen mirroring data, a player is created. When the player is configured, a frame dropping attribute is set, and this set frame dropping attribute is to instruct the decoder to set the frame dropping attribute. The decoder determines whether to drop the second image frames according to the frame dropping attribute. At the same time when the display device receives the first screen mirroring data, the display device also parses the first screen mirroring data and stores it in the buffer module. The player side reads the first screen mirroring data from the buffer module, demultiplexes (demux) it to obtain the second screen mirroring data (ES data), and sends the second screen mirroring data to the decoder. When decoding, the decoder detects whether the second screen mirroring data is decoded from the target image frame and whether the pixel points conform to the preset rules. If it is found that the second screen mirroring data is decoded from the target image frame and the pixel points conform to the preset rules, then the second screen mirroring data is discarded.

[0142] An embodiment of the present application further provides a display device, including: a display and a controller;

[0143] A controller is configured to: when receiving first screen mirroring data sent by a terminal device, control a display to display a prompt message and create a player; wherein, the prompt message is used to prompt the user that screen mirroring is in progress; demultiplex the first screen mirroring data to obtain second screen mirroring data, where the second screen mirroring data includes first image frames arranged in sequence; decode the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames; if a second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, then discard the second image frame, where the target image frame is the first image frame at the beginning of the second screen mirroring data; when discarding the second image frame, the second image frames decoded from other first image frames in the picture group where the target image frame is located are black screen images; detect whether the undiscarded second image frames are black screen images; if the undiscarded second image frames are not black screen images, then render the undiscarded second image frames to obtain rendered second image frames; play the rendered second image frames using the player, and control the display to stop displaying the prompt message.

[0144] In the above embodiment, a display device and a method for displaying screen mirroring data are provided. The method decodes the first image frames in the second screen mirroring data obtained by demultiplexing to obtain second image frames. If a second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, that is, the second image frame is an error frame such as a green screen or a mosaic screen, then discard the second image frame. When discarding the second image frame, the second image frames decoded from other first image frames in the picture group where the target image frame is located are black screen images; detect whether the undiscarded second image frames are black screen images; if they are not black screen images, then render the undiscarded second image frames to obtain rendered second image frames; play the rendered second image frames using the player, and control the display to stop displaying the prompt message. This can ensure that when error frames such as black screen images, mosaic screens or green screens occur, the second image frames are not rendered, and the prompt message is always displayed on the display device, improving the user's viewing experience.

[0145] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0146] For the sake of convenience in explanation, the above description has been made in connection with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, Including: A display; A controller configured to: When receiving the first screen mirroring data sent by a terminal device, control the display to display a prompt message and create a player; wherein, the prompt message is used to prompt the user that screen mirroring is in progress; Demultiplex the first screen mirroring data to obtain second screen mirroring data, where the second screen mirroring data includes first image frames arranged in sequence; Decode the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames; If the second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, discard the second image frame, where the target image frame is the first image frame in the second screen mirroring data, and the preset rule is the rule that the pixel points of an error frame have, and the error frame includes at least a garbled screen or a green screen; when discarding the second image frame, the second image frame obtained by decoding other first image frames in the picture group where the target image frame is located is a black screen image; Detect whether the undiscarded second image frame is a black screen image; if the undiscarded second image frame is not a black screen image, render the undiscarded second image frame to obtain a rendered second image frame; play the rendered second image frame using the player and control the display to stop displaying the prompt message.

2. The display device according to claim 1, wherein The controller is further configured to: If the second image frame is not decoded from the target image frame, and / or the pixel points of the second image frame do not conform to the preset rule, do not discard the second image frame.

3. The display device according to claim 1, wherein The controller is further configured to: If it is a black screen image, do not render the undiscarded second image frame.

4. The display device according to claim 1, wherein When the controller executes decoding the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames, it is further configured to: Arrange the first image frames in the second screen mirroring data in sequence in a decoding queue; Decode the first image frames in the decoding queue in sequence to obtain corresponding second image frames.

5. The display device according to claim 1, characterized in that, The second screen mirroring data further includes audio data; the controller is further configured to: Synchronize the undiscarded second image frame with the audio data in the second screen mirroring data, and render the synchronized undiscarded second image frame and audio data to obtain synchronized rendered second image frame and audio data; Play the synchronized rendered second image frame and audio data using the player and control the display to stop displaying the prompt message.

6. The display device according to claim 1, wherein The controller includes an application layer, a Java interface layer, a native implementation layer, a framework layer, and a hardware implementation layer; the controller is configured to: When the application layer receives the first screen mirroring data sent by the terminal device, control the display to display a prompt message and send a first notification to the Java interface layer; wherein, the prompt message is used to prompt the user that screen mirroring is in progress; When the Java interface layer receives the first notification, create a player so that the player sends the first screen mirroring data to the native implementation layer; The native implementation layer demultiplexes the first screen mirroring data to obtain second screen mirroring data, and sends the first image frames in the second screen mirroring data to the framework layer in sequence; The framework layer decodes the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames; if the second image frame is decoded from a target image frame and the pixel points of the second image frame conform to a preset rule, the second image frame is discarded, where the target image frame is the first image frame in the second screen mirroring data; when discarding the second image frame, the second image frame obtained by decoding other first image frames in the picture group where the target image frame is located is a black screen image; the second image frame is sent to the native implementation layer; The native implementation layer detects whether the undiscarded second image frame is a black screen image; if it is not a black screen image, the undiscarded second image frame is rendered to obtain a rendered second image frame, so that the player plays the rendered second image frame; after rendering, a second notification is sent to the application layer so that the application layer stops displaying the prompt message.

7. The display device according to claim 6, characterized in that, The controller is further configured to: If the framework layer detects that the second image frame is not decoded from the target image frame, and / or the pixel points of the second image frame do not conform to the preset rule, the second image frame is not discarded.

8. The display device according to claim 6, characterized in that, The controller is further configured to: If the native implementation layer detects that the undiscarded second image frame is a black screen image, the undiscarded second image frame is not rendered.

9. The display device according to claim 6, wherein Before the native implementation layer renders the second image frame, the controller is further configured to: The native implementation layer synchronizes the undiscarded second image frame with the audio data in the second screen mirroring data, and renders the synchronized undiscarded second image frame and audio data, so that the player plays the synchronized rendered second image frame and audio data; After rendering, a second notification is sent to the application layer so that the application layer stops displaying the prompt message.

10. A method for displaying screen mirroring data, characterized in that, including: When receiving the first screen mirroring data sent by the terminal device, control the display to display a prompt message and create a player; wherein, the prompt message is used to prompt the user that screen mirroring is in progress; Demultiplex the first screen mirroring data to obtain second screen mirroring data, and the second screen mirroring data includes first image frames arranged in sequence; Decode the first image frames in the second screen mirroring data in sequence to obtain corresponding second image frames; If the second image frame is decoded from the target image frame and the pixel points of the second image frame conform to a preset rule, the second image frame is discarded, where the target image frame is the first first image frame in the second screen projection data, and the preset rule is the rule that the pixel points of an error frame have, and the error frame includes at least a scrambled screen or a green screen; when the second image frame is discarded, the second image frame obtained by decoding other first image frames in the picture group where the target image frame is located is a black screen image; Detect whether the undiscarded second image frame is a black screen image; if the undiscarded second image frame is not a black screen image, render the undiscarded second image frame to obtain a rendered second image frame; use the player to play the rendered second image frame and control the display to stop displaying the prompt message.

Images