A display method, device, apparatus and storage medium

Abstract

The embodiment of the application discloses a display method and display equipment, relates to the technical field of display, and can solve the green screen problem in the process of screen projection, and improves the use experience of a user. The specific scheme comprises the following steps: accepting video data sent from a screen projection equipment; frame splitting the video data to obtain a plurality of frame data; discarding first frame data in the case that an image corresponding to the first frame data is a full green image; the first frame data is frame data in the plurality of frame data; displaying second frame data, and the second frame data is frame data except the first frame data in the plurality of frame data.

Description

Technical Field

[0001] This application relates to the field of screen projection display, and more particularly to a display method and display device. Background Technology

[0002] Currently, screen mirroring has been widely used in home and business settings, becoming one of the mainstream ways people watch videos. Meanwhile, in the context of the 5G era's interconnectedness of everything and screens, screen mirroring is beginning to carry even more value and possibilities.

[0003] However, due to network fluctuations or other factors, a green screen phenomenon may still occur during screen mirroring. That is, a certain frame corresponds to a completely green image. Since there are no errors in the data stream corresponding to the green screen frame, the green screen frame cannot be discarded by detecting erroneous frames. As a result, the green screen frame is still rendered and played to the screen being mirrored, affecting the user's viewing experience. Summary of the Invention

[0004] This application provides a display method and display device that can detect and discard green screen frames, improve the green screen problem during screen projection, and enhance the user's viewing experience.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, a display device is provided, which may include: a display; a communicator configured to receive video data sent from a projection device; and a processor coupled to the display and the communicator, and configured to: perform frame-segmentation processing on the video data to obtain multiple frame data; discard the first frame data if the image corresponding to the first frame data is a fully green image; the first frame data is frame data among the multiple frame data; and control the display to display a second frame data, the second frame data being frame data other than the first frame data among the multiple frame data.

[0007] In conjunction with the first aspect, in another possible implementation, the processor is configured to discard the first frame data if the image corresponding to the first frame data is an all-green image, including: determining that the image corresponding to the first frame data is an all-green image if the color portion of the first frame data is entirely a preset string; wherein the color portion of the first frame data is used to indicate the color of all pixels of the image corresponding to the first frame data; and discarding the first frame data.

[0008] In conjunction with the first aspect, in another possible implementation, the processor is configured to discard the first frame data when the image corresponding to the first frame data is a completely green image, including: parsing the first frame data to obtain parsed data corresponding to the first frame data when the first frame data contains a preset number of consecutive preset strings; performing green screen detection on the parsed data corresponding to the first frame data; and discarding the parsed data corresponding to the first frame data when the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a completely green image.

[0009] In conjunction with the first aspect, in another possible implementation, the processor is configured to perform green screen detection on the parsed data corresponding to the first frame data, including: when the color encoding method of the parsed data corresponding to the first frame data is YUV, converting the parsed data corresponding to the first frame data into target data; the color encoding method of the target data is RGB; and when the green color value indicated by the target data is within a preset range, determining that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image.

[0010] In conjunction with the first aspect, in another possible implementation, the processor is further configured to: determine, when the green color value indicated by the target data is outside a preset range, that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is not a fully green image; perform audio and video synchronization operation on the parsed data corresponding to the first frame data to obtain the first data to be played; render the first data to be played; and control the display to play the rendered first data to be played.

[0011] In conjunction with the first aspect, in another possible implementation, the processor is also configured to display a prompt message when the green screen detection result of the parsed data corresponding to the first frame of data indicates that the image corresponding to the first frame of data is a fully green image; the prompt message is used to indicate to the user that the display device is in the process of screen projection.

[0012] In conjunction with the first aspect, in another possible implementation, the processor is further configured to: parse the first frame data to obtain parsed data corresponding to the first frame data if the first frame data does not contain a preset number of consecutive preset strings; perform audio and video synchronization operation on the parsed data corresponding to the first frame data to obtain second data to be played; render the second data to be played; and control the display to play the rendered second data to be played.

[0013] Secondly, a display method is provided, applied to a display device. This method may include: receiving video data sent from a projection device; performing frame-segmentation on the video data to obtain multiple frame data; discarding the first frame data if the image corresponding to the first frame data is a completely green image; the first frame data being frame data among the multiple frame data; and displaying a second frame data, which is frame data among the multiple frame data excluding the first frame data.

[0014] Thirdly, a display device is provided that has the function of implementing the method of the second aspect described above. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0015] Fourthly, a display device is provided, comprising a receiving module, a framing module, a frame dropping module, and a display module. The receiving module receives video data sent from a projection device; the framing module performs framing processing on the video data received by the receiving module to obtain multiple frame data; the frame dropping module discards the first frame data if the image corresponding to the first frame data obtained by the framing module is a completely green image; and the display module displays the second frame data obtained by the framing module, wherein the second frame data consists of all frame data except the first frame data.

[0016] Fifthly, a display device is provided, comprising: a processor and a memory; the memory is used to store computer execution instructions, and when the first device is running, the processor executes the computer execution instructions stored in the memory to cause the first device to perform the display method provided in the second aspect above.

[0017] In a sixth aspect, a computer-readable storage medium is provided, which stores instructions that, when executed on a computer, enable the computer to perform the display method provided in the second aspect above.

[0018] In a seventh aspect, a computer program product containing instructions is provided, which, when run on a display device, enables the display device to execute the display method provided in the second aspect above.

[0019] Eighthly, an apparatus (e.g., a chip system) is provided, comprising a processor for supporting a display device in implementing the functions described in the second aspect above. In one possible design, the apparatus further comprises a memory for storing program instructions and data necessary for the display device. When the apparatus is a chip system, it may be composed of chips or may include chips and other discrete devices.

[0020] In summary, because the technical solution provided in this application embodiment can detect the frame data corresponding to the green screen frame during the display process, and prevent the green screen frame from being rendered and played to the projected device (i.e., the display device) by dropping frames, the user experience is improved. Attached Figure Description

[0021] Figure 1A This is a schematic diagram of the structure of a display system provided in an embodiment of this application;

[0022] Figure 1B This application provides a schematic diagram of an operational scenario between a display device and a control device, as illustrated in an embodiment of the present application.

[0023] Figure 2 This is a schematic diagram of the structure of a control device provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the structure of a display device provided in an embodiment of this application;

[0025] Figure 4 This application provides a schematic diagram of the software architecture of a display device.

[0026] Figure 5 A flowchart illustrating a display method provided in an embodiment of this application;

[0027] Figure 6 A flowchart illustrating a display method provided in an embodiment of this application. Figure 2 ;

[0028] Figure 7 A flowchart illustrating a display method provided in an embodiment of this application. Figure 3 ;

[0029] Figure 8 A flowchart illustrating a display method provided in an embodiment of this application. Figure 4 ;

[0030] Figure 9 A flowchart illustrating a display method provided in an embodiment of this application. Figure 5 ;

[0031] Figure 10 A flowchart illustrating a display method provided in an embodiment of this application. Figure 6 ;

[0032] Figure 11 A flowchart illustrating a display method provided in an embodiment of this application. Figure 7 ;

[0033] Figure 12 A schematic diagram illustrating another display method provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of another display device provided in an embodiment of this application;

[0035] Figure 14 This is a schematic diagram of the structure of another display device provided in an embodiment of this application. Detailed Implementation

[0036] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0037] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0038] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0039] The terms “comprising” and “having” and any variations thereof in this application are intended to cover but not exclude inclusion, for example, a product or device that includes a series of components is not necessarily limited to all the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0040] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0041] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete implementation on its own. It should be noted that the brief descriptions of terminology in this application are merely for the convenience of understanding the embodiments described below, and are not intended to limit the implementation of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0042] First, the terms used in this application are explained as follows:

[0043] Screen mirroring: Also known as screen sharing or multi-screen interaction, screen mirroring is a technology that uses a mobile device A (such as a mobile phone, iPad, or computer) to display the screen of another device B (such as a projector, conference tablet, video wall, electronic whiteboard, or television) in real time. The output content includes various multimedia information such as photos, videos, and audio, as well as the real-time operation screen on device A.

[0044] Green screen: Due to poor network conditions or the influence of the surrounding environment, packet loss or other possible reasons may occur during screen mirroring, resulting in an occasional phenomenon where the screen image is entirely green (i.e., green screen image).

[0045] Parse: This refers to parsing and segmenting data streams. It is used to parse a data stream (such as the basic bitstream ES) and divide the data stream into compressed, encoded frames.

[0046] ES: The bitstream formed by sampling sound and image data in the real world and compressing it using audio and video compression techniques is called the elementary stream (ES). The ES contains information necessary for the decoder to decode the file, such as video dimensions, sampling format, audio sampling rate, and audio channels. For ease of transmission and playback, the audio and video ES data is packaged into a single file; this file is called the audio / video ES stream encapsulation.

[0047] NAL: Network Abstraction Layer. The network abstraction layer (NAL) of H.264 (a video coding format) can be divided into two layers: VCL (video coding layer) and NAL (network abstraction layer). The NAL stores H.264-related parameter information and image information.

[0048] NALU: This stands for NAL-unit. A NAL is composed of multiple NALU units. Each NALU is a long byte string with variable syntax elements, including a one-byte header (indicating the data type) and several integer bytes of payload data (e.g., RBSP).

[0049] RBSP: Raw Byte Sequence Payload. RBSP is a bit string that is a data bit string (SODB) with a terminating bit added. Since SODB byte lengths are not uniform, adding a terminating bit to RBSP can achieve a uniform byte length. Essentially, RBSP is an SODB with a terminating bit added to make the SODB byte length uniform.

[0050] Dropping a frame is the process of intentionally discarding frame data. Video transmission often encounters problems such as unstable networks and insufficient terminal performance. To solve these problems, in addition to continuously optimizing the processing capabilities of hardware and software, frame dropping can be used to provide degraded services.

[0051] Render: This refers to the process of generating video based on data.

[0052] YUV: YUV is a color encoding method that uses luminance and chrominance to specify the color of a pixel. Y represents luminance (Luminance, Luma), while U and V represent chrominance (Chrominance, Chroma). There are three main sampling methods for YUV images: YUV4:4:4 sampling, YUV4:2:2 sampling, and YUV4:2:0 sampling. YUV4:4:4 sampling means that the sampling ratio of the Y, U, and V components is the same, so in the generated image, each pixel has complete information for all three components, each 8 bits (one byte). YUV4:2:2 sampling means that the UV components are sampled at half the rate of the Y components, with the Y and UV components sampled in a 2:1 ratio. YUV4:2:0 sampling means that in each scan, only one chrominance component (U or V) is scanned, and it is sampled in a 2:1 ratio with the Y component. For example, if YU is sampled in a 2:1 ratio in the first scan, then YV is sampled in a 2:1 ratio in the second scan. For each chromaticity component, its horizontal and vertical sampling ratios are 2:1 compared to the Y component.

[0053] RGB: The three letters RGB stand for Red, Green, and Blue, which are called the three primary colors. Adding them in different proportions produces a wide variety of colors. In image display, a 1280*720 image means it has 1280*720 pixels. The color of each pixel is displayed using the RGB encoding method; taking different values ​​for RGB results in different colors. In an RGB image, each pixel has three primary colors: red, green, and blue. Each primary color occupies 8 bits, or one byte, so one pixel occupies 24 bits, or three bytes.

[0054] Miracast: Miracast is a wireless connectivity standard introduced in 2012 that connects devices (such as laptops, tablets, or smartphones) to displays (such as TVs, monitors, or projectors). At the transport layer, it uses TCP (transmission control protocol) or UDP (user datagram protocol). At the application layer, it uses RTSP (real-time streaming protocol) and RTP (real-time transport protocol) to initiate and control the stream for data transmission.

[0055] AirPlay: AirPlay is a wireless technology developed by Apple, primarily used to enable iOS devices, including iPhones, iPads, and MacBooks, to transmit images, audio, and video to AirPlay-enabled display devices via WiFi. It is a form of screen mirroring.

[0056] Demux: Demux refers to parsing the container format of a video to obtain the original audio and video bitstream. The shorter the demux time, the faster the video stream is obtained.

[0057] Ijkplayer is a media player software developed based on ffmpeg, supporting Android and iOS platforms.

[0058] Avsync: Audio and video synchronization. After media data undergoes demultiplexing, audio and video decoding are independent, and they are played independently. The playback speed of the video / audio streams can be determined by the video frame rate and audio sampling rate. Ideally, audio and video playback should be synchronized; however, this requires modulating the playback timestamps of the audio / video streams to achieve audio and video synchronization.

[0059] Currently, screen mirroring has been widely used in home and business settings, becoming one of the mainstream ways people watch videos. Meanwhile, in the context of the 5G era's interconnectedness of everything and screens, screen mirroring is beginning to carry even more value and possibilities.

[0060] During screen mirroring, the mirroring device sends video data to the display device, which then displays that video data. However, video data may contain erroneous frames during transmission, such as missing pixels, duplicate pixels, or pixelation. To ensure a good viewing experience for the user, the display device typically handles these erroneous frames by dropping them. That is, when the display device decodes the video data, if it detects an erroneous frame, it will discard it, preventing the image corresponding to that erroneous frame from being rendered and played to the display device.

[0061] However, due to network fluctuations or other factors, when a display device decodes video data, one frame in the corresponding video data may be entirely green. The display device will not identify this frame as an error frame; that is, it will not discard it but will display it. In other words, a completely green screen will appear on the display device during screen mirroring. Current technology cannot detect and discard green screen frames, resulting in these frames still being rendered and played on the projected device, impacting the user's viewing experience.

[0062] To address the aforementioned issues, this application proposes a display method applicable to display devices. The display device can detect video data sent by a projection device. When a frame corresponding to a completely green image is detected, that frame is discarded. Frames that are not completely green are displayed normally. In other words, this application's solution can detect the frame data corresponding to a green screen frame (i.e., the frame data corresponding to a completely green image) during projection and prevent that green screen frame from being rendered and played to the display device by dropping frames, thus improving the user experience. The display method provided in this application's embodiments is described in detail below with reference to the accompanying drawings.

[0063] The display method provided in this application can be applied to a display system, which may include a projection device and a display device. The projection device can establish a connection with the display device, such as a wireless or wired connection.

[0064] For example, Figure 1A This is a schematic diagram illustrating the composition of a display system used in a display method according to an exemplary embodiment. (Refer to...) Figure 1A As shown, the system includes a display device 01 and a projection device 02. For example, the display device 01 can be a television, and the projection device 02 can be a mobile phone.

[0065] Users can control the projection device 01 via a control device. This control device can be a remote control, and communication between the remote control and the projection device 01 can be achieved through infrared protocol communication, Bluetooth protocol communication, wireless, or other wired methods. Users can control the projection device 01 by inputting user commands through buttons on the remote control, voice input, or control input methods. Furthermore, the projection device 01 can also directly receive user voice input or voice commands through its internally configured voice command acquisition module (e.g., a microphone). In some examples, tablets, computers, laptops, and other smart devices can also be used to control the projection device 01.

[0066] In some examples, the projection device 02 and the display device 01 can have the same or compatible software applications installed, enabling them to connect and communicate via network protocols, thereby achieving one-to-one control and data communication. In this case, the audio and video content displayed on the projection device 02 can also be transmitted to the display device 01 for synchronized display.

[0067] The display device 01 and the projection device 02 can communicate via wired or wireless means. The projection device 02 can provide various content and interactions to the display device 01. For example, the projection device 02 can send video data involved in the display method provided in the embodiments of this application to the display device 01, thereby enabling the display device 02 to play the video from the projection device 01. Alternatively, the projection device 02 can cooperate with the display device 01 to realize a projection display scheme (i.e., the display method provided in the embodiments of this application).

[0068] For example, in the embodiments of this application, the display device 01 can have various implementation forms. For instance, the display device 01 can be a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, or other display devices capable of voice input. This application embodiment does not limit the specific form of the display device 01. This application embodiment uses a television as an example for illustrative purposes.

[0069] For example, in this embodiment, the projection device 02 can be a device with an HDMI interface capable of providing video data, such as a set-top box or a PC (personal computer). In this embodiment, the projection device 02 can send the video sampled and compressed to the display device 01. This embodiment does not limit the specific form of the projection device 02. This embodiment uses a mobile phone as an example for illustration.

[0070] Figure 1B This is a schematic diagram illustrating the operational scenario between the display device and the control device in an embodiment of this application. Figure 1B As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0071] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0072] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0073] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0074] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0075] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0076] Figure 2 An exemplary block diagram of a possible control device 200 is shown. Figure 2 As shown, the control device 200 includes a controller 210, a communication interface 230, a user input / output interface 240, a memory, and a power supply. The control device can receive user input operation commands (e.g., voice commands) and convert the operation commands into commands that the display device 01 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 01.

[0077] For example, taking a television set as the display device, Figure 3 A schematic diagram of the structure of a display device 01 provided in an embodiment of this application is shown.

[0078] like Figure 3 The display device 01 includes at least one of the following: a tuner 110, a communicator 120, a detector 130, an external device interface 140, a controller 150 (or processor 150), a display 160, an audio output interface 170, a memory, a power supply, and a user interface.

[0079] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0080] The display 160 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user interface (UI) for displaying video content, image content, and a menu control interface.

[0081] The display 160 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0082] The communicator 120 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 01 can establish the transmission and reception of control signals and data signals with an external control device or projection device 02 through the communicator 120.

[0083] The user interface can be used to receive control signals from control devices (such as infrared remote controls).

[0084] Detector 130 is used to acquire signals from the external environment or to interact with the external environment. For example, detector 130 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 130 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 130 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0085] The external device interface 140 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0086] The tuner / demodulator 110 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0087] In some embodiments, the controller 150 and the tuner 110 may be located in different separate devices, that is, the tuner 110 may also be located in an external device of the main device where the controller 150 is located, such as an external set-top box.

[0088] The controller 150 controls the operation of the display device 01 and responds to user operations through various software control programs stored in the memory. The controller 150 controls the overall operation of the display device 01. For example, in response to receiving a user command to select a UI object to display on the monitor 160, the controller 150 can perform operations related to the object selected by the user command.

[0089] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0090] Users can input commands through a graphical user interface (GUI) displayed on the monitor 160, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0091] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of a display device. Controls can include at least one of the following visual interface elements: icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0092] Understandably, in general, the implementation of display device functions requires not only the support of the aforementioned hardware, but also the cooperation of software.

[0093] In some embodiments, taking the Android system used by the display device 01 as an example, refer to... Figure 4 As shown, the system of display device 01 can be divided into five layers, from top to bottom: application layer, Java interface layer, native implementation layer, framework layer, and hardware implementation layer.

[0094] In some embodiments, at least one application runs in the application layer. These applications can be Windows programs, system settings programs, or clock programs that come with the operating system; they can also be applications developed by third-party developers. In this embodiment, the application layer can execute the Mircast projection protocol, which is specifically used to execute the real-time streaming protocol (RTSP) to create and control the dialogue between the display device 01 and the projection device 02, process media information streams, and parse data packets sent by the projection device 02. In specific implementations, the projection protocols that the application layer can execute are not limited to the examples above. The application layer may include applications with application projection functionality.

[0095] The Java Interface Layer (JIL) provides the Java Application Programming Interface (Java API) to the application layer. Written in Java code, the JIL includes predefined functions and services. It acts as a central processing unit, determining the actions that applications in the application layer should take. Through the Java API, applications can access system resources and obtain system services during execution.

[0096] like Figure 4As shown, the Java interface layer in this embodiment includes a player creation module, a surface setting module, a setMediaDataSource module, a prepare module, and an info module. The player creation module is used to create a player, such as the ijkplayer player; the surface setting module is used to set the display interface; the setMediaDataSource module is used to configure the media data source; the prepare module is used to preprocess data packets; and the info module is used to return information to the application layer.

[0097] In some examples, after the Java interface layer receives the video data stream from the application layer, the player creation module first creates the player. Next, the surface setting module configures the player's display interface. Then, the setMediaDataSource module retrieves data packets based on the media data source configured in the video data stream. Finally, the prepare module preprocesses the data packets, and then sends the preprocessed data packets to the native implementation layer.

[0098] In some embodiments, the native implementation layer provides support for the upper layer, namely the Java interface layer. When the Java interface layer is used, the operating system runs the C / C++ libraries contained in the native implementation layer to implement the functions to be implemented by the Java interface layer.

[0099] like Figure 4 As shown, in this embodiment of the application, the native implementation layer includes a video stream configuration module, a video stream processing module, a decapsulation module, a cross-layer communication module, a synchronous rendering module, an audio and video synchronization module, and a rendering and playback module.

[0100] In some examples, the video stream configuration module is used to initialize and configure relevant parameters; the video stream processing module is used to retrieve data from the data buffer for decoding, and then put the decoded data into the corresponding data buffer; the decapsulation module is used to decapsulate the video data stream to obtain the ES stream, then parse a frame of ES data from the ES stream, and detect whether there are multiple consecutive "F7DF7D" in the RBSP of the ES data; the cross-layer communication module is used to send the ES data containing multiple consecutive "F7DF7D" in the RBSP to the mediacodec module in the framework layer; the synchronous rendering module is used to render the video, and perform green screen detection based on the YUV data sent by the mediacodec module. If the green screen detection result indicates that the image corresponding to the YUV data is a completely green image, then the YUV data is discarded; the audio and video synchronization module is used to set the playback time of audio and video; and the rendering and playback module is used to play the final video.

[0101] In some examples, after the native implementation layer receives the preprocessed data packet sent by the Java interface layer, firstly, the video stream configuration module initializes and configures relevant parameters, and puts the preprocessed data packet into the corresponding data buffer; secondly, the video stream processing module retrieves data from the data buffer for decoding, and after decoding, puts the decoded data into the corresponding data buffer; then, the decapsulation module decapsulates the video data stream to obtain the ES stream, then parses a frame of ES data from the ES stream, and checks whether there are multiple consecutive "F7DF7D" in the RBSP in the ES data; finally, the cross-layer communication module sends the ES data containing multiple consecutive "F7DF7D" in the RBSP to the mediacodec module in the framework layer, so that the mediacodec module processes the ES data to obtain YUV data and returns it to the synchronous rendering module.

[0102] While the decapsulation module decapsulates the video data stream, the synchronous rendering module renders the video. Upon receiving YUV data from the mediacodec module, it performs a green screen detection on the YUV data. If the green screen detection result indicates that the image corresponding to the YUV data is entirely green, the YUV data is discarded. Then, the audio and video synchronization module sets the playback time of the audio and video and performs audio and video synchronization. Finally, the rendering and playback module plays the final video.

[0103] In some examples, the framework layer is used to provide encoding / decoding interfaces to the native implementation layer, for example, see reference Figure 4 As shown, the framework layer contains the mediacodec module, which is used to receive data from the native implementation layer, convert it into YUV format, and then return the YUV data to the native implementation layer.

[0104] In some examples, the hardware implementation layer is the layer between hardware and software. The hardware implementation layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), MIC driver, and power driver, etc.

[0105] The video data involved in this application may be data authorized by the user or fully authorized by all parties.

[0106] The methods in the following embodiments can be implemented in a display device having the above-described hardware and software structures. The display methods provided in the embodiments of this application will be described.

[0107] Reference Figure 5As shown, this application embodiment provides a display method applied to a display device (i.e., a device to which the screen is projected), and the method may include S510-S540:

[0108] S510 receives video data sent from the projection device.

[0109] In some examples, the display device can receive video data sent from the projection device. This video data can be data formed by sampling and compressing video, and may include multiple frames. For example, the projection device could be a mobile phone using the Mircast projection protocol, and the display device could be a television.

[0110] In some examples, when the projection device and the display device are on the same local area network, after receiving the projection command, the projection device and the display device can establish a communication connection. The projection device compresses or modulates the video data to be projected, forms a video data stream, and sends it to the display device through the local area network, so that the display device can receive the video data.

[0111] Understandably, in other examples, where the projection device and the display device are on the same local area network, the projection device and the display device can establish a communication connection first, and then the projection device receives the projection command. This application does not limit the order in which the projection device receives the projection command and the projection device establishes a communication connection with the display device.

[0112] For example, the screen mirroring device can be a mobile phone using the Mircast screen mirroring protocol, and the display device can be a television using the Mircast screen mirroring protocol. When the mobile phone and the television are on the same local area network, the screen mirroring protocol end of the mobile phone and the screen mirroring protocol end of the television establish a communication connection. The screen mirroring protocol end of the mobile phone sends the compressed and modulated video data to the screen mirroring protocol end of the television. The screen mirroring protocol end of the mobile phone and the screen mirroring protocol end of the television can execute the same screen mirroring protocol.

[0113] It is understood that, in this embodiment, the video data sent by the projection device may include not only audio and video data, but also image data, audio data, mixed audio and video data, and other multimedia data, as well as mirrored data of the projection device's display interface. This embodiment does not limit the specific type of data sent by the projection device. In this embodiment, to determine which frames in the video data correspond to images that are entirely green, the video data first needs to be divided into multiple frames, i.e., step S520 is executed.

[0114] Combination Figure 4As shown, when the projection device and the display device are on the same local area network (LAN), the projection device compresses or modulates the video data to be projected, forming a video data stream, and sends it to the display device via the LAN using a first application with projection functionality. The second application (an application with projection functionality) in the display device's application layer receives the video data sent from the projection device (i.e., S510). In practice, the first application and the second application can be the same application or different applications capable of establishing a projection connection.

[0115] S520: Perform frame-segmentation on the video data to obtain multiple frame data.

[0116] In some examples, video data is processed into multiple frames by dividing the video data into frames. This can include dividing the video data according to the boundary information corresponding to the video frames to obtain multiple frames.

[0117] For example, segmenting video data into multiple frames can involve first determining boundary information such as video frame width, video frame height, aspect ratio, frame rate, bit rate, and sampling format. Then, the video data is segmented according to this boundary information to obtain multiple frames.

[0118] To prevent a fully green image from being rendered and played to the display device, it is first necessary to detect whether the video contains a fully green image, so that the fully green image can be deleted, i.e., S530 is executed.

[0119] Combination Figure 4 As shown, S520 can specifically be a decapsulation module in the native implementation layer of the display device that performs frame-segmentation processing on the received video data to obtain multiple frame data.

[0120] S530: If the image corresponding to the first frame of data is a completely green image, discard the first frame of data.

[0121] The first frame data refers to the frame data corresponding to a completely green image, which is one of the multiple frames segmented from the video data. It should be noted that in practice, there may be multiple first frames among the multiple frames segmented from the video data.

[0122] After obtaining multiple frame data through frame segmentation, the display device can first determine whether a first frame exists among the multiple frame data. The first frame is the frame data corresponding to the all-green image. If the first frame exists, it is deleted. That is, if the first frame exists among the multiple frame data and the image corresponding to the first frame is an all-green image, the display device will discard the first frame.

[0123] Even if the first frame of data is not available, the display device can display the multiple frames of data.

[0124] For example, taking a television as the display device, the display device can draw the image corresponding to the frame data through the frame data, and then determine whether the image corresponding to the frame data is a full green image. If the image corresponding to the frame data is a full green image, then the frame data is determined to be the first frame data, and the first frame data is discarded to prevent the first frame data from being rendered and played to the television.

[0125] Combination Figure 4 As shown, the specific implementation of S530 can be as follows: After receiving the YUV data sent by the mediacodec module of the framework layer, the synchronous rendering module of the display device performs green screen detection on the YUV data. If the green screen detection result indicates that the image corresponding to the YUV data is a full green image, then the YUV data is discarded.

[0126] In some examples, the display device discards the first frame of data, but the remaining frame data still needs to be rendered and played.

[0127] Based on this, after step S530, the method further includes step S540:

[0128] S540. Display the second frame data. The second frame data is the frame data other than the first frame data among multiple frame data.

[0129] The second frame data is the frame data for which the corresponding image is not entirely green. It is another part of the multiple frame data segmented from the video data, excluding the first frame data.

[0130] In some examples, the display device controls the display to play the video after the second frame of data has been rendered.

[0131] Combination Figure 4 As shown, the specific implementation of S540 can be as follows: the audio and video synchronization module and the rendering and playback module of the native implementation layer of the display device perform audio and video synchronization operation and rendering and playback operation on the second frame data, respectively.

[0132] In the technical solutions corresponding to S510 to S540, the display device can perform frame-by-frame processing on the video data to obtain multiple frame data. Among these frame data, the first frame data is determined, and if the image corresponding to the first frame data is entirely green, the first frame data is discarded. This improves the green screen issue during projection, prevents entirely green images from being rendered and played to the display device, and enhances the user experience.

[0133] In some examples, to determine which frames among all the images corresponding to all the frame data correspond to are entirely green, one can determine whether the image corresponding to each frame is entirely green by examining the frame data itself. For instance, if the color portion of the first frame data (indicating the color of all pixels in the image corresponding to that frame) is entirely a preset string, then the image corresponding to that frame data can be determined to be entirely green. Based on this, combined with... Figure 5 , refer to Figure 6 As shown, S530 may specifically include S610-S620.

[0134] S610. If the color part of the first frame data is all a preset string, determine that the image corresponding to the first frame data is a full green image.

[0135] In some examples, the color portion of the first frame data is used to indicate the color of all pixels in the image corresponding to the first frame data, and the default string is a string used to indicate that the pixel color is green. For example, the color portion of the first frame data can be the color-related part in the RBSP of the frame data, and the default string can be "F7DF7D".

[0136] In some examples, after receiving video data from the projection device, the display device first retrieves the first frame of data from the video data. Then, it detects the portion of the first frame that indicates the color of the corresponding image pixels. If this portion is entirely a preset string, the image corresponding to the first frame is determined to be entirely green. If this portion is not entirely a preset string, the image is determined to be not entirely green.

[0137] For example, if the display device is a television and the projection device is a mobile phone, the color portion of the frame data can be the color-related part of the RBSP in the frame data, and the default string can be "F7DF7D". After receiving the video data sent by the mobile phone, the television first decapsulates the video data and parses it into an ES frame. Then, it checks the string of the color-related part of the RBSP in the ES frame data. If the entire string in this part is "F7DF7D", which indicates that the pixel color is green, then the image corresponding to the ES frame data can be determined to be a full green image. If the entire string in this part is not "F7DF7D", which indicates that the pixel color is green, then the image corresponding to the ES frame data can be determined to be not a full green image.

[0138] In some examples, after detecting that the image corresponding to the first frame of data is a completely green image, the first frame of data needs to be discarded, i.e., S620 is executed.

[0139] S620, Discard the first frame of data.

[0140] In some examples, S620 specifically deletes the first frame of data that is determined to be a completely green image.

[0141] In the corresponding technical solutions in S610 to S620, if the color portion of the first frame data consists entirely of a preset string, the image corresponding to the first frame data is determined to be an all-green image, and the first frame data is discarded. In this way, the above steps can accurately determine the first frame data to be discarded from the video data, i.e., the frame data corresponding to the all-green image, without any omissions, ensuring that all frame data corresponding to all-green images are discarded.

[0142] In some examples, detecting all frame data would incur a large amount of unnecessary computation, as not every frame needs to be detected. Therefore, in other examples, a subset of frame data can be selected first, and then the frames corresponding to the all-green images that need to be discarded can be determined from the selected frame data. Based on this, combined with Figure 5 ,refer to Figure 7 As shown, S530 may also include S710-S730.

[0143] S710. If the first frame of data contains a preset number of consecutive preset strings, the first frame of data is parsed to obtain the parsed data corresponding to the first frame of data.

[0144] For example, the preset number can be 3. In practice, the preset number can be determined according to specific needs, and this application does not impose any specific restrictions on it.

[0145] In some examples, if the display device parses the first frame of data, which contains a preset number of consecutive preset strings, to obtain the parsed data corresponding to the first frame. Here, the first frame is any one of multiple frames after the video data has been segmented, and the preset strings can be strings indicating that a pixel's color is green. The parsed data corresponding to the first frame can be used to indicate the pixel brightness and chromaticity of the image corresponding to the frame.

[0146] In some examples, after receiving video data from the projection device, the display device first retrieves the first frame of data from the video data and then detects the raw byte payload in that first frame. If a predetermined number of consecutive preset strings are found in the raw byte payload, it can be assumed that the image corresponding to the first frame has a certain probability of being a completely green image. Subsequently, the first frame can be decoded to obtain parsed data, which facilitates green screen detection to confirm whether the image corresponding to the first frame is indeed a completely green image.

[0147] If the original byte payload does not contain a set number of consecutive preset strings, then the image corresponding to the first frame of data is not considered to be entirely green. Subsequently, the first frame of data can be decoded to obtain parsed data, which can then be used for operations such as audio / video synchronization, rendering, and playback.

[0148] For example, taking a television as the display device, the first frame of data can be ES data, and the preset string can be "F7DF7D". The parsed data corresponding to the first frame can be data encoded in YUV color. After receiving the video data, the television first extracts one frame of ES data and then detects the original byte payload of that ES data. If a preset number of consecutive "F7DF7D" characters are found in the original byte payload, meaning that the image corresponding to that frame of ES data has consecutive green pixels, then that frame of ES data is sent to the decoder for decoding. The decoder outputs parsed data encoded in YUV, and then performs green screen detection on this parsed data to confirm whether the image corresponding to the first frame of data is a completely green image.

[0149] Combination Figure 4 As shown, the specific implementation of S710 can be as follows: the cross-layer communication module of the native implementation layer of the display device sends the first frame data containing a preset number of consecutive preset strings to the mediacodec module of the framework layer. The mediacodec module parses the first frame data into the parsed data corresponding to the first frame data.

[0150] In some examples, after filtering the first frame of data containing a preset number of consecutive preset strings, it can be determined that the image corresponding to the first frame of data has a preset number of consecutive green pixels, but it cannot be determined that the image corresponding to the first frame of data is a completely green image. Therefore, further green screen detection is required for these first frame data, i.e., S720 is executed.

[0151] S720: Perform green screen detection on the parsed data corresponding to the first frame of data.

[0152] The green screen detection is used to determine whether all the pixels in the image corresponding to the first frame of data are green. For example, the color of a pixel can be determined based on its RGB color value, and then whether the image is a completely green image can be determined by whether all the pixels are green.

[0153] It is understandable that green screen detection can also be performed on the parsed data corresponding to the first frame of data in other ways. For example, an image corresponding to the first frame of data can be drawn based on the parsed data, and then the image can be compared with one or more all-green sample images. If the color of any pixel in the image is inconsistent with the color of the all-green sample image, then the image is determined not to be an all-green image; otherwise, the image is determined to be an all-green image. This application does not limit the green screen detection method that can detect an image as an all-green image.

[0154] In the first frame data that has been filtered out and contains a preset number of consecutive preset strings, after further determining the first frame data corresponding to the all-green image, the first frame data corresponding to the pure green image is discarded, and other first frame data containing green pixels are retained, i.e., S730 is executed.

[0155] S730: If the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a completely green image, discard the parsed data corresponding to the first frame data.

[0156] For example, the display device could be a television set. After the television set performs green screen detection on the parsed data corresponding to the first frame of data, if it determines that the image corresponding to the first frame of data is a completely green image, then it deletes the parsed data corresponding to the first frame of data.

[0157] In the technical solutions corresponding to S710 to S730 above, when the first frame data contains a preset number of consecutive preset strings, the first frame data is parsed to obtain parsed data corresponding to the first frame data. Green screen detection is then performed on the parsed data corresponding to the first frame data. If the green screen detection result indicates that the image corresponding to the first frame data is entirely green, the parsed data corresponding to the first frame data is discarded. It can be seen that this technical solution reduces the computational load when processing video data and shortens the screen projection execution time by selecting the first frame data corresponding to a portion of the frame data containing consecutive green pixels and further performing green screen detection.

[0158] In some examples, the frame data is color-coded in YUV format. This format indicates the luminance and chrominance of the corresponding image, but it does not directly indicate the color of the corresponding image pixels. Therefore, in some examples, to quantify and determine the display of green screen images, the YUV data can be converted to RGB data. Specifically, this can be done by combining... Figure 7 ,refer to Figure 8 As shown, step S720 may include steps S810 and S820.

[0159] S810, if the color encoding method of the parsed data corresponding to the first frame data is YUV, convert the parsed data corresponding to the first frame data into target data.

[0160] The target data is encoded in RGB color.

[0161] In some examples, converting YUV encoded data to RGB encoded data means converting the Y, U, and V components of all pixels in the image into R, G, and B components.

[0162] For example, the display device can be a television. After acquiring YUV data, the television can calculate the RGB components of each pixel based on the Y, U, and V components of each pixel. See Formula 1 and Formula 2 below for details, where Y indicates the pixel's luminance, U and V indicate the pixel's chrominance, R indicates the pixel's red color value, G indicates the pixel's green color value, and B indicates the pixel's blue color value.

[0163] Formula 1:

[0164]

[0165] Formula 2:

[0166]

[0167] In some examples, the definition of "green" is not strictly speaking a color with an RGB color value of 0:255:0. Images with a G color value within a certain range can be judged as full green images.

[0168] S820. If the green color value indicated by the target data is within a preset range, then the green screen detection result of the parsed data indicates that the image corresponding to the parsed data is a full green image.

[0169] The preset range is used to indicate the range of colors that can be identified as green. For example, the preset range can be 235-255.

[0170] In some examples, it is determined whether the green color value of each pixel, i.e. the G value in RGB, is within a preset range. If the green color value of all pixels is within the preset range, it can be determined that all pixels are green, and thus it can be determined that the image corresponding to the parsed data is a full green image.

[0171] For example, the display device can be a television, the preset range can be 235-255, and the target data is the color data of a 1280×720 image. When the green color value of all 1280×720 pixels is within 235-255, the television determines that the 1280×720 image is a full green image, that is, the image corresponding to the target data is a full green image.

[0172] In the technical solutions corresponding to S810 and S820 described above, when the color encoding method of the parsed data corresponding to the first frame data is YUV, the display device converts the parsed data corresponding to the first frame data into target data. If the green color value indicated by the target data is within a preset range, then the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image. Since RGB data more directly indicates the color of pixels than the chromaticity and grayscale indicated by YUV data, converting YUV data into RGB data and performing green screen detection on the RGB data can obtain green screen detection results more quickly and accurately.

[0173] Furthermore, because the color range of green is reasonably defined in this scheme, the flexibility of the display scheme provided by this application is increased, and other images with green values ​​other than 255 that are likely to be perceived as full green by users are avoided from being rendered and played to the client, thus improving the user experience.

[0174] It should be noted that if it is determined that the image corresponding to a certain frame of data is not a completely green image, then that frame of data will still be rendered and played to the display device. For example, combining... Figure 8 ,refer to Figure 9 As shown, rendering and playing a non-full green screen image may include the following steps S910-S940:

[0175] S910. If the green color value indicated by the target data is outside the preset range, determine that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is not a fully green image.

[0176] The specific definition of the preset range can be found in the relevant descriptions in the foregoing embodiments, and will not be repeated here.

[0177] In some examples, the display device determines whether the green color value of each pixel, i.e. the G value in RGB, is outside a preset range. If the green color value of any one or more pixels is outside the preset range, it can be determined that there are pixels that are not green, and thus it can be determined that the image corresponding to the parsed data is not a fully green image.

[0178] For example, taking a television as the display device, the preset range can be 235-255. The target data is the color data of a 1280×720 image. When the green color value of any one or more pixels is outside the range of 235-255, the display device can determine that the 1280×720 image is not a full green image, that is, the image corresponding to the target data is not a full green image.

[0179] If a frame of data is determined to be entirely green, audio and video synchronization can be performed on that frame to generate a corresponding video for display device rendering and playback, i.e., S920 is executed.

[0180] S920: Perform audio and video synchronization operation on the parsed data corresponding to the first frame of data to obtain the first data to be played.

[0181] Audio-video synchronization refers to the process of correctly matching and playing video and audio. For example, audio-video synchronization can be achieved through the avsync operation.

[0182] In some examples, the display device sets an external clock independent of the video and audio playback clocks, and uses this external clock as a reference for both video and audio playback speeds. When the video playback clock is slower than the audio playback clock, the display device pauses the audio and waits for the video to catch up, or the video catches up with the audio by dropping frames. When the video playback clock is faster than the audio playback clock, the display device pauses the video and waits for the audio to catch up, or the audio catches up with the video by dropping frames. After audio and video synchronization is complete, the display device uses the synchronized data as the data to be played.

[0183] In some examples, the synchronized audio and video data is rendered and played, i.e., S930 is executed.

[0184] S930, renders the first data to be played.

[0185] In some examples, the display device renders the first data to be played. Rendering is the process of forming a video based on the data to be played. The display device determines parameters such as the color of each pixel, source image brightness, output image brightness, source image height, output image height, source image width, output image width, video playback timestamp, and audio playback timestamp based on the data to be played. It then synthesizes the audio and video data according to these parameters and plays it on the display interface of the display device, i.e., executes S940.

[0186] S940, the first data to be played after rendering.

[0187] In some examples, the display device's controller controls the display to play the first data to be played after rendering.

[0188] In the technical solutions corresponding to S910 to S940 above, when the green color value indicated by the target data is outside the preset range, the display device determines that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is not a fully green image. It then performs audio-visual synchronization operations on the parsed data corresponding to the first frame data to obtain the first data to be played, renders the first data to be played, and plays the rendered first data to be played. This technical solution ensures that all images containing consecutive green pixels but not entirely green can be rendered and played, guaranteeing the integrity of the video and the overall solution.

[0189] Understandably, in practical applications, after discarding or rendering a frame of data, the display device needs to check if there are any other frames that have not been discarded or rendered. If there are any frames that have not been discarded or rendered, the display device will check if the image corresponding to that frame is a completely green image, so that it can be discarded or rendered subsequently. If there are no frames that have not been discarded or rendered, the current screen projection process ends.

[0190] In some examples, there might be multiple consecutive frames where the images are all green. Discarding all of these would be noticeable to the viewer, thus degrading the viewing experience. Therefore, combined with... Figure 10 As shown, this method also includes: displaying a prompt message when the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a completely green image.

[0191] The notification message is used to indicate to the user that the display device is in the process of screen mirroring.

[0192] In some examples, when the image corresponding to the first frame of data is determined to be entirely green, an image containing a prompt message is played on the display device's screen instead of the entirely green image. The prompt message can be "Casting in progress, please wait" or other information indicating that the casting device is in the casting process. Furthermore, when multiple consecutive frames are determined to be entirely green, a prompt video of a certain duration is played in rotation on the display device's screen. The prompt video can contain various carriers of the prompt message, such as images, sound effects, artificial voice, and animation.

[0193] This avoids the visually fragmented experience for viewers caused by discarding consecutive all-green images.

[0194] In some embodiments, even if the first frame of data does not contain a preset number of consecutive preset strings, parsing and rendering operations are still performed on the first frame of data to enable playback of that first frame. Specifically, in conjunction with Figure 7 ,refer to Figure 11As shown, rendering and playing an image that does not contain consecutive green pixels includes the following steps S1110 to S1140.

[0195] S1110. If the first frame of data does not contain a preset number of consecutive preset strings, parse the first frame of data to obtain the parsed data corresponding to the first frame of data.

[0196] The parsed data is in YUV color encoding format.

[0197] In some examples, the display device sends frame data that does not contain consecutive green pixel images to the decoder, which decodes the frame data into YUV format data.

[0198] In some examples, the display device performs audio and video synchronization on images that do not contain consecutive green pixels for subsequent rendering and playback, i.e., performs S1120.

[0199] S1120. Perform audio and video synchronization operation on the parsed data corresponding to the first frame data to obtain the second data to be played.

[0200] In some examples, the display device performs audio-video synchronization on the parsed data corresponding to the first frame of data to obtain the second frame of data to be played. Audio-video synchronization involves correctly matching and playing the video and audio. For example, audio-video synchronization could be an AVSync operation.

[0201] In some examples, the display device uses an external clock independent of the video and audio playback clocks. The playback speed of both video and audio is based on this external clock. When the video playback clock is slower than the audio playback clock, the audio is paused until the video catches up, or the video catches up by dropping frames to achieve audio-video synchronization. Conversely, when the video playback clock is faster than the audio playback clock, the video is paused until the audio catches up, or the audio catches up by dropping frames to achieve audio-video synchronization. The synchronized audio-video data is then used as the data to be played.

[0202] S1130, Render the second data to be played.

[0203] In some examples, a second set of data to be played is rendered. Rendering is the process of creating a video based on the data to be played.

[0204] In some examples, the display device determines parameters such as the color of each pixel, the brightness of the source image, the brightness of the output image, the height of the source image, the height of the output image, the width of the source image, the width of the output image, the video playback timestamp, and the audio playback timestamp based on the second data to be played. The display device then synthesizes the audio and video data based on the above parameters and plays it on the display interface of the display device, i.e., executes S1140.

[0205] In S1140, the second data to be played after rendering.

[0206] In some examples, the display device's controller controls the display to play a second set of data to be played after it has been rendered.

[0207] Based on the technical solutions corresponding to S1110-S1140, the display device can perform operations such as parsing, audio-video synchronization, and rendering playback on the first frame of data when the first frame does not contain a preset number of consecutive preset strings. This ensures that all images that are not entirely green can be rendered and played, guaranteeing the integrity of the video and the overall solution. In one example embodiment of this application, using a mobile phone as the screen mirroring device, a television as the screen-mirrored device, ijkplayer as the player, and mircast as the screen mirroring protocol, the display method provided by this application may include, for example... Figure 12 The process shown is for reference only. Figure 12 As shown, the display method provided in this application may specifically include:

[0208] 1. The mobile phone samples and compresses the video data. For example, the screen projection device encapsulates the video data according to the RTSP protocol and transmits it in real time, that is, it encapsulates the video stream data according to the RTP data packet format and sends it to the display device.

[0209] 2. The television parses data packets. For example, after receiving data packets sent by the projection device, the communicator of the display device decodes the data packets according to the same protocol to obtain the ES (basic stream) data.

[0210] 3. The television sets create the ijkplayer player. For example, the display device's controller creates the ijkplayer player.

[0211] 4. Initialize the TV player and configure relevant parameters. For example, initialize the ijkplayer player interface, set the playback path, and preprocess the decoded ES data.

[0212] 5. De-packing the television.

[0213] For example, the demultiplexer is first initialized, then the encapsulated Elasticsearch data is sent to the demultiplexer, and finally the Elasticsearch data is decapsulated by the demultiplexer.

[0214] 6. The TV parses frames.

[0215] For example, determine the boundary information such as video frame width, video frame height, aspect ratio, frame rate, bit rate, and sampling format, and then segment the ES data according to the above boundary information to obtain multiple frame data.

[0216] The specific implementation of step 6 can be referred to the specific description of S520 in the aforementioned embodiment, and will not be repeated here.

[0217] 7. The TV detects strings.

[0218] For example, if the RBSP in the NALU of the detected frame data contains multiple consecutive "F7DF7D" strings, then execute step 8; otherwise, execute step 9.

[0219] The specific implementation of 7 can be referred to the specific description of S710 in the aforementioned embodiments, and will not be repeated here.

[0220] 8. The TV will notify the renderer of the string detection result.

[0221] For example, record frame data in the RBSP of the NALU containing multiple consecutive "F7DF7D" strings and notify the renderer.

[0222] 9. Television decoding. For example, the frame data containing multiple consecutive "F7DF7D" strings in the RBSP of the NALU is sent to the decoder for decoding to obtain parsed data in YUV color encoding format.

[0223] The specific implementation of step 9 can be referred to the specific description of S710 in the aforementioned embodiments, and will not be repeated here.

[0224] 10. Television Decoding. For example, frame data in NALU that does not contain multiple consecutive "F7DF7D" strings in the RBSP is sent to the decoder for decoding, resulting in parsed data in YUV color encoding format. After decoding, step 13 is executed directly for rendering and playback.

[0225] The specific implementation of 10 can be referred to the specific description of S1110 in the aforementioned embodiments, and will not be repeated here.

[0226] 11. TV Green Screen Detection. For example, convert the YUV format parsed data to RGB format parsed data, and determine whether the image corresponding to the parsed data is a completely green image by using the value of G. If it is a completely green image, proceed to step 12; otherwise, proceed to step 13.

[0227] The specific implementation of 11 can be referred to the specific description of S810 in the aforementioned embodiments, and will not be repeated here.

[0228] 12. Television frame drop handling. For example, deleting parsed data indicating that the corresponding image is a completely green image from the green screen detection result.

[0229] The specific implementation of 12 can be referred to the specific description of S820 in the aforementioned embodiments, and will not be repeated here.

[0230] 13. Television rendering and playback. For example, first, audio and video are synchronized; after synchronization, the data is rendered and parsed into video and then played.

[0231] The specific implementation of 13 can be referred to the specific description of S540 in the aforementioned embodiments, and will not be repeated here.

[0232] 14. The television checks if there is still frame data. For example, if unprocessed frame data is detected, it returns to S1260; otherwise, it proceeds to 15.

[0233] 15. Process complete.

[0234] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0235] This application embodiment can divide the display device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0236] Reference Figure 13 As shown in the figure, this application embodiment provides a display device, which may include a receiving module 131, a framing module 132, a frame dropping module 133, and a display module 134.

[0237] Specifically, the receiving module 131 is used to receive video data sent from the projection device; the frame segmentation module 132 is used to perform frame segmentation processing on the video data to obtain multiple frame data; the frame dropping module 133 is used to discard the first frame data when the image corresponding to the first frame data among the multiple frame data obtained by the frame segmentation module is a completely green image; the first frame data is the frame data among the multiple frame data; the display module 134 is used to display the second frame data among the multiple frame data obtained by the frame segmentation module, the second frame data is the frame data among the multiple frame data excluding the first frame data.

[0238] In one possible implementation, the frame dropping module 133 is specifically used to: determine that the image corresponding to the first frame data is a full green image when the color part of the first frame data is all preset strings; wherein, the color part of the first frame data is used to indicate the color of all pixels of the image corresponding to the first frame data; and discard the first frame data.

[0239] In one possible implementation, the frame dropping module 133 is specifically used to: parse the first frame data to obtain the parsed data corresponding to the first frame data when the first frame data contains a preset number of consecutive preset strings; perform green screen detection on the parsed data corresponding to the first frame data; and discard the parsed data corresponding to the first frame data when the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image.

[0240] In one possible implementation, the frame dropping module 133 is specifically used to: convert the parsed data corresponding to the first frame data into target data when the color encoding method of the parsed data corresponding to the first frame data is YUV; the color encoding method of the target data is RGB; and if the green color value indicated by the target data is within a preset range, determine that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image.

[0241] In one possible implementation, the frame dropping module 133 is specifically used to: determine that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is not a fully green image when the green color value indicated by the target data is outside the preset range; perform audio and video synchronization operation on the parsed data corresponding to the first frame data to obtain the first data to be played; render the first data to be played; and control the display to play the rendered first data to be played.

[0242] In one possible implementation, the frame dropping module 133 is specifically used to: display a prompt message when the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image; the prompt message is used to indicate that the user's display device is in the process of screen projection.

[0243] In one possible implementation, the frame dropping module 133 is specifically used to: parse the first frame data to obtain the parsed data corresponding to the first frame data when the first frame data does not contain a preset number of consecutive preset strings; perform audio and video synchronization operation on the parsed data corresponding to the first frame data to obtain the second data to be played; render the second data to be played; and control the display to play the rendered second data to be played.

[0244] It should be understood that the division of units or modules (hereinafter referred to as units) in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, all units in the device can be implemented in software through processing element calls; all units can be implemented in hardware; or some units can be implemented in software through processing element calls, and some units can be implemented in hardware.

[0245] For example, each unit can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as a program in memory, invoked and executed by a processing element within the device. Furthermore, these units can be integrated in whole or in part, or implemented independently. The processing element here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In implementation, each step of the above method or each of the above units can be implemented through integrated logic circuits in the processor element or through software invoked by the processing element.

[0246] In one example, the unit in the above device may be one or more integrated circuits configured to implement the above methods, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0247] For example, when the units in the device can be implemented through a processing element scheduler, the processing element can be a general-purpose processor, such as a CPU or other processor capable of calling programs. Alternatively, these units can be integrated together to form a system-on-a-chip (SoC).

[0248] In one implementation, the units that implement the corresponding steps in the above method can be implemented in the form of a processing element scheduler. For example, the device may include a processing element and a storage element, wherein the processing element calls a program stored in the storage element to execute the display method of the above method embodiment. The storage element may be a storage element located on the same chip as the processing element, i.e., an on-chip storage element.

[0249] In another implementation, the program used to perform the above method can be located on a storage element on a different chip than the processing element, i.e., an off-chip storage element. In this case, the processing element calls or loads the program from the off-chip storage element onto the on-chip storage element to call and execute the display method of the above method embodiment.

[0250] Reference Figure 14As shown, this application embodiment also provides a display device, including a display 141; a communicator 143 configured to receive video data sent from a projection device; wherein the video data includes multiple frame data; a processor 142 coupled to the display and the communicator respectively, and configured to: perform frame-segmentation processing on the video data to obtain multiple frame data; discard the first frame data if the image corresponding to the first frame data is a full green image; the first frame data is frame data among the multiple frame data; control the display to display the second frame data, the second frame data being frame data other than the first frame data among the multiple frame data.

[0251] In one possible implementation, the processor 142 is configured to discard the first frame data when the image corresponding to the first frame data is an all-green image, including: determining that the image corresponding to the first frame data is an all-green image when the color portion of the first frame data is entirely a preset string; wherein the color portion of the first frame data is used to indicate the color of all pixels of the image corresponding to the first frame data; and discarding the first frame data.

[0252] In one possible implementation, the processor 142 is configured to discard the first frame data when the image corresponding to the first frame data is a completely green image, including: parsing the first frame data to obtain parsed data corresponding to the first frame data when the first frame data contains a preset number of consecutive preset strings; performing green screen detection on the parsed data corresponding to the first frame data; and discarding the parsed data corresponding to the first frame data when the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a completely green image.

[0253] In one possible implementation, the processor 142 is configured to perform green screen detection on the parsed data corresponding to the first frame data, including: when the color encoding method of the parsed data corresponding to the first frame data is YUV, converting the parsed data corresponding to the first frame data into target data; the color encoding method of the target data is RGB; and when the green color value indicated by the target data is within a preset range, determining that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image.

[0254] In one possible implementation, the processor 142 is further configured to: determine that the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is not a fully green image when the green color value indicated by the target data is outside a preset range; perform audio and video synchronization operation on the parsed data corresponding to the first frame data to obtain the first data to be played; render the first data to be played; and control the display to play the rendered first data to be played.

[0255] In one possible implementation, the processor 142 is further configured to display a prompt message when the green screen detection result of the parsed data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image; the prompt message is used to indicate that the user's display device is in the process of screen projection.

[0256] In one possible implementation, the processor 142 is further configured to: parse the first frame data to obtain parsed data corresponding to the first frame data if the first frame data does not contain a preset number of consecutive preset strings; perform audio and video synchronization operation on the parsed data corresponding to the first frame data to obtain second data to be played; render the second data to be played; and control the display to play the rendered second data to be played.

[0257] In one possible implementation, the default string is F7DF7D.

[0258] This application also provides a display device, which may include a display screen, a memory, and one or more processors. The display screen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the display device can perform various functions or steps performed by the display device (such as a television) in the above method embodiments.

[0259] For example, this application also provides a chip that can be applied to the aforementioned display device or server. The chip includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the processor receives and executes computer instructions from the display device's memory through the interface circuits to implement the methods described in the above method embodiments.

[0260] This application also provides a computer-readable storage medium storing computer program instructions (or instructions). When the computer program instructions are executed by a display device, the display device can implement the display method described above.

[0261] This application also provides a computer program product, including computer instructions for operation of the display device as described above. When the computer program product is run in the display device, the display device enables the display method as described above.

[0262] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0263] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0264] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0265] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0266] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product, such as a program. This software product is stored in a program product, such as a computer-readable storage medium, and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0267] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A display device, characterized by comprising: The display device comprises: a display; a communicator configured to receive video data transmitted from a casting device; a processor coupled to the display and the communicator, and configured to: perform frame splitting processing on the video data to obtain a plurality of frame data; in a case where an image corresponding to first frame data is a full green image, discard the first frame data; wherein the first frame data is frame data in the plurality of frame data; control the display to display second frame data, the second frame data being frame data in the plurality of frame data except the first frame data; wherein the processor, in a case where the image corresponding to the first frame data is a full green image, is configured to discard the first frame data, comprising: in a case where the first frame data contains a preset number of continuous preset strings, performing analysis on the first frame data to obtain analysis data corresponding to the first frame data; in a case where a color coding mode of the analysis data corresponding to the first frame data is YUV, converting the analysis data corresponding to the first frame data into target data; the color coding mode of the target data being RGB; in a case where a color value of green indicated by the target data is within a preset range, determining that a green screen detection result of the analysis data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image; in a case where the green screen detection result of the analysis data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image, discarding the analysis data corresponding to the first frame data, and displaying prompt information; the prompt information being used to indicate a user that the display device is in a casting process; in a case where the first frame data does not contain a preset number of continuous preset strings, performing analysis on the first frame data to obtain analysis data corresponding to the first frame data; performing audio-video synchronization operation on the analysis data corresponding to the first frame data to obtain second to-be-played data; rendering the second to-be-played data; controlling the display to play the second to-be-played data after rendering.

2. The display device of claim 1, wherein, The processor, in a case where the image corresponding to the first frame data is a full green image, is configured to discard the first frame data, comprising: in a case where a color part of the first frame data is all preset strings, determining that the image corresponding to the first frame data is a full green image; wherein the color part of the first frame data is used to indicate colors of all pixels of the image corresponding to the first frame data; discarding the first frame data.

3. The display device of claim 1, wherein, The processor is further configured to: in a case where the color value of green indicated by the target data is outside the preset range, determining that the green screen detection result of the analysis data corresponding to the first frame data indicates that the image corresponding to the first frame data is not a full green image; performing audio-video synchronization operation on the analysis data corresponding to the first frame data to obtain first to-be-played data; rendering the first to-be-played data; controlling the display to play the first to-be-played data after rendering.

4. A display method characterized by comprising: The display method applied to a display device, comprising: receive video data sent by a screen projection device; perform frame splitting on the video data to obtain a plurality of frame data; in a case where an image corresponding to first frame data is a full green image, discard the first frame data; the first frame data is frame data in the plurality of frame data; display second frame data, the second frame data being frame data in the plurality of frame data except the first frame data; in a case where an image corresponding to first frame data is a full green image, the step of discarding the first frame data comprises: in a case where the first frame data contains a preset number of continuous preset strings, analyze the first frame data to obtain analysis data corresponding to the first frame data; in a case where a color coding mode of the analysis data corresponding to the first frame data is YUV, convert the analysis data corresponding to the first frame data into target data; a color coding mode of the target data is RGB; in a case where a color value of green indicated by the target data is within a preset range, determine that a green screen detection result of the analysis data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image; in a case where a green screen detection result of the analysis data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image, discard the analysis data corresponding to the first frame data, and display prompt information; the prompt information is used to indicate a user that the display device is in a screen projection process; in a case where the first frame data does not contain a preset number of continuous preset strings, analyze the first frame data to obtain analysis data corresponding to the first frame data; perform audio and video synchronization operation on the analysis data corresponding to the first frame data to obtain second to-be-played data; render the second to-be-played data; control the display to play the second to-be-played data after rendering.

5. A display device, characterized by comprising: The display device comprises: a receiving module configured to receive video data sent by a screen projection device; a frame splitting module configured to perform frame splitting on the video data received by the receiving module to obtain a plurality of frame data; a frame discarding module configured to discard first frame data in the plurality of frame data obtained by the frame splitting module in a case where an image corresponding to the first frame data is a full green image; a display module configured to display second frame data in the plurality of frame data obtained by the frame splitting module, the second frame data being frame data in the plurality of frame data except the first frame data; in a case where the first frame data contains a preset number of continuous preset strings, the frame discarding module is further configured to analyze the first frame data to obtain analysis data corresponding to the first frame data; in a case where a color coding mode of the analysis data corresponding to the first frame data is YUV, the frame discarding module is further configured to convert the analysis data corresponding to the first frame data into target data; a color coding mode of the target data is RGB; In a case where the color value of green indicated by the target data is within a preset range, it is determined that the green screen detection result of the analysis data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image. In a case where the green screen detection result of the analysis data corresponding to the first frame data indicates that the image corresponding to the first frame data is a full green image, the analysis data corresponding to the first frame data is discarded, and prompt information is displayed; the prompt information is used to indicate that the display device is in a screen projection process. In a case where the first frame data does not contain a preset number of continuous preset strings, the first frame data is analyzed to obtain analysis data corresponding to the first frame data. The audio and video synchronization operation is performed on the analysis data corresponding to the first frame data to obtain second to-be-played data. The second to-be-played data is rendered. The display is controlled to play the second to-be-played data after rendering.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions in the computer readable storage medium are executed by the processor of the display device, the display device can execute the display method of claim 4.

Images