Ultra-low time delay screen projection method and device based on Android system

Abstract

The invention discloses an ultra-low time delay screen projection method and device based on an Android system, and the method comprises the steps: a screen projection transmitting device creates a picture collector and an encoder, collects pictures in real time, and transmits the pictures to an encoder input queue for coding; polling an encoder output queue, taking out coded data and sending the coded data to screen projection receiving equipment; the screen projection receiving equipment sends the received NALU to a decoder input queue for decoding; the polling decoder outputs queue decoding data, and if the queue decoding data is a decoding configuration frame, a DRM buffer list used for displaying is created; and if the frame is a picture frame, putting the DRM buffer corresponding to the picture frame into an on-screen queue, taking out the DRM buffer by an on-screen thread, and outputting the DRM buffer. According to the method and the device, the control surface and the data surface of the projection screen application are separated, the data surface is realized through the Android service, a system interface is directly called to realize picture acquisition, coding, data transmission and picture and on-screen functions, the processing flow of each step of projection screen is optimized, and the projection screen time delay is reduced under the condition that the picture quality is not changed.

Description

Technical Field

[0001] The present invention relates to the field of screen projection technology, and in particular to an ultra-low latency screen projection method and device based on an Android system. Background Art

[0002] Compared with using the built-in screen projection software of electronic devices, wireless screen projectors do not require the installation of additional software or applications on the projection device. They are plug-and-play and do not require complicated settings. They also support multiple screen projection protocols and are highly secure. Therefore, they are still widely used in business presentations, education and training, video conferencing, game live streaming and other fields.

[0003] Existing wireless screen projectors are usually connected to the sending end, and the screen of the sending end is projected to the receiving end in two ways:

[0004] The first method is to call the interface provided by the Android SDK to collect images, capture the screen, call MediaCodec for encoding and decoding, and use the Surface object in the Android SDK to display the image on the screen. However, this method uses the Android SDK to make system calls, which has performance loss. When MediaCodec calls hard encoding and decoding, it relies on the system's encapsulation of the encoding and decoding interface, and cannot fully utilize the hardware performance, resulting in a higher delay in the projection of the screen.

[0005] The second method is system screen projection, which requires support for the DLNA standard. Because the DLNA standard uses multiple protocols, the projection screen data needs to be packaged and unpacked by multiple protocols, and the projection delay is relatively high, exceeding 100ms on average.

[0006] It can be seen that the technology used by the existing wireless screen projection devices cannot fully exert the hardware performance, and the delay of the screen projection is high. For some application scenarios that require real-time interaction with the projection screen, the user's screen projection experience is poor. Summary of the invention

[0007] In view of the above-mentioned defects, the technical problem to be solved by the present invention is to provide an ultra-low latency screen projection method and method based on the Android system, so as to solve the problem that the prior art cannot fully exert the hardware performance and the latency of the projection picture is high.

[0008] To this end, a first aspect of an embodiment of the present application provides an ultra-low latency screen projection method based on an Android system, which is implemented using a screen projection receiving device and a screen projection sending device, and includes the following steps:

[0009] After the screen projection receiving device is started, it automatically runs the screen projection service of the receiving end, creates the corresponding decoder according to the pre-configured decoder type, and listens for the connection signal of the screen projection sending device;

[0010] After the screen projection sending device is started, the screen projection service of the sending end will automatically run and wait for the input signal of the picture;

[0011] When the screen projection sending device detects an image input signal, it establishes a TCP Socket connection based on the pre-configured IP address and port of the screen projection receiving device;

[0012] The screen projection sending device creates a picture collector to obtain the input picture size and format, and creates a corresponding encoder according to the pre-configured encoder type; the picture collector continuously collects pictures and sends them to the input queue of the encoder, which encodes them in sequence;

[0013] The screen projection sending device polls the output queue of the encoder, takes out the encoded data in order, and sends NALUs in sequence through the established Socket connection;

[0014] The screen projection receiving device sends the received NALU to the input queue of the decoder for decoding in sequence;

[0015] The screen projection receiving device polls the output queue of the decoder and takes out the decoded data in order; if the decoded data is a decoding configuration frame, a DRM buffer list for screen display is created and the DRM buffer list is associated with the decoder output; if the decoded data is a picture frame, the DRM buffer corresponding to the picture frame is placed in the screen display queue;

[0016] The upper screen thread of the screen projection receiving device continuously takes out the DRM buffer from the upper screen queue and outputs it through the output port.

[0017] Based on the above technical solution, this application is implemented using a screen projection receiving device and a screen projection sending device to separate the control plane and data plane of the screen projection application. The data plane is implemented through Android services, and the system interface is directly called to realize picture acquisition, encoding, data transmission, as well as picture and screen functions, thereby optimizing the processing flow of each step of the screen projection and reducing the screen projection delay without changing the picture quality.

[0018] In the above technical solution, preferably, once the screen projection sending device detects that the picture input signal stops, it stops the picture collector and encoder, disconnects the Socket connection, and waits for the picture input signal to be input again.

[0019] In the above technical solution, preferably, once the screen projection receiving device detects that the Socket connection is disconnected, it clears the upper screen image, clears the DRM buffer list, and waits for the screen projection sending device to connect again.

[0020] In the above technical solution, preferably, the screen projection services of the screen projection sending device and the screen projection receiving device are both run with root permissions.

[0021] In the above technical solution, preferably, the screen projection service of the screen projection sending device directly calls the graphics card driver interface of the Android system to obtain HDMI or DP picture input.

[0022] A second aspect of the embodiments of the present application provides an adaptive video encoding and decoding device based on transmission rate change, including a screen projection receiving device and a screen projection sending device,

[0023] The screen projection sending device is provided with:

[0024] Encoders, created from pre-configured encoder types;

[0025] A collection and creation module is used to create a picture collector according to the detected picture input signal, and the picture collector continuously collects pictures and sends them to the input queue of the encoder, and the encoder encodes them in sequence;

[0026] The connection module is used to establish a TCP Socket connection and send a connection signal according to the pre-configured IP address and port of the screen projection receiving device when a picture input signal is detected;

[0027] A sending module, used to poll the output queue of the encoder, take out the encoded data in order, and send NALUs in sequence through the established Socket connection;

[0028] The screen projection receiving device is provided with:

[0029] Decoders, created from pre-configured decoder types;

[0030] A monitoring module is used to monitor the connection signal sent by the screen projection sending device, and report the IP address and port of the screen projection receiving device according to the connection signal;

[0031] A receiving module, used for sending the received NALU into the input queue of the decoder for decoding in sequence;

[0032] The on-screen thread is used to poll the output queue of the decoder and take out the decoded data in sequence; if the decoded data is a decoding configuration frame, a DRM buffer list for the on-screen is created, and the DRM buffer list is associated with the decoder output; if the decoded data is a picture frame, the DRM buffer corresponding to the picture frame is placed in the on-screen queue, and the DRM buffer is taken out from the on-screen queue in sequence and output through the output port.

[0033] In the above technical solution, preferably, once the screen projection sending device detects that the picture input signal stops, it stops the picture collector and the encoder, disconnects the Socket connection, and waits for the picture input signal to be input again.

[0034] In the above technical solution, preferably, once the screen projection receiving device detects that the Socket connection is disconnected, it clears the upper screen image, clears the DRM buffer list, and waits for the screen projection sending device to connect again.

[0035] In the above technical solution, preferably, according to the detected picture input signal, the V4L2 interface is called to create the picture collector.

[0036] In the above technical solution, preferably, the acquisition creation module directly calls the graphics card driver interface of the Android system to obtain HDMI or DP picture input.

[0037] It can be seen from the above technical solutions that the ultra-low latency screen projection method and device based on the Android system provided by the present invention solves the problem that the existing technology cannot fully exert the hardware performance and the latency of the screen projection is high. Compared with the existing technology, the present invention has the following beneficial effects:

[0038] In the present application scheme, the screen projection sending end and receiving end are respectively connected to the screen projection sending device and the main screen projection receiving device. The screen projection sending device establishes a Socket connection according to the picture input signal, and creates a picture collector to collect pictures, sends them to the input queue of the encoder, and sends NALUs to the screen projection receiving device in sequence; the screen projection receiving device sends the received NALUs to the input queue of the decoder for decoding in sequence. After taking out the decoded data, if the decoded data is a decoding configuration frame, a DRM buffer list for screen display is created, and the DRM buffer list is associated with the decoder output; if the decoded data is a picture frame, the DRM buffer corresponding to the picture frame is placed in the screen display queue and output through the output port. The control plane and data plane of the screen projection application are separated. The data plane is implemented through Android services, and the system interface is directly called to implement picture acquisition, encoding, data transmission, and picture and screen display functions, optimizing the processing flow of each step of screen projection, fully exerting the hardware performance, and reducing the screen projection delay without changing the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces and describes the drawings required for use in the embodiments of the present invention or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of an ultra-low latency screen projection method based on the Android system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the accompanying drawings of the embodiments of the present invention to clearly and completely describe the technical solutions of the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] In order to more clearly explain and illustrate the technical solution and implementation of the present invention, several preferred specific embodiments for implementing the technical solution of the present invention are introduced below.

[0043] It should be noted that the directional words such as "inside, outside", "front, back" and "left, right" in this article are expressed based on the product usage status. Obviously, the use of the corresponding directional words does not constitute a limitation on the protection scope of this scheme.

[0044] The screen projection sending device in this application is used to connect to the sending end (such as a laptop, mobile phone, PAD), and the screen projection receiving device is used to connect to the receiving end (such as a TV, large-screen device). The screen projection operation from the sending end to the receiving end is realized through the screen projection sending device and the screen projection receiving device. The sending end and the receiving end do not need to install the screen projection software, nor do they need to consider the screen projection protocol. It is plug-and-play and easy to use. In addition, the hardware performance can be fully utilized, and the screen projection screen has ultra-low latency. For some application scenarios that require real-time interaction with the screen projection screen, the user's screen projection experience is good.

[0045] The following uses the screen projection device connected by WIFI as an example.

[0046] like Figure 1 As shown, an ultra-low latency screen projection method based on the Android system provided in an embodiment of the present application is implemented by a screen projection receiving device and a screen projection sending device. The screen projection receiving device and the screen projection sending device are connected to the same WIFI and are located in the same local area network. The ultra-low latency screen projection method based on the Android system includes the following steps:

[0047] Step 110, after the screen projection receiving device is started, the screen projection service of the receiving end is automatically run, and a corresponding decoder is created according to the pre-configured decoder type. At the same time, the connection signal of the screen projection sending device is listened to according to the pre-configured port, and waits for the screen projection sending device to be connected.

[0048] Step 120, after the screen projection sending device is started, the screen projection service of the sending end is automatically run, waiting for the input signal of the picture.

[0049] In this step, the screen projection service of the screen projection sending device directly calls the graphics card driver interface of the Android system to obtain HDMI or DP picture input.

[0050] Step 130, when the screen projection sending device detects a picture input signal, it establishes a TCP Socket connection based on the pre-configured IP address and port of the screen projection receiving device.

[0051] It should be understood that during this process, the screen projection receiving device will first report its own IP address and port to the screen projection sending device.

[0052] In this application, the screen projection sending device has an HDMI or DP input interface, and polls the HDMI device status file in real time to detect whether HDMI is plugged in. When an external signal is inserted, for example, when a sending end such as a laptop or mobile phone is connected to the HDMI or DP input interface of the screen projection sending device via an HDMI or DP cable, the screen projection sending device will detect that there is a picture signal input signal.

[0053] Step 140: The screen projection sending device creates a picture collector, obtains the input picture size and format, and creates a corresponding encoder according to the pre-configured encoder type.

[0054] In this step, the screen projection sending device calls the V4L2 interface to create the picture collector according to the detected picture input signal.

[0055] The image collector continuously collects images and sends them to the input queue of the encoder in sequence, which is then encoded in sequence by the encoder.

[0056] Step 150, the screen projection sending device polls the output queue of the encoder, takes out the encoded data in sequence, and sends the NALUs in sequence through the established Socket connection.

[0057] NALU (Network Abstraction Layer Unit) is a network abstraction layer unit in H.264 encoding. It is the basic unit used to encapsulate and transmit video data in the H.264 video encoding standard. Each NALU contains a piece of encoded video data, which can be a complete frame or a part of a frame.

[0058] The screen projection data of this application is transmitted in NALU format, which avoids multiple protocol packet unpacking processing and can effectively reduce the overall delay of screen projection.

[0059] Step 160: The screen projection receiving device continuously collects NALUs and sends the NALUs to the input queue of the decoder created in step 110 for decoding.

[0060] Step 170: The screen projection receiving device polls the output queue of the decoder and takes out the decoded data in order. If the decoded data is a decoding configuration frame, a DRM buffer list for the screen is created and the DRM buffer list is associated with the decoder output; if the decoded data is a picture frame, the DRM buffer corresponding to the picture frame is placed in the screen queue.

[0061] Step 180, the upper screen thread of the screen projection receiving device continuously takes out the DRM (Linux graphics driver concept) buffer from the upper screen queue and outputs it through the output port.

[0062] The screen projection receiving device in this application is provided with an HDMI or DP output interface, which is connected to the HDMI or DP input interface of the receiving end, so that the output picture of the screen projection receiving device can be displayed on the screen of the receiving end, such as a smart TV or a smart large screen.

[0063] In the above process, once the screen projection sending device detects that the picture input signal stops, it stops the picture collector and encoder, disconnects the Socket connection, and waits for the picture input signal to be input again.

[0064] In the above process, once the screen projection receiving device detects that the Socket connection is disconnected, it clears the upper screen image, clears the DRM buffer list, and waits for the screen projection device to connect again.

[0065] In this application, the screen projection services of the screen projection sending device and the screen projection receiving device are run with root permissions to circumvent the Android security mechanism, so that shared memory can be used, thereby achieving zero copy and ultimately reducing the screen projection delay.

[0066] Based on the above method, the present application also provides an ultra-low latency screen projection device based on the Android system, including a screen projection receiving device and a screen projection sending device.

[0067] The screen projection sending device is provided with:

[0068] Encoders, created from pre-configured encoder types;

[0069] A collection and creation module is used to create a picture collector according to the detected picture input signal, and the picture collector continuously collects pictures and sends them to the input queue of the encoder, and the encoder encodes them in sequence;

[0070] The connection module is used to establish a TCP Socket connection and send a connection signal according to the pre-configured IP address and port of the screen projection receiving device when a picture input signal is detected;

[0071] A sending module, used to poll the output queue of the encoder, take out the encoded data in order, and send NALUs in sequence through the established Socket connection;

[0072] The screen projection receiving device is provided with:

[0073] Decoders, created from pre-configured decoder types;

[0074] A monitoring module is used to monitor the connection signal sent by the screen projection sending device, and report the IP address and port of the screen projection receiving device according to the connection signal;

[0075] A receiving module, used for sending the received NALU into the input queue of the decoder for decoding in sequence;

[0076] The on-screen thread is used to poll the output queue of the decoder and take out the decoded data in sequence; if the decoded data is a decoding configuration frame, a DRM buffer list for the on-screen is created, and the DRM buffer list is associated with the decoder output; if the decoded data is a picture frame, the DRM buffer corresponding to the picture frame is placed in the on-screen queue, and the DRM buffer is taken out from the on-screen queue in sequence and output through the output port.

[0077] In the above device, the encoder, acquisition creation module, connection module and sending module are managed and run by the screen projection service of the sending end; the decoder, monitoring module, receiving module and screen thread are managed and run by the screen projection service of the receiving end.

[0078] Based on the description of the above specific embodiments, the ultra-low latency screen projection method and device based on the Android system provided by the present invention has the following advantages compared with the prior art:

[0079] First, separate the control plane and data plane of the screen projection application. The data plane is implemented through Android services, and the system interface is directly called to implement image acquisition, encoding, data transmission, and image and screen functions. This can save Android service encapsulation loss and Android SDK call time, optimize the processing flow of each step of screen projection, and reduce the screen projection delay without changing the image quality.

[0080] Second, the data plane is implemented through Android services, which can avoid issues related to root permissions.

[0081] Third, the present invention uses Android native system services (i.e. the screen projection service of the receiving end and the screen projection service of the sending end mentioned above, which run automatically when the device is started) to implement the screen projection function. The screen projection service directly calls the system interface to implement screen projection picture acquisition, picture encoding, data transmission, picture decoding, and picture on the screen. There is no need to call the Android SDK for secondary encapsulation of the system interface, and the performance is better.

[0082] Fourth, hard coding and decoding directly call the system codec SDK to bring out the full performance of the hardware.

[0083] Fifth, the image encoding and decoding, image acquisition, and image on-screen interaction processes are optimized to avoid memory copying.

[0084] Sixth, Android SDK encapsulates the system interface as a MediaCodec interface. The MediaCodec implementation needs to be compatible with various application scenarios and is not optimized for extreme scenarios. For example, the MediaCodec call model uses a single-threaded encoding and decoding queue and dequeue, while the present invention separates the encoding and decoding queue and dequeue, so it can fully exert the hardware performance. For some application scenarios that require real-time interaction with the projection screen, ultra-low latency projection is achieved, and the user's projection experience is better.

[0085] Finally, it should be noted that the terms "comprises", "includes" or any other variants thereof as used herein are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0086] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone should be aware that any structural changes made under the inspiration of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention.

Claims

1. An ultra-low latency screen projection method based on the Android system, implemented by a screen projection receiving device and a screen projection sending device, characterized in that: The following steps are involved: After the screen projection receiving device is started, it automatically runs the screen projection service of the receiving end, creates the corresponding decoder according to the pre-configured decoder type, and listens for the connection signal of the screen projection sending device; After the screen projection sending device is started, the screen projection service of the sending end will automatically run and wait for the input signal of the picture; When the screen projection sending device detects an image input signal, it establishes a TCP Socket connection based on the pre-configured IP address and port of the screen projection receiving device; The screen projection sending device creates a picture collector to obtain the input picture size and format, and creates a corresponding encoder according to the pre-configured encoder type; The image collector continuously collects images and sends them to the input queue of the encoder, which encodes them in sequence. The screen projection sending device polls the output queue of the encoder, takes out the encoded data in order, and sends NALUs in sequence through the established Socket connection; The screen projection receiving device sends the received NALU to the input queue of the decoder for decoding in sequence; The screen projection receiving device polls the output queue of the decoder and retrieves the decoded data in sequence; If the decoded data is a decoding configuration frame, a DRM buffer list for on-screen is created and the DRM buffer list is associated with the decoder output; if the decoded data is a picture frame, the DRM buffer corresponding to the picture frame is placed in the on-screen queue; The upper screen thread of the screen projection receiving device continuously takes out the DRM buffer from the upper screen queue and outputs it through the output port.

2. The method according to claim 1, characterized in that Once the screen projection sending device detects that the picture input signal stops, it stops the picture collector and encoder, disconnects the Socket connection, and waits for the picture input signal to be input again.

3. The method according to claim 1, characterized in that Once the screen projection receiving device detects that the Socket connection is disconnected, it will clear the upper screen image, clear the DRM buffer list, and wait for the screen projection sending device to connect again.

4. The method according to claim 3, characterized in that: The screen casting services of both the screen casting sending device and the screen casting receiving device run with root permissions.

5. The method according to claim 2, characterized in that: The screen projection service of the screen projection sending device directly calls the graphics card driver interface of the Android system to obtain HDMI or DP image input.

6. An ultra-low latency screen projection device based on Android system, characterized in that: Including screen projection receiving device and screen projection sending device, The screen projection sending device is provided with: Encoders, created from pre-configured encoder types; A collection and creation module is used to create a picture collector according to the detected picture input signal, and the picture collector continuously collects pictures and sends them to the input queue of the encoder, and the encoder encodes them in sequence; The connection module is used to establish a TCP Socket connection and send a connection signal according to the pre-configured IP address and port of the screen projection receiving device when a picture input signal is detected; A sending module, used to poll the output queue of the encoder, take out the encoded data in order, and send NALUs in sequence through the established Socket connection; The screen projection receiving device is provided with: Decoders, created from pre-configured decoder types; A monitoring module is used to monitor the connection signal sent by the screen projection sending device, and report the IP address and port of the screen projection receiving device according to the connection signal; A receiving module, used for sending the received NALU into the input queue of the decoder for decoding in sequence; The on-screen thread is used to poll the output queue of the decoder and take out the decoded data in sequence; If the decoded data is a decoding configuration frame, a DRM buffer list for the upper screen is created and the DRM buffer list is associated with the decoder output; if the decoded data is a picture frame, the DRM buffer corresponding to the picture frame is placed in the upper screen queue, and the DRM buffer is taken out from the upper screen queue in turn and output through the output port.

7. The device according to claim 6, characterized in that Once the screen projection sending device detects that the picture input signal stops, it stops the picture collector and encoder, disconnects the Socket connection, and waits for the picture input signal to be input again.

8. The device according to claim 6, characterized in that Once the screen projection receiving device detects that the Socket connection is disconnected, it clears the upper screen image, clears the DRM buffer list, and waits for the screen projection device to connect again.

9. The device according to claim 6, characterized in that According to the detected picture input signal, the V4L2 interface is called to create the picture collector.

10. The device according to claim 6, characterized in that The acquisition creation module directly calls the graphics card driver interface of the Android system to obtain HDMI or DP picture input.

Images