Video transmission method, apparatus, device, storage medium, and program product

Abstract

This application relates to a video transmission method, apparatus, device, storage medium, and program product. The method includes: acquiring an original video frame to be transmitted; performing video encoding on the original video frame to obtain a first encoded video stream corresponding to the original video frame; in a video decoding pipeline, decoding and reconstructing the first encoded video stream and compressing the display frame after decoding and reconstructing the first encoded video stream to obtain a second encoded video stream; and sending the second encoded video stream to a display device. This method can reduce video transmission latency.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and in particular to a video transmission method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of display technology, 4K, 8K and even higher resolution display devices are becoming increasingly popular. At the same time, refresh rates of 120Hz, 240Hz and even higher have become standard features of display devices, which has led to an exponential increase in the data bandwidth required for video transmission.

[0003] In related technologies, in order to cope with the increased data bandwidth required for video transmission, the video to be transmitted can be encoded and decoded. Typically, video coding standards such as High Efficiency Video Coding (HEVC) and Open Media Consortium Video 1 (AV1) are used to significantly compress the original video data before transmission.

[0004] However, encoding and decoding the video to be transmitted involves lossy compression and is computationally complex. After the video is sent to the receiving display device, the display device needs a complete decoding process, which introduces significant processing delays and results in large video transmission delays. Summary of the Invention

[0005] Therefore, it is necessary to provide a video transmission method, apparatus, device, storage medium, and program product that can reduce video transmission latency in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides a video transmission method, including:

[0007] Obtain the raw video frames to be transmitted;

[0008] The original video frame is video encoded to obtain the first encoded video stream corresponding to the original video frame;

[0009] In the video decoding pipeline, the first encoded video stream is decoded and reconstructed, and the display frames after the first encoded video stream is decoded and reconstructed are compressed to obtain the second encoded video stream.

[0010] The second encoded video stream is sent to the display device.

[0011] In some embodiments, the step of video encoding the original video frame to obtain a first encoded video stream corresponding to the original video frame includes:

[0012] Obtain the buffer information corresponding to the encoder of the display stream compression, wherein the buffer information is used to indicate the buffer capacity of the buffer information;

[0013] Based on the buffer information, the image unit size used in the video encoding process is determined;

[0014] Based on the image unit size used in the video encoding process, the original video frame is video encoded to obtain the first encoded video stream corresponding to the original video frame.

[0015] In some embodiments, when there are multiple original video frames, the step of video encoding the original video frames to obtain a first encoded video stream corresponding to the original video frames includes:

[0016] During the video encoding process of the original video frames, the target frame is disabled to obtain the first encoded video stream that does not include the target frame. The target frame is used for bidirectional prediction of adjacent frames.

[0017] In some embodiments, the step of compressing the display frames after decoding and reconstructing the first encoded video stream to obtain a second encoded video stream includes:

[0018] Obtain the encoding complexity information of the first encoded video stream and the network transmission status information of the display device;

[0019] Based on the encoding complexity information and / or the network transmission status information, determine the compression parameters used for the display stream compression;

[0020] Based on the compression parameters, the display frames after decoding and reconstructing the first encoded video stream are compressed to obtain the second encoded video stream.

[0021] In some embodiments, the decoding and reconstruction of the first encoded video stream includes:

[0022] Obtain the video encoding type corresponding to the first encoded video stream;

[0023] Perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream;

[0024] Based on the frame type and video encoding type of each display frame, display frame prediction is performed to obtain the display frame after decoding and reconstructing the first encoded video stream.

[0025] Secondly, this application provides a video transmission device, comprising:

[0026] The acquisition module is used to acquire the raw video frames to be transmitted.

[0027] An encoding module is used to encode the original video frame to obtain a first encoded video stream corresponding to the original video frame;

[0028] The decoding and compression module is used to decode and reconstruct the first encoded video stream and compress the display frames after decoding and reconstructing the first encoded video stream in the video decoding pipeline to obtain the second encoded video stream.

[0029] A transmission module is used to send the second encoded video stream to a display device.

[0030] In some embodiments, the encoding module is specifically configured to obtain buffer information corresponding to the encoder of the display stream compression, the buffer information being used to indicate the buffer capacity of the buffer information; determine the image unit size used in the video encoding process based on the buffer information; and perform video encoding on the original video frame based on the image unit size used in the video encoding process to obtain a first encoded video stream corresponding to the original video frame.

[0031] In some embodiments, when there are multiple original video frames, the encoding module is specifically used to disable the target frame during the video encoding process of the original video frames to obtain the first encoded video stream excluding the target frame, wherein the target frame is used for bidirectional prediction of adjacent frames.

[0032] In some embodiments, the decoding and compression module is specifically used to obtain the encoding complexity information of the first encoded video stream and the network transmission status information of the display device; determine the compression parameters used for compressing the display stream based on the encoding complexity information and / or the network transmission status information; and compress the display frames after decoding and reconstructing the first encoded video stream according to the compression parameters to obtain the second encoded video stream.

[0033] In some embodiments, the decoding and compression module is specifically used to obtain the video encoding type corresponding to the first encoded video stream; perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream; and perform display frame prediction based on the frame type of each display frame and the video encoding type to obtain the display frames after decoding and reconstructing the first encoded video stream.

[0034] Thirdly, this application also provides a video transmission device, which includes: a video encoder, a video decoder, and a video transmission component. The video decoder is connected to the video encoder and the video transmission component, respectively, and a display stream compression encoder is coupled to the video decoder.

[0035] The video encoder is used to acquire the original video frame to be transmitted; and to encode the original video frame to obtain the first encoded video stream corresponding to the original video frame.

[0036] The video decoder is used to decode and reconstruct the first encoded video stream in the video decoding pipeline and to compress the display frames after decoding and reconstructing the first encoded video stream to obtain the second encoded video stream.

[0037] The video transmission component is used to send the second encoded video stream to the display device.

[0038] In some embodiments, the video encoder is specifically used to obtain buffer information corresponding to the encoder of the display stream compression, the buffer information being used to indicate the buffer capacity of the buffer information; determine the image unit size used in the video encoding process based on the buffer information; and perform video encoding on the original video frame based on the image unit size used in the video encoding process to obtain a first encoded video stream corresponding to the original video frame.

[0039] In some embodiments, when there are multiple original video frames, the video encoder is specifically used to disable the target frame during the video encoding process of the original video frames to obtain the first encoded video stream excluding the target frame, wherein the target frame is used for bidirectional prediction of adjacent frames.

[0040] In some embodiments, the video decoder is specifically configured to acquire encoding complexity information of the first encoded video stream and network transmission status information of the display device; determine compression parameters for compressing the display stream based on the encoding complexity information and / or the network transmission status information; and compress the display frames after decoding and reconstructing the first encoded video stream based on the compression parameters to obtain a second encoded video stream.

[0041] In some embodiments, the video decoder is specifically used to obtain the video encoding type corresponding to the first encoded video stream; perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream; and perform display frame prediction based on the frame type of each display frame and the video encoding type to obtain the display frames after decoding and reconstructing the first encoded video stream.

[0042] Fourthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the video transmission method of the first aspect described above.

[0043] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video transmission method of the first aspect described above.

[0044] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the video transmission method of the first aspect described above.

[0045] The aforementioned video transmission method, apparatus, device, storage medium, and program product acquire the original video frame to be transmitted; encode the original video frame to obtain a first encoded video stream corresponding to the original video frame; in a video decoding pipeline, decode and reconstruct the first encoded video stream and compress the display frame after decoding and reconstructing the first encoded video stream to obtain a second encoded video stream; and send the second encoded video stream to a display device. Because the video stream is decoded first and then the display frame after decoding and reconstruction is compressed in the same video decoding pipeline, it is equivalent to serializing the decoding and display stream compression processes, thereby avoiding the handling and buffering of full-size frame data between decoding and display stream compression on the internal bus, thus significantly shortening the end-to-end data transmission latency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 An application environment diagram of a video transmission method provided in an embodiment of this application;

[0048] Figure 2 A flowchart illustrating a video transmission method provided in an embodiment of this application;

[0049] Figure 3 A flowchart illustrating another video transmission method provided in an embodiment of this application;

[0050] Figure 4 A structural block diagram of a video transmission device provided in an embodiment of this application;

[0051] Figure 5 A structural block diagram of a video transmission device provided in an embodiment of this application;

[0052] Figure 6 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The relevant technologies will be explained below.

[0055] With the rapid development of display technology, 4K, 8K, and even higher resolution display devices are becoming increasingly common. Meanwhile, refresh rates of 120Hz, 240Hz, and even higher have become standard features in display devices. This has led to an exponential increase in the data bandwidth required for video transmission. For example, transmitting a single 8K, 120Hz uncompressed video stream requires more than 100 gigabits per second (Gbps) of bandwidth, far exceeding the physical capacity of current display interfaces.

[0056] In related technologies, to cope with the increased data bandwidth required for video transmission, the video to be transmitted can be encoded and decoded, typically using video encoding standards such as HEVC and AV1 to significantly compress the original video data before transmission. However, encoding and decoding the video to be transmitted is a lossy compression process and computationally complex. After the video is sent to the receiving display device, the display device needs to complete the decoding process, which introduces significant processing latency, resulting in a large video transmission delay. In applications that are extremely sensitive to latency, such as cloud gaming and VR / AR, even a delay of tens of milliseconds can lead to a severe deterioration in user experience.

[0057] Furthermore, Display Stream Compression (DSC) is a visually lossless compression standard typically integrated into the transmission link of the display interface, for example, by integrating a DSC encoder into the display controller of the Graphics Processing Unit (GPU). Accordingly, before the pixel data to be displayed is transmitted, the data in the frame buffer can be compressed by the DSC encoder and sent to the display device, where it is then decompressed by the timing controller. However, the DSC encoder compresses only the final, displayable pixel data and cannot be optimized in conjunction with the upstream video encoding / decoding process.

[0058] Therefore, relying solely on video encoding and decoding in related technologies cannot meet transmission latency requirements; and since DSC applications operate independently of the video encoding and decoding process, they cannot compress the encoded video stream. Consequently, video transmission latency is relatively high in these related technologies.

[0059] To address the aforementioned issues, embodiments of this application provide a video transmission method, apparatus, device, storage medium, and program product. Since the video stream is decoded first and then the display stream is compressed after decoding and reconstruction in the same video decoding pipeline, it is equivalent to serializing the decoding and display stream compression processes. This avoids the need to transport and buffer full-size frame data between decoding and display stream compression on the internal bus, thereby significantly shortening the end-to-end data transmission latency.

[0060] The application scenarios of the video transmission method provided in the embodiments of this application will be described below.

[0061] The video transmission method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, video transmitter 101 interacts with video receiver 102, which includes a display device. Video transmitter 101 can acquire the original video frame to be transmitted. Next, video transmitter 101 can encode the original video frame to obtain a first encoded video stream corresponding to the original video frame. Then, in a video decoding pipeline, video transmitter 101 can decode and reconstruct the first encoded video stream and compress the display frame after decoding and reconstructing the first encoded video stream to obtain a second encoded video stream. Finally, video transmitter 101 sends the second encoded video stream to the display device, which acts as the receiving device.

[0062] It should be noted that the video transmitter 101 and video receiver 102 described above can be any terminal. For example, the terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0063] In one exemplary embodiment, such as Figure 2 As shown, a video transmission method is provided, which is applied to... Figure 1 Taking the video sending end as an example, this video transmission method includes S201 to S204:

[0064] S201. Obtain the original video frames to be transmitted.

[0065] In some embodiments, the transmitting device may acquire the original video frame to be transmitted, process the original video frame, and then send it to the display device for display.

[0066] The original video frame can be any frame or multiple frames of the video to be displayed.

[0067] It should be understood that the embodiments of this application do not limit how the original video frames to be transmitted are obtained. In some embodiments, the original video frames may be pre-stored in the memory of the transmitting device, and the pre-stored original video frames can be retrieved from the memory after receiving a display command input by the user. In other embodiments, the transmitting device may receive the original video frames to be transmitted sent by an external device.

[0068] S202. Perform video encoding on the original video frame to obtain the first encoded video stream corresponding to the original video frame.

[0069] In this step, after the sending device obtains the original video frame to be transmitted, it can perform video encoding on the original video frame to obtain the first encoded video stream corresponding to the original video frame.

[0070] The aforementioned video encoding can be a process of prediction, transformation, quantization, and entropy coding of video frames. Video encoding can achieve high compression efficiency for video frames and is mainly used for video storage and network transmission. It should be noted that video encoding has high latency and is a lossy compression method; image quality loss is related to the compression ratio. For example, video encoding can include various video encoding types such as AVC1 and HEVC.

[0071] It should be understood that during the video encoding process of the original video frames, the way the video stream is divided can be restricted so that the generated first encoded video stream can be adapted to the subsequent display stream compression.

[0072] In some embodiments, the transmitting device can obtain buffer information corresponding to the encoder of the display stream compression. Subsequently, the transmitting device can determine the image unit size used in the video encoding process based on the buffer information. Finally, the transmitting device can perform video encoding on the original video frame based on the image unit size used in the video encoding process to obtain the first encoded video stream corresponding to the original video frame.

[0073] Among them, buffer information is used to indicate the buffer capacity. Image unit size is used to characterize the length and width pixel dimensions of the image units divided in each frame, such as 8×8 pixels, 16×16 pixels, 32×32 pixels, etc.

[0074] For example, the transmitting device can dynamically adjust the image unit size used in the video encoding process based on the buffer capacity of the encoder corresponding to the display stream compression. When the buffer capacity of the encoder is small, the image unit size can be adjusted to a smaller value, for example, 8×8 pixels or 16×16 pixels. When the buffer capacity of the encoder is large, the image unit size can be adjusted to a smaller value, for example, 32×32 pixels.

[0075] In this application, the image unit size of the display frame used in the video encoding process is dynamically adjusted according to the buffer information corresponding to the encoder of the display stream compression, so that the first encoded video stream can be adapted to the buffer capacity capability of the encoder of the display stream compression, thereby improving the subsequent display stream compression efficiency.

[0076] It should be understood that the embodiments of this application do not limit how the original video frames are encoded. In some embodiments, the original video frames can be encoded according to the above-mentioned image unit size by a video encoder in the transmitting device. For example, the above-mentioned video encoder may include an AV1 encoder, an HEVC encoder, etc.

[0077] In some embodiments, when there are multiple original video frames, the transmitting device disables the target frame during the video encoding process of the original video frames to obtain a first encoded video stream that does not include the target frame.

[0078] The target frame is used for bidirectional prediction of adjacent frames. For example, the target frame can be a bidirectional predicted (B) frame in video coding, where a B frame is a frame that performs bidirectional prediction by referring to both the previous and subsequent frames during the coding process.

[0079] It should be understood that since the target frame (B-frame) needs to wait for future frames before it can be encoded, this embodiment of the application disables the target frame (B-frame), thereby eliminating the need to wait for future frames and directly reducing latency in the video encoding process. Here, the future frame refers to the frame following the B-frame used as a reference for B-frame encoding.

[0080] S203. In the video decoding pipeline, the first encoded video stream is decoded and reconstructed, and the display frames after the first encoded video stream is decoded and reconstructed are compressed to obtain the second encoded video stream.

[0081] In this step, after the sending device encodes the original video frame to obtain the first encoded video stream corresponding to the original video frame, it can decode and reconstruct the first encoded video stream in the video decoding pipeline, and compress the display stream of the display frame after decoding and reconstructing the first encoded video stream to obtain the second encoded video stream.

[0082] The video decoding pipeline is a continuous and ordered processing path within the processor that processes the encoded video stream step by step. Because decoding reconstruction and display stream compression are performed in the video decoding pipeline, the full-size frame data between decoding reconstruction and display stream compression on the internal bus does not require additional handling and buffering, thus significantly reducing end-to-end display latency.

[0083] It should be understood that the embodiments of this application do not limit how the first encoded video stream is decoded and reconstructed. In some embodiments, the transmitting device can obtain the video encoding type of the original video frames. Secondly, the transmitting device can perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream. Finally, the transmitting device performs display frame prediction based on the frame type and video encoding type of each display frame to obtain the display frames after decoding and reconstructing the first encoded video stream.

[0084] The video encoding type can include AVC1, HEVC, etc. Frame types can include intra-coded (I) frames, predictive (P) frames, etc. I-frames are frames encoded independently of any other frames, while P-frames are frames predicted by referencing previous frames during the encoding process. Display frame prediction can include intra-frame prediction and inter-frame prediction.

[0085] For example, the decoder in the transmitting device can first obtain the video encoding type of the original video frame. Then, the decoder can perform bitstream parsing on the first encoded video stream, obtain the frame header information of each display frame, and determine the frame type from the frame header information. Finally, for I-frames, intra-frame prediction can be performed based on the video encoding type to obtain the display frame after decoding and reconstructing the first encoded video stream; for P-frames, inter-frame prediction can be performed based on the video encoding type to obtain the display frame after decoding and reconstructing the first encoded video stream.

[0086] It should be understood that the embodiments of this application do not limit how the display frames after decoding and reconstructing the first encoded video stream are compressed to obtain the second encoded video stream. In some embodiments, the sending device may first obtain the encoding complexity information of the first encoded video stream and the network transmission status information of the display device. Subsequently, the sending device may determine the compression parameters to be used for display stream compression based on the encoding complexity information and / or the network transmission status information. Finally, the sending device may compress the display frames after decoding and reconstructing the first encoded video stream according to the compression parameters to obtain the second encoded video stream.

[0087] The compression parameters for display stream compression include bitrate and / or slice width parameters. By adjusting the bitrate and / or slice width parameters, the compression level of display frames in the second coded video stream can be adjusted, thereby ensuring bandwidth stability.

[0088] For example, encoding complexity information is used to characterize the encoding complexity of the first encoded video stream, and this encoding complexity information can be mapped to the video encoding type. That is, the encoding complexity information corresponding to each video encoding type can be pre-configured, and by identifying the video encoding type, the encoding complexity information of the corresponding original video frame can be determined.

[0089] For example, network transmission status information is used to characterize the network transmission status between the sending device and the receiving device (i.e., the display device). This network transmission status information may include network congestion information, network transmission rate, etc., which are not limited in this embodiment.

[0090] For example, the sending device can sense encoding complexity information and network transmission status information, thereby dynamically adjusting the compression parameters used for display stream compression. When the encoding complexity is high and / or the network transmission status is poor, the compression level of the display stream can be increased to maintain a stable total bandwidth; when the encoding complexity is low or the network transmission status is good, the compression level of the display stream can be decreased to maintain a stable total bandwidth. Based on this, the receiving device dynamically and intelligently configures the compression parameters of the display stream compression based on the sensed encoding complexity information and network transmission status information, thereby achieving system-level bandwidth and quality optimization.

[0091] Display stream compression can be performed using a display stream compression algorithm, and this application does not impose any restrictions on this.

[0092] It should be noted that the decoding and reconstruction of the first encoded video stream can be performed by the video decoder in the transmitting device. The video decoder is coupled with a display stream compression encoder so that after the video decoding is completed in the video decoding pipeline, the display stream compression encoder performs display stream compression on the display frames after the decoding and reconstruction of the first encoded video stream.

[0093] In this embodiment, the video stream data does not need to be moved or cached after the video decoder restores the original image. Instead, it is directly compressed into a second encoded video stream for display, and then sent to the display device. Since display stream compression is visually lossless and has extremely low decoding latency (typically less than one line scan time), placing display stream compression after video decoding avoids writing the full-size reconstructed frame data to the bandwidth-constrained display interface, thereby greatly reducing latency during data transmission. Simultaneously, it saves the reuse of DSC's encoding and decoding capabilities, reducing additional encoding requirements and lowering GPU costs.

[0094] S204. Send the second encoded video stream to the display device.

[0095] In this step, when the sending device decodes and reconstructs the first encoded video stream and compresses the display frames after decoding and reconstructing the first encoded video stream in the video decoding pipeline to obtain the second encoded video stream, it can send the second encoded video stream to the display device, which is the receiving device.

[0096] In some embodiments, the display device may transmit the second encoded video stream to the display device via a physical transmission link.

[0097] The physical transmission links may include DisplayPort, High-Definition Multimedia Interface (HDMI), and Universal Serial Bus Generation 4 (USB4).

[0098] In some embodiments, the display device includes a display interface controller, a DSC decoder, and a display screen. After receiving the second encoded video stream through the display interface controller, the display device can perform DSC decoding on the second encoded video stream using the DSC decoder, and the display screen can then display the video based on the decoded video stream. Since decoding of the display stream compression is more efficient than video encoding, data transmission latency can be greatly reduced.

[0099] The video transmission method provided in this application serially connects high-latency video encoding / decoding with low-latency display stream compression at the transmitting end, instead of connecting them independently in parallel. This avoids the need for transporting and buffering full-size video frame data on the internal bus of the transmitting end, effectively shortening the data processing path and significantly reducing end-to-end display latency. This method is particularly suitable for interactive applications such as cloud gaming and virtual reality, which are extremely sensitive to latency. Simultaneously, it optimizes system hardware costs and architecture, significantly reducing chip design costs through encoders, memory reuse, and other methods.

[0100] The video transmission method provided in this application involves: acquiring an original video frame to be transmitted; performing video encoding on the original video frame to obtain a first encoded video stream corresponding to the original video frame; in a video decoding pipeline, decoding and reconstructing the first encoded video stream and compressing the display frame after decoding and reconstructing the first encoded video stream to obtain a second encoded video stream; and sending the second encoded video stream to a display device. Since the video stream is decoded first and then the display frame after decoding and reconstruction is compressed in the same video decoding pipeline, it is equivalent to serializing the decoding and display stream compression processes, thereby avoiding the handling and buffering of full-size frame data between decoding and display stream compression on the internal bus, thus significantly shortening the end-to-end data transmission latency.

[0101] In one exemplary embodiment, such as Figure 3 As shown, another video transmission method is provided, which can be applied to... Figure 1 Taking the video sending end as an example, this video transmission method includes S301 to S311:

[0102] S301. Obtain the raw video frames to be transmitted.

[0103] S302. Obtain the buffer information corresponding to the encoder of the display stream compression.

[0104] Among them, the buffer information is used to indicate the buffer capacity of the buffer information.

[0105] S303. Based on the buffer information, determine the size of the image unit used in the video encoding process.

[0106] S304. During the video encoding process of the original video frame, the target frame is disabled and the original video frame is video encoded according to the image unit size used in the video encoding process to obtain a first encoded video stream that does not include the target frame.

[0107] The target frame is used for bidirectional prediction of adjacent frames.

[0108] S305. Obtain the video encoding type corresponding to the first encoded video stream.

[0109] S306. Perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream.

[0110] S307. Based on the frame type and video encoding type of each display frame, perform display frame prediction to obtain the display frames after decoding and reconstructing the first encoded video stream.

[0111] S308. Obtain the encoding complexity information of the first encoded video stream and the network transmission status information of the display device.

[0112] S309. Determine the compression parameters to be used for display stream compression based on the encoding complexity information and / or network transmission status information.

[0113] S310. In the video decoding pipeline, the display frames after decoding and reconstructing the first encoded video stream are compressed according to the compression parameters to obtain the second encoded video stream.

[0114] S311. Send the second encoded video stream to the display device.

[0115] The video transmission method provided in this application involves: acquiring an original video frame to be transmitted; performing video encoding on the original video frame to obtain a first encoded video stream corresponding to the original video frame; in a video decoding pipeline, decoding and reconstructing the first encoded video stream and compressing the display frame after decoding and reconstructing the first encoded video stream to obtain a second encoded video stream; and sending the second encoded video stream to a display device. Since the video stream is decoded first and then the display frame after decoding and reconstruction is compressed in the same video decoding pipeline, it is equivalent to serializing the decoding and display stream compression processes, thereby avoiding the handling and buffering of full-size frame data between decoding and display stream compression on the internal bus, thus significantly shortening the end-to-end data transmission latency.

[0116] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0117] Based on the same inventive concept, this application also provides a video transmission apparatus for implementing the video transmission method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more video transmission apparatus embodiments provided below can be found in the limitations of the video transmission method described above, and will not be repeated here.

[0118] In one exemplary embodiment, such as Figure 4 As shown, a video transmission device 400 is provided, including: an acquisition module 401, an encoding module 402, a decoding and compression module 403, and a transmission module 404, wherein:

[0119] The acquisition module 401 is used to acquire the original video frames to be transmitted.

[0120] The encoding module 402 is used to encode the original video frame to obtain the first encoded video stream corresponding to the original video frame.

[0121] The decoding and compression module 403 is used to decode and reconstruct the first encoded video stream and compress the display frames after decoding and reconstructing the first encoded video stream in the video decoding pipeline to obtain the second encoded video stream.

[0122] The transmission module 404 is used to send the second encoded video stream to the display device.

[0123] In some embodiments, the encoding module 402 is specifically used to obtain buffer information corresponding to the encoder of the display stream compression, the buffer information being used to indicate the buffer capacity of the buffer information; determine the image unit size used in the video encoding process based on the buffer information; and perform video encoding on the original video frame based on the image unit size used in the video encoding process to obtain the first encoded video stream corresponding to the original video frame.

[0124] In some embodiments, when there are multiple original video frames, the encoding module 402 is specifically used to disable the target frame during the video encoding process of the original video frames to obtain a first encoded video stream that does not include the target frame, wherein the target frame is used for bidirectional prediction of adjacent frames.

[0125] In some embodiments, the decoding and compression module 403 is specifically used to obtain the encoding complexity information of the first encoded video stream and the network transmission status information of the display device; determine the compression parameters to be used for display stream compression based on the encoding complexity information and / or network transmission status information; and compress the display frames after decoding and reconstructing the first encoded video stream according to the compression parameters to obtain the second encoded video stream.

[0126] In some embodiments, the decoding compression module 403 is specifically used to obtain the video encoding type corresponding to the first encoded video stream; perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream; and perform display frame prediction based on the frame type and video encoding type of each display frame to obtain the display frames after decoding and reconstructing the first encoded video stream.

[0127] Each module in the aforementioned video transmission device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0128] In one exemplary embodiment, such as Figure 5 As shown, a video transmission device 500 is provided, including: a video encoder 501, a video decoder 502 and a video transmission component 503. The video decoder 502 is connected to the video encoder 501 and the video transmission component 503 respectively, and a display stream compression encoder is coupled in the video decoder 502.

[0129] The video encoder 501 is used to acquire the original video frame to be transmitted; and to encode the original video frame to obtain the first encoded video stream corresponding to the original video frame.

[0130] The video decoder 502 is used in the video decoding pipeline to decode and reconstruct the first encoded video stream and compress the display frames after decoding and reconstructing the first encoded video stream to obtain the second encoded video stream.

[0131] The video transmission component 503 is used to send the second encoded video stream to the display device.

[0132] In some embodiments, the video encoder 501 is specifically used to obtain buffer information corresponding to the encoder of the display stream compression, the buffer information being used to indicate the buffer capacity of the buffer information; determine the image unit size used in the video encoding process based on the buffer information; and perform video encoding on the original video frame based on the image unit size used in the video encoding process to obtain the first encoded video stream corresponding to the original video frame.

[0133] In some embodiments, when there are multiple original video frames, the video encoder 501 is specifically used to disable the target frame during the video encoding process of the original video frames to obtain a first encoded video stream that does not include the target frame, wherein the target frame is used for bidirectional prediction of adjacent frames.

[0134] In some embodiments, the video decoder 502 is specifically used to obtain encoding complexity information of the first encoded video stream and network transmission status information of the display device; determine compression parameters for compressing the display stream based on the encoding complexity information and / or network transmission status information; and compress the display frames after decoding and reconstructing the first encoded video stream according to the compression parameters to obtain the second encoded video stream.

[0135] In some embodiments, the video decoder 502 is specifically used to obtain the video encoding type corresponding to the first encoded video stream; perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream; and perform display frame prediction based on the frame type and video encoding type of each display frame to obtain the display frames after decoding and reconstructing the first encoded video stream.

[0136] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a video transmission method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0137] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the video transmission method described above.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video transmission method described above.

[0140] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the video transmission method described above.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0144] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video transmission method, characterized in that, The method includes: Obtain the raw video frames to be transmitted; The original video frame is video encoded to obtain the first encoded video stream corresponding to the original video frame; In the video decoding pipeline, the first encoded video stream is decoded and reconstructed, and the display frames after the first encoded video stream is decoded and reconstructed are compressed to obtain the second encoded video stream. The second encoded video stream is sent to the display device.

2. The method according to claim 1, characterized in that, The step of encoding the original video frame to obtain the first encoded video stream corresponding to the original video frame includes: Obtain the buffer information corresponding to the encoder of the display stream compression, wherein the buffer information is used to indicate the buffer capacity of the buffer information; Based on the buffer information, the image unit size used in the video encoding process is determined; Based on the image unit size used in the video encoding process, the original video frame is video encoded to obtain the first encoded video stream corresponding to the original video frame.

3. The method according to claim 1, characterized in that, When there are multiple original video frames, the step of encoding the original video frames to obtain the first encoded video stream corresponding to the original video frames includes: During the video encoding process of the original video frames, the target frame is disabled to obtain the first encoded video stream that does not include the target frame. The target frame is used for bidirectional prediction of adjacent frames.

4. The method according to claim 1, characterized in that, The step of compressing the display frames after decoding and reconstructing the first encoded video stream to obtain the second encoded video stream includes: Obtain the encoding complexity information of the first encoded video stream and the network transmission status information of the display device; Based on the encoding complexity information and / or the network transmission status information, determine the compression parameters used for the display stream compression; Based on the compression parameters, the display frames after decoding and reconstructing the first encoded video stream are compressed to obtain the second encoded video stream.

5. The method according to claim 1, characterized in that, The decoding and reconstruction of the first encoded video stream includes: Obtain the video encoding type corresponding to the first encoded video stream; Perform bitstream parsing on the first encoded video stream to determine the frame type of each display frame in the first encoded video stream; Based on the frame type and video encoding type of each display frame, display frame prediction is performed to obtain the display frame after decoding and reconstructing the first encoded video stream.

6. A video transmission device, characterized in that, The device includes: The acquisition module is used to acquire the raw video frames to be transmitted. An encoding module is used to encode the original video frame to obtain a first encoded video stream corresponding to the original video frame; The decoding and compression module is used to decode and reconstruct the first encoded video stream and compress the display frames after decoding and reconstructing the first encoded video stream in the video decoding pipeline to obtain the second encoded video stream. A transmission module is used to send the second encoded video stream to a display device.

7. A video transmission device, characterized in that, The video transmission device includes: a video encoder, a video decoder, and a video transmission component. The video decoder is connected to the video encoder and the video transmission component, respectively. A display stream compression encoder is coupled to the video decoder. The video encoder is used to acquire the original video frame to be transmitted; and to encode the original video frame to obtain the first encoded video stream corresponding to the original video frame. The video decoder is used to decode and reconstruct the first encoded video stream in the video decoding pipeline and to compress the display frames after decoding and reconstructing the first encoded video stream to obtain the second encoded video stream. The video transmission component is used to send the second encoded video stream to the display device.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images