Error-resistant network transmission method for auxiliary stream based on deep learning video codec

Abstract

This application provides an error-resistant network transmission method for an auxiliary stream based on a deep learning video codec, relating to the field of video transmission technology. The decoding end uses an erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, resulting in a slightly distorted decoded reconstructed frame APn-1. The decoding end uses the decoded reconstructed frame APn-1 as a reference image and refreshes its decoding buffer. Encoding and decoding are performed with all reference content except the reference image set to None, resulting in a correctly decoded reconstructed frame APn. This method is a low-error network transmission method that reduces transmission bandwidth requirements while ensuring video reconstruction quality, thus enhancing the error-resistant robustness of the deep learning-based video codec.

Description

Technical Field

[0001] This application relates to the field of video transmission technology, and in particular to an error-resistant network transmission method for auxiliary streams based on deep learning video codecs. Background Technology

[0002] With the rapid development of artificial intelligence technology, it has become possible to comprehensively optimize video encoding and decoding systems using data-driven methods. Ensuring the accuracy of video data transmission during transmission is therefore particularly important. Currently, two new frame types have been proposed for error-resistant network transmission: Switching Intra-coded (SI) frames and Switching Predictive (SP) frames, to improve compression ratios while preventing errors from affecting subsequent frames. However, error-resistant network transmission protocols based on SP frames require buffering previously decoded frames, while protocols based on SI frames require transmitting more extra bits.

[0003] Compared to traditional coding standards, deep learning-based video codecs achieve higher compression rates while maintaining the same reconstruction quality. However, deep learning-based video codecs have extremely poor error robustness; if errors such as bit flipping or bit erasure occur during transmission, it will have a catastrophic impact on the decoding and reconstruction results.

[0004] Therefore, there is an urgent need for a new error-resistant transmission method for video codecs based on deep learning. Summary of the Invention

[0005] This application provides an error-resistant network transmission method for assisted streams based on deep learning video codecs to solve the aforementioned problems.

[0006] In a first aspect of this application, a method for error-resistant network transmission of assisted streams based on deep learning video codecs is proposed, the method comprising:

[0007] The decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, resulting in a slightly distorted decoded reconstructed frame APn-1.

[0008] The decoding end uses the decoded and reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the decoding end, and performs encoding and decoding with all reference content except the reference image set to None, so as to obtain the correct decoded and reconstructed frame APn.

[0009] In an optional embodiment of this application, before encoding and decoding using the erroneous bitstream at the decoding end, the method further includes:

[0010] The encoding end performs encoding and decoding to obtain four bit streams of the encoded original frame Pn-1, and sends them to the decoding end;

[0011] The decoding end verifies that the erroneous bitstream is one of the four bitstreams of the original frame Pn-1, and sends a NACK feedback message to inform the encoding end of the error and the erroneous bitstream.

[0012] In an optional embodiment of this application, after the encoding end receives the NACK feedback information from the decoding end, the method further includes:

[0013] The encoding end sets the erroneous bitstream to all zeros, and re-encodes and decodes the bitstream after setting it to zeros to obtain a slightly distorted encoded reconstruction frame APn-1.

[0014] The encoding end uses the slightly distorted encoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the encoding end, and performs encoding and decoding with all reference content except the reference image set to None, to obtain the correctly encoded original frame APn.

[0015] In one optional embodiment of this application, the encoding end and the decoding end respectively use a deep video codec model as a codec for encoding and decoding. The four bitstreams transmitted between the encoding end and the decoding end include: motion vector prior bitstream, motion vector bitstream, residual prior bitstream, and residual bitstream.

[0016] In one optional embodiment of this application, the method further includes:

[0017] When the erroneous bitstream is the residual prior bitstream, the decoding result corresponding to the residual prior bitstream is set to all zeros;

[0018] Residual decoding is performed using the decoding result set to all zeros to obtain a slightly distorted decoded reconstructed frame APn-1.

[0019] In one optional embodiment of this application, the method further includes:

[0020] The encoding end and the decoding end perform encoding and decoding based on the SI frame protocol respectively. When encoding and decoding the nth frame, the (n-2)th frame is not buffered, and the (n-1)th frame is used as a reference for encoding and decoding.

[0021] A second aspect of this application provides an auxiliary stream error-resistant network transmission device for deep learning-based video codecs, the device comprising:

[0022] The decoding and reconstruction module is used by the decoding end to encode and decode the erroneous bitstream. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros to obtain a slightly distorted decoded and reconstructed frame APn-1.

[0023] The correct decoding and reconstruction module is used by the decoding end to take the decoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the decoding end, and perform encoding and decoding with all reference content except the reference image set to None, so as to obtain the correct decoded and reconstructed frame APn.

[0024] In an optional embodiment of this application, the apparatus further includes, prior to the decoding and reconstruction module:

[0025] The raw frame bitstream acquisition module is used by the encoding end to perform encoding and decoding, obtain four bitstreams of the encoded raw frame Pn-1, and send them to the decoding end;

[0026] The confirmation module is used by the decoding end to verify that the erroneous bitstream is one of the four bitstreams of the original frame Pn-1, and to send NACK feedback information to the encoding end to inform it of the error and the erroneous bitstream.

[0027] In a third aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method for error-resistant network transmission of assisted stream based on a deep learning video codec as described in any of the first aspects above.

[0028] In a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, which, when executed by a processor, implements the error-resistant network transmission method for auxiliary streams based on a deep learning video codec as described in any of the first aspects above.

[0029] This application offers the following advantages: It provides an error-resistant network transmission method for an auxiliary stream based on a deep learning video codec. The decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, resulting in a slightly distorted decoded reconstructed frame APn-1. The decoding end uses the decoded reconstructed frame APn-1 as a reference image and refreshes its decoding buffer. Encoding and decoding are performed with all reference content except the reference image set to None, resulting in a correctly decoded reconstructed frame APn. This method is a low-error network transmission method that can adapt to dynamic transmission environments, reducing bandwidth requirements while ensuring video reconstruction quality, and enhancing the error-resistant robustness of the deep learning-based video codec. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of an existing technology that uses SI frames and SP frames for error compensation.

[0032] Figure 2 This is a schematic diagram illustrating the steps of an auxiliary stream error-resistant network transmission method based on a deep learning video codec, as provided in an embodiment of this application.

[0033] Figure 3 This is a flowchart illustrating an auxiliary stream error-resistant network transmission method based on a deep learning video codec, as provided in an embodiment of this application.

[0034] Figure 4 This is a schematic diagram illustrating the encoding and decoding process of an encoder in a depth video codec according to an embodiment of this application;

[0035] Figure 5 This is a schematic diagram illustrating the encoding and decoding process of a decoder in a depth video codec proposed in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of a refresh decoding buffer proposed in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram illustrating the acquisition of auxiliary frames when an error occurs, as proposed in an embodiment of this application.

[0038] Figure 8 This is a schematic diagram of the actual network transmission environment of the encoding / decoding end separation model proposed in this application;

[0039] Figure 9 This is a comparison diagram of a network transmission method without error-resistant transmission and a method using auxiliary frames, as proposed in this application.

[0040] Figure 10 This is a functional module diagram of an auxiliary stream error-resistant network transmission device based on a deep learning video codec, as proposed in an embodiment of this application.

[0041] Figure 11 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] With the rapid development of artificial intelligence technology, it has become possible to achieve comprehensive optimization of video encoding and decoding systems using data-driven methods. Ensuring the accuracy of video data transmission during transmission is therefore particularly important. Currently, two new frame types have been proposed for error-resistant network transmission: Switching Intra-coded (SI) frames and Switching Predictive (SP) frames. The encoding principle of SP frames is similar to that of P frames, but the reference frame is different. The encoding efficiency of SP frames is slightly lower than that of P frames, but still much higher than that of I frames. Furthermore, due to its excellent network affinity, SP frames are particularly suitable for transmission in wireless channel environments with high packet loss rates and significant noise interference. Unlike I frames, which use intra-coding throughout, SI frames only use intra-coding for segments that require switching frames. This improves the overall compression ratio of the video. SI frames can improve the compression ratio while avoiding errors affecting more subsequent frames. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of an existing technology that uses SI frames and SP frames for error compensation. Figure 1 (a) is a schematic diagram of the compensation method based on SI frames. Figure 1 (b) is a schematic diagram of the compensation method based on SP frames. During video stream transmission, if a P frame cannot be decoded correctly and the encoder receives an error report from the user, the encoder can send an auxiliary SP frame to prevent the error from affecting more subsequent frames. Similarly, the decoder can also achieve the same error recovery function by using an auxiliary SI frame. However, error-resistant network transmission protocols based on SP frames need to buffer previously decoded frames, while protocols based on SI frames need to transmit more extra bits.

[0044] Compared to traditional coding standards, deep learning-based video codecs achieve higher compression rates while maintaining the same reconstruction quality. However, deep learning-based video codecs have extremely poor error robustness; if errors such as bit flipping or bit erasure occur during transmission, it will have a catastrophic impact on the decoding and reconstruction results.

[0045] Based on this, this application proposes an error-resistant network transmission method for auxiliary streams based on deep learning video codecs, which improves robustness and accuracy of reconstruction results.

[0046] In a first aspect of this application, a method for error-resistant network transmission of assisted streams based on deep learning video codecs is provided, see [reference]. Figure 2 , Figure 2 This is a schematic diagram illustrating the steps of an auxiliary stream error-resistant network transmission method based on a deep learning video codec, provided in an embodiment of this application. The method includes the following steps:

[0047] Step 201: The decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros to obtain a slightly distorted decoded reconstruction frame APn-1.

[0048] Step 202: The decoding end uses the decoded and reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the decoding end, and performs encoding and decoding with all reference content except the reference image set to None, so as to obtain the correct decoded and reconstructed frame APn.

[0049] To clearly describe the error-resistant network transmission method for assisted streams based on deep learning video codecs proposed in this application, the following will be combined with... Figure 3 Provide a detailed explanation. Figure 3 This is a flowchart illustrating an auxiliary stream error-resistant network transmission method based on a deep learning video codec, as provided in an embodiment of this application.

[0050] In specific implementation step 201, during the process of the video codec detecting transmitted video frames, if an error is detected in transmitted video frame Pn-1, the decoding end uses the bitstream of the erroneous video frame Pn-1 for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, resulting in a slightly distorted decoded reconstructed frame APn-1. The aforementioned slightly distorted decoded reconstructed frame APn-1 is the erroneous decoded video frame. The core of the auxiliary stream error-resistant network transmission method proposed in this application lies in the synchronization of the reference frames of the decoding end and the encoding end. Once they are out of sync, catastrophic errors will occur. Since the distortion of the current erroneous frame is not severe, this application sets this frame as an AP frame (auxiliary frame) and uses it as a reference frame for subsequent encoding and decoding.

[0051] In specific implementation step 202, when the slightly distorted decoded and reconstructed frame APn-1 is used as the reference frame for encoding and decoding, the decoding end uses the decoded and reconstructed frame APn-1 as the reference image, refreshes the decoding buffer of the decoding end, and performs encoding and decoding with all reference content except the aforementioned reference image set to None, thereby obtaining the correctly decoded and reconstructed frame APn. In this application, the encoding end and the decoding end each include one of the aforementioned decoding buffers.

[0052] By using the slightly distorted decoded and reconstructed frame APn-1 as an auxiliary frame for encoding and decoding, and using this frame as a reference frame for the following code, buffer space is greatly saved. Furthermore, since the location of errors is randomly generated during video transmission, and the decoding result of the corresponding bitstream becomes random once an error occurs, this application directly sets the decoding result corresponding to the erroneous bitstream to all zeros when decoding the reconstructed frame APn-1. This allows for the most efficient synchronization between the encoding and decoding ends, eliminating the additional encoding overhead for the error location.

[0053] Furthermore, since the reference content used during transmission propagates through multiple frames, it also becomes distorted and corrupted when errors occur, requiring more bits to correct. Therefore, this application directly refreshes all reference content except the reference image, retaining only the reference image as a reference for future inter-frame coding, thus reducing the bits used to correct distortions.

[0054] In one optional embodiment of this application, before the decoding end uses the erroneous bitstream for encoding and decoding, the above method further includes the following process: the encoding end encodes and decodes the original frame Pn-1 to obtain four bitstreams of the encoded original frame Pn-1, and sends them to the decoding end; the decoding end verifies that the erroneous bitstream is one of the four bitstreams of the encoded original frame Pn-1, sends NACK (Negative Acknowledgement) feedback information, and informs the above-mentioned encoding end of the error and the erroneous bitstream.

[0055] In one optional embodiment of this application, after the encoding end receives the NACK feedback information from the decoding end, the method further includes: the encoding end sets the erroneous bitstream to all zeros, and re-encodes and decodes the bitstream after setting it to zeros to obtain a slightly distorted encoded reconstructed frame APn-1. The slightly distorted encoded reconstructed frame APn-1 is used as the erroneously encoded video frame and is used as an auxiliary frame for subsequent encoding and decoding to achieve synchronization of the reference frames between the encoding end and the decoding end; then, the encoding end uses the slightly distorted encoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the encoding end, and performs encoding and decoding with all reference content except the reference image set to None to obtain the correctly encoded original frame APn.

[0056] Since the distortion of the current erroneous frame is not severe, this frame is set as an AP frame and used as the reference image frame for the next frame. This method saves a lot of buffer space compared to the SP frame transmission method.

[0057] Furthermore, considering that smaller coding granularity can better control error propagation when the probability of transmission errors is low, and that deep video encoding / decoding models have higher compression ratios, the encoding end and decoding end used in this embodiment employ deep video encoding / decoding models as codecs for encoding and decoding. The deep video encoding / decoding model mainly includes the following processes: context-based conditional encoding / decoding, optical flow network estimation of motion vectors and completion of encoding / decoding, and context feature generation. The context-based conditional encoding / decoding process inputs the frame to be encoded. and context features To generate reconstructed frames and characteristics The optical flow network estimates motion vectors and performs encoding / decoding by inputting the frame to be encoded. and decoded frames To generate reconstructed motion vectors The input reconstructed motion vector is used during the generation of contextual features. Features of the reconstructed frame before propagation and decoded frames To generate contextual features To address the context feature generation process, the deep video encoding / decoding model introduces an offset prediction mechanism and a cross-group fusion module, further enhancing the propagation of contextual information within the network. Furthermore, the model proposes entropy coding based on quadtree partitioning, thereby estimating the probability distribution more accurately. This coding method offers finer coding granularity, better parallelism, and higher compression ratio compared to previous models.

[0058] For details on the encoding and decoding processes in the deep video encoding / decoding model, please refer to [link / reference]. Figure 4 and Figure 5 , Figure 4 This is a schematic diagram illustrating the encoding and decoding process of an encoder in a depth video codec according to an embodiment of this application. Figure 5 This is a schematic diagram illustrating the encoding and decoding process of the decoder in a depth video codec proposed in this application. The processing of video frames in the encoder mainly includes two steps: encoding and decoding. The decoder only needs to decode the original image from the received bitstream. Therefore, when describing a video codec, only the encoder is usually mentioned, and the decoder is a subset of the encoder.

[0059] The encoding and decoding process of the decoder in the aforementioned depth video codec is as follows: First, the previous decoded frame is extracted from the decoding buffer as the reference frame. and the current video frame to be encoded Together, motion estimation is performed to generate motion vectors. The encoded data is then encoded to obtain a motion vector bitstream. The encoder then decodes the encoded motion vector bitstream to obtain the motion vectors. Motion estimation results are generated by combining the reference frame with motion compensation. , and then With the frame to be encoded Operation To obtain the residual This is then encoded into a residual bitstream. The operation involves... This includes motion estimation and motion compensation operations, and context feature learning operations. Motion estimation and motion compensation operations are implemented using methods such as deep learning optical flow estimation. The context feature learning operation results in a bitstream containing both residual and residual prior information. Compared to traditional video coding standards, The subtraction operation results in a single, undivided residual bitstream. Further, decoding the residual bitstream yields the decoded residual. After that, proceed The result is obtained by inverse operation. Finally, post-processing is performed to obtain the decoded frame. (Right now Figure 4 Reconstructed frames in The reference features and reference motion vectors are then fed into the decoding buffer for use as a reference in the encoding of the next frame. In the depth video encoding and decoding model, reference features and reference motion vectors are also calculated and fed into the decoding buffer as a reference.

[0060] In the aforementioned deep video codec, the encoder's encoding and decoding process is as follows: First, the received motion vector bitstream is decoded to obtain motion vectors, and motion compensation is performed using a reference frame to obtain the motion compensation result. Then, the received residual bitstream is decoded to obtain the decoding residual, which, together with the motion compensation result, undergoes an inverse operation to obtain the final result. Finally, post-processing is performed to obtain the decoded frame. (Right now Figure 5 Reconstructed frames in ).

[0061] In traditional video encoders, the encoding process typically includes DCT transform, quantization, entropy coding, and reordering. The resulting bitstream is a single, indivisible entity, containing only motion vectors and residuals. However, in deep video coding models, the encoded and decoded bitstream comprises both the content from the traditional encoder and its prior information. The four bitstreams transmitted between the encoder and decoder in a deep video coding model are: a motion vector prior stream, a motion vector bitstream, a residual prior stream, and a residual bitstream. When a deep learning-based video codec decouples a block of bitstream into four streams, the coding granularity is reduced. Therefore, an error in one bitstream will not immediately have a significant impact.

[0062] In one optional embodiment of this application, see [reference] Figure 6 , Figure 6This is a schematic diagram of refreshing a decoding buffer according to an embodiment of this application. The reference content of the decoding buffer in the deep video coding model includes: a reference image, reference features, reference motion features, reference motion vectors, and reference residuals. In this embodiment, the decoding buffers used at both the encoding and decoding ends are decoding buffers from the deep video coding model, while traditional decoding buffers only contain reference images. The reference content other than the reference image is unique to the deep video coding model. This content performs secondary predictions on variables that need to be transmitted, such as motion vectors and residuals, thereby further improving the compression ratio. Since the reference content used during transmission propagates through multiple frames, when errors occur, the reference content also becomes erroneous and distorted. Correcting these distortions requires more bits. Therefore, in this application, the reference content other than the reference image is directly refreshed, retaining only the reference image as a reference for future inter-frame coding. Figure 1 Compared to the SI frame protocol in the past, the method proposed in this application corrects the distortion with fewer bits, reducing additional bit overhead.

[0063] In one optional embodiment of this application, the four bitstreams transmitted in the deep video coding model include: a motion vector prior stream, a motion vector stream, a residual prior stream, and a residual stream. See also... Figure 7 , Figure 7 This is a schematic diagram illustrating the acquisition of an auxiliary frame when an error occurs, as proposed in an embodiment of this application. When the erroneous bitstream is the residual beyond-priority bitstream among the four bitstreams, the erroneous bitstream is informed to the decoding end. The decoding end then uses the erroneous bitstream for normal decoding; that is, the decoding end sets the decoding result corresponding to the erroneous residual beyond-priority bitstream to all zeros, and decodes the remaining bitstreams normally. Then, residual decoding is performed using the decoded result set to all zeros to obtain a slightly distorted decoded reconstructed frame APn-1. If the erroneous bitstream is one of the other bitstreams, the decoding result of the erroneous bitstream is set to all zeros, and the next bitstream is decoded.

[0064] In one optional embodiment of this application, when encoding and decoding the nth frame at the encoding end, the (n-2)th frame is not buffered; encoding and decoding are performed only with the (n-1)th frame as a reference. When transmitting the nth frame, it is... Figure 1 Compared to the SP frame protocol in the past, the method proposed in this application does not require caching the previously decoded extra frames (the (n-2)th frame), which greatly saves cache space.

[0065] For example, this application embodiment uses a deep video codec model as the video codec. Packet loss experiments are conducted on one of the following streams: motion vector prior, motion vector, residual prior, and residual, while other streams are transmitted normally. Reconstruction quality is measured using PSNR (Peak Signal-to-Noise Ratio), and bandwidth usage is measured using Bits per Pixel (BPP). The changes in these two metrics after packet loss are observed between transmission methods without and with auxiliary frames. The experiment uses 1080p resolution videos from the UVG dataset, randomly cropped to... To obtain the experimental results, 318 repeated experiments were conducted for each quantization step size and each stream, based on the pixel resolution.

[0066] This experiment was conducted based on a simulation setup with separate encoding and decoding ends. The aforementioned packet loss refers to discarding the stream after an error occurs and replacing it with a completely zero-based stream. A schematic diagram illustrating the actual network transmission environment by separating the encoding and decoding ends is shown below. Figure 8 As shown ( Figure 8 This is a schematic diagram of the actual network transmission environment of the encoding / decoding end separation model proposed in this application.

[0067] Table 1 shows the changes in BPP and PSNR for each stream under different quantization step sizes, based on SI frame compensation. When an error occurs in frame 0, the SI frame compensation scheme uses I frames for encoding, which have a larger bit overhead. In contrast, the scheme proposed in this invention reduces the average PSNR by 0.0229% within 4 frames after the error, but reduces the average BPP increment by 60.53%, significantly reducing the number of extra bits used for compensation without having almost any impact on reconstruction quality. The BPP and PSNR outside the 4 frames after the error are basically consistent with those before the error occurred, indicating that the compensation is essentially complete, and therefore will not be elaborated further.

[0068] Table 1. Variations of BPP and PSNR for different flow losses under different quantization step sizes.

[0069]

[0070] For a comparison of transmission methods using non-error-resistant networks and transmission methods using auxiliary frames when using deep video coding models as video codecs, please refer to [link to relevant documentation]. Figure 9 , Figure 9This diagram compares a network transmission method without error-resistant transmission and a method using auxiliary frames, as proposed in this application. Frame number 0 indicates that an error occurred during frame transmission. When an error occurs, color blocks will appear on the image and spread rapidly until the entire image becomes undecodeable. That is, when the error-resistant network transmission method is not used, an error will have catastrophic consequences; the error will rapidly propagate across the video and eventually lead to complete undecoding. In contrast, the auxiliary frame compensation method can immediately correct the error and prevent its propagation.

[0071] The error-resistant network transmission method for auxiliary streams based on deep learning video codecs proposed in this application involves the decoding end using an erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, resulting in a slightly distorted decoded reconstructed frame APn-1. The decoding end uses the decoded reconstructed frame APn-1 as a reference image and refreshes its decoding buffer. Encoding and decoding are then performed with all reference content except the reference image set to None, resulting in a correctly decoded reconstructed frame APn. This method is a low-error network transmission method that can adapt to dynamic transmission environments, reducing bandwidth requirements while ensuring video reconstruction quality, and enhancing the error-resistant robustness of deep learning-based video codecs.

[0072] In a second aspect of this application, an auxiliary stream error-resistant network transmission device based on a deep learning video codec is provided, see [reference]. Figure 10 , Figure 10 This is a functional module diagram of an auxiliary stream error-resistant network transmission device based on a deep learning video codec, as proposed in an embodiment of this application. The device includes:

[0073] The decoding and reconstruction module 1001 is used by the decoding end to encode and decode the erroneous bitstream. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros to obtain a slightly distorted decoded and reconstructed frame APn-1.

[0074] The correct decoding and reconstruction module 1002 is used by the decoding end to take the decoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the decoding end, and perform encoding and decoding with all reference content except the reference image set to None, so as to obtain the correct decoded and reconstructed frame APn.

[0075] The device further includes the following components before the decoding and reconstruction module:

[0076] The raw frame bitstream acquisition module is used by the encoding end to perform encoding and decoding, obtain four bitstreams of the encoded raw frame Pn-1, and send them to the decoding end;

[0077] The confirmation module is used by the decoding end to verify that the erroneous bitstream is one of the four bitstreams of the original frame Pn-1, and to send NACK feedback information to the encoding end to inform it of the error and the erroneous bitstream.

[0078] The device further includes, after the confirmation module:

[0079] The encoding reconstruction module is used to set the erroneous bitstream to all zeros at the encoding end, and re-encode and decode the bitstream after setting it to zeros to obtain an encoded reconstruction frame APn-1 with slight distortion.

[0080] The correct encoding and reconstruction module is used by the encoding end to take the slightly distorted encoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the encoding end, and perform encoding and decoding with all reference content except the reference image set to None, so as to obtain the correctly encoded original frame APn.

[0081] The device further includes:

[0082] The codec confirmation module is used for encoding and decoding at the encoding end and the decoding end respectively using a deep video codec model as the codec. The four bitstreams transmitted between the encoding end and the decoding end include: motion vector prior bitstream, motion vector bitstream, residual prior bitstream, and residual bitstream.

[0083] The device further includes:

[0084] The bitstream zeroing module is used to set the decoding result corresponding to the residual beyond prior bitstream to all zeros when the bitstream that is erroneous is the residual beyond prior bitstream.

[0085] The residual decoding module is used to perform residual decoding using the decoding result set to all zeros, to obtain a slightly distorted decoded reconstructed frame APn-1.

[0086] The device further includes:

[0087] The encoding / decoding reference frame confirmation module is used for the encoding end and the decoding end to encode and decode based on the SI frame protocol respectively. When encoding and decoding the nth frame, the (n-2)th frame is not buffered, and the (n-1)th frame is used as a reference for encoding and decoding.

[0088] Based on the same concept, this application discloses an electronic device in a third aspect. Figure 11 A schematic diagram of an electronic device disclosed in an embodiment of this application is shown, such as... Figure 11As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory of the electronic device is not less than 12G, and the main frequency of the processor is not less than 2.4GHz. The memory 110 and the processor 120 are connected via a bus communication connection. The memory 110 stores a computer program, which can run on the processor 120 to implement the error-resistant network transmission method for auxiliary stream based on deep learning video codec disclosed in the embodiments of this application.

[0089] Based on the same concept, this application discloses a computer-readable storage medium storing a computer program / instructions thereon in a fourth aspect. When the computer program / instructions are executed by a processor, they implement an error-resistant network transmission method for an auxiliary stream based on a deep learning video codec disclosed in this application.

[0090] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0091] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0094] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0095] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0096] The above provides a detailed description of an error-resistant network transmission method for auxiliary streams based on a deep learning video codec. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for error-resistant network transmission of auxiliary streams based on deep learning video codecs, characterized in that, The method includes: The decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, resulting in a slightly distorted decoded reconstructed frame APn-1. The decoding end uses the decoded and reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the decoding end, and performs encoding and decoding with all reference content except the reference image set to None, so as to obtain the correct decoded and reconstructed frame APn.

2. The method for error-resistant network transmission of auxiliary streams based on deep learning video codecs according to claim 1, characterized in that, Before using the erroneous bitstream for encoding and decoding at the decoding end, the method further includes: The encoding end performs encoding and decoding to obtain four bit streams of the encoded original frame Pn-1, and sends them to the decoding end; The decoding end verifies that the erroneous bitstream is one of the four bitstreams of the original frame Pn-1, and sends a NACK feedback message to inform the encoding end of the error and the erroneous bitstream.

3. The method for error-resistant network transmission of auxiliary streams based on deep learning video codecs according to claim 2, characterized in that, After the encoding end receives the NACK feedback information from the decoding end, the method further includes: The encoding end sets the erroneous bitstream to all zeros, and re-encodes and decodes the bitstream after setting it to zeros to obtain a slightly distorted encoded reconstruction frame APn-1. The encoding end uses the slightly distorted encoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the encoding end, and performs encoding and decoding with all reference content except the reference image set to None, to obtain the correctly encoded original frame APn.

4. The method for error-resistant network transmission of auxiliary streams based on deep learning video codecs according to claim 1, characterized in that, The encoding end and the decoding end each use a deep video codec model as the codec for encoding and decoding. The four bitstreams transmitted between the encoding end and the decoding end include: motion vector prior bitstream, motion vector bitstream, residual prior bitstream, and residual bitstream.

5. The method for error-resistant network transmission of auxiliary streams based on deep learning video codecs according to claim 4, characterized in that, The method further includes: When the erroneous bitstream is the residual prior bitstream, the decoding result corresponding to the residual prior bitstream is set to all zeros; Residual decoding is performed using the decoding result set to all zeros to obtain a slightly distorted decoded reconstructed frame APn-1.

6. The method for error-resistant network transmission of auxiliary streams based on deep learning video codecs according to claim 1, characterized in that, The method further includes: The encoding end and the decoding end perform encoding and decoding based on the SI frame protocol respectively. When encoding and decoding the nth frame, the (n-2)th frame is not buffered, and the (n-1)th frame is used as a reference for encoding and decoding.

7. An auxiliary stream error-resistant network transmission device for deep learning-based video codecs, characterized in that, The device includes: The decoding and reconstruction module is used by the decoding end to encode and decode the erroneous bitstream. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros to obtain a slightly distorted decoded and reconstructed frame APn-1. The correct decoding and reconstruction module is used by the decoding end to take the decoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the decoding end, and perform encoding and decoding with all reference content except the reference image set to None, so as to obtain the correct decoded and reconstructed frame APn.

8. The auxiliary stream error-resistant network transmission device for deep learning-based video codecs according to claim 7, characterized in that, Prior to the decoding and reconstruction module, the device further includes: The raw frame bitstream acquisition module is used by the encoding end to perform encoding and decoding, obtain four bitstreams of the encoded raw frame Pn-1, and send them to the decoding end; The confirmation module is used by the decoding end to verify that the erroneous bitstream is one of the four bitstreams of the original frame Pn-1, and to send NACK feedback information to the encoding end to inform it of the error and the erroneous bitstream.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the error-resistant network transmission method for assisted streams based on deep learning video codecs, as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, It stores a computer program / instruction, which, when executed by a processor, implements the error-resistant network transmission method for auxiliary streams based on deep learning video codecs as described in any one of claims 1-6.

Images