Auxiliary stream anti-error code network transmission method for video codec based on deep learning

Abstract

The invention provides an auxiliary stream anti-error code network transmission method for a video codec based on deep learning, and relates to the technical field of video transmission, a decoding end performs coding and decoding by using an error code stream, and in the coding and decoding process, a decoding result corresponding to the error code stream is set to be all zero to obtain a slightly distorted decoding reconstruction frame APn-1; and the decoding end takes the decoded reconstructed frame APn-1 as a reference image, refreshes a decoding buffer of the decoding end, and performs coding and decoding under the condition that all reference contents except the reference image are set as None, so as to obtain a correctly decoded reconstructed frame APn. The method is a low-error network transmission method, the transmission bandwidth requirement is reduced while the video reconstruction quality is ensured, and the error code resistance robustness of the video codec based on deep learning is enhanced.

Description

Technical Field

[0001] The present application relates to the field of video transmission technology, and in particular to an auxiliary stream error-resistant network transmission method based on a deep learning video codec. Background Art

[0002] With the rapid development of artificial intelligence technology, it has become possible to use data-driven methods to achieve comprehensive optimization of video encoding and decoding systems. How to ensure the accuracy of video data transmission during the transmission process is particularly important. Currently, two new frame types are proposed in error-resistant network transmission: Switching Intra-coded (SI) frames and Switching Predictive (SP) frames to improve the compression rate while avoiding the impact of error codes on more subsequent frames. However, the error-resistant network transmission protocol based on SP frames needs to cache previously decoded frames, and the protocol based on SI frames needs to transmit more additional bits.

[0003] Compared with traditional coding standards, deep learning-based video codecs have higher compression rates at the same reconstruction quality. However, deep learning-based video codecs have very poor error robustness. Once errors such as bit flips or bit erasures occur during transmission, they will have a catastrophic impact on the decoding and reconstruction results.

[0004] Based on this, a new deep learning-based video codec error-resistant transmission method is urgently needed. Summary of the invention

[0005] The present application provides an auxiliary stream error-resistant network transmission method based on deep learning video codec to solve the above-mentioned existing problems.

[0006] In a first aspect of an embodiment of the present application, a method for error-resistant network transmission of an auxiliary stream based on a deep learning video codec is proposed, the method comprising: The decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, thereby obtaining a decoded reconstructed frame APn-1 with slight distortion. The decoding end uses the decoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the decoding end, and performs encoding and decoding with all reference contents except the reference image set to None, thereby obtaining a correctly decoded reconstructed frame APn.

[0007] In an optional embodiment of the present application, before the decoding end uses the erroneous bit stream for encoding and decoding, the method further includes: The encoder performs encoding and decoding to obtain four code streams of the encoded original frame Pn-1 and sends them to the decoder; The decoding end confirms through checking that the erroneous code stream is one of the four code streams encoding the original frame Pn-1, and sends NACK feedback information to inform the encoding end of the error generation and the erroneous code stream.

[0008] In an optional embodiment of the present application, after the encoding end receives NACK feedback information from the decoding end, the method further includes: The encoding end sets the erroneous code stream to all zeros, and re-encodes and decodes the code stream after setting to zero, to obtain a coded reconstructed frame APn-1 with slight distortion; The encoding end uses the slightly distorted encoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the encoding end, sets all reference contents except the reference image to None for encoding and decoding, and obtains the correctly encoded original frame APn.

[0009] In an optional embodiment of the present application, the encoding end and the decoding end respectively use a deep video codec model as a codec for encoding and decoding, and the four code streams transmitted between the encoding end and the decoding end include: a motion vector super-prior code stream, a motion vector code stream, a residual super-prior code stream, and a residual code stream.

[0010] In an optional embodiment of the present application, the method further includes: When the erroneous bitstream is the residual super a priori bitstream, setting the corresponding decoding result of the residual super a priori bitstream to all zeros; Residual decoding is performed using the decoding result set to all zeros to obtain a decoded reconstructed frame APn-1 with slight distortion.

[0011] In an optional embodiment of the present application, the method further includes: The encoding end and the decoding end perform encoding and decoding based on the SI frame protocol respectively. When encoding and decoding the nth frame, the n-2th frame is not cached, and encoding and decoding is performed only with reference to the n-1th frame.

[0012] In a second aspect of an embodiment of the present application, a device for network transmission of auxiliary streams with robust error tolerance based on a deep learning video codec is provided, the device comprising: A decoding and reconstruction module is used for the decoding end to use the erroneous bit stream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bit stream is set to all zeros to obtain a slightly distorted decoded and reconstructed frame APn-1; The correct decoding and reconstruction module is used for the decoding end to use the decoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the decoding end, and perform encoding and decoding with all reference contents except the reference image set to None to obtain the correctly decoded and reconstructed frame APn.

[0013] In an optional embodiment of the present application, before decoding the reconstruction module, the device further includes: The original frame code stream acquisition module is used for encoding and decoding at the encoding end to obtain four code streams of the encoded original frame Pn-1 and send them to the decoding end; The confirmation module is used for the decoding end to confirm through verification that the erroneous code stream is one of the four code streams encoding the original frame Pn-1, and send NACK feedback information to inform the encoding end of the error generation and the erroneous code stream.

[0014] In a third aspect of an embodiment of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the auxiliary stream error-resistant network transmission method based on a deep learning video codec as described in any one of the first aspects above.

[0015] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is proposed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the auxiliary stream error-resistant network transmission method based on a deep learning video codec described in any one of the first aspects above is implemented.

[0016] The present application includes the following advantages: The present application provides an auxiliary stream error-resistant network transmission method for deep learning video codecs, the decoding end uses the erroneous bitstream for encoding and decoding, and in the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros to obtain a slightly distorted decoded reconstructed frame APn-1; the decoding end uses the decoded reconstructed frame APn-1 as a reference image, and refreshes the decoding buffer of the decoding end, and sets all reference contents except the reference image to None for encoding and decoding, and obtains a correctly decoded reconstructed frame APn. This method is a low-error network transmission method that can adapt to dynamic transmission environments, reduce the bandwidth requirements for transmission while ensuring the quality of video reconstruction, and enhance the error-resistant robustness of the deep learning-based video codec. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 It is a schematic diagram of using SI frame and SP frame for error compensation proposed in the prior art; Figure 2It is a schematic diagram of the steps of a method for auxiliary stream error-resistant network transmission based on deep learning video codec provided by an embodiment of the present application; Figure 3 It is a flowchart of a method for auxiliary stream error-resistant network transmission based on deep learning video codec provided by an embodiment of the present application; Figure 4 It is a schematic diagram of a process of encoding and decoding by an encoder in a deep video codec proposed in an embodiment of the present application; Figure 5 It is a schematic diagram of a process of encoding and decoding by a decoder in a deep video codec proposed in an embodiment of the present application; Figure 6 is a schematic diagram of refreshing a decoding buffer proposed in an embodiment of the present application; Figure 7 is a schematic diagram of obtaining an auxiliary frame when an error occurs, as proposed in an embodiment of the present application; Figure 8 This is a schematic diagram of an actual network transmission environment of a codec-side separation model proposed in this application; Fig. 9 It is a comparison diagram of a network transmission method without using error-resistant network transmission and a transmission method using auxiliary frames proposed in the present application; Fig.10 It is a functional module diagram of a device for auxiliary stream error-resistant network transmission based on deep learning video codec proposed in an embodiment of the present application; Fig.11 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] With the rapid development of artificial intelligence technology, it has become possible to use data-driven methods to achieve comprehensive optimization of video encoding and decoding systems. How to ensure the accuracy of video data transmission during the transmission process is particularly important. At present, two new frame types are proposed in error-resistant network transmission: Switching Intra-coded (SI) frames and Switching Predictive (SP) frames. The encoding principle of SP frames is similar to that of P frames, but the reference frames are different. The coding efficiency of SP frames is slightly lower than that of P frames, but still much higher than that of I frames. Due to their excellent network affinity, SP frames are particularly suitable for transmission in wireless channel environments with high packet loss rates and large noise interference. Unlike I frames that use intra-frame coding for all frames, SI frames only use intra-frame coding for slices that need to use switching frames, which improves the overall compression rate of the video. SI frames can improve the compression rate while avoiding the impact of error codes on more subsequent frames. Specifically, refer to Figure 1 , Figure 1 is a schematic diagram of using SI frames and SP frames for error compensation proposed in the prior art. Figure 1 (a) is a schematic diagram of compensation method based on SI frame. Figure 1 (b) is a schematic diagram of the compensation method based on SP frames. During video streaming, if a P frame cannot be decoded correctly and the encoder receives an error report from the user, the encoder can send an auxiliary SP frame to prevent the error from affecting more subsequent frames. Similarly, the decoding end can also achieve the same error recovery function by using an auxiliary SI frame. However, the error-resistant network transmission protocol based on SP frames needs to cache previously decoded frames, and the protocol based on SI frames needs to transmit more additional bits.

[0021] Compared with traditional coding standards, deep learning-based video codecs have higher compression rates at the same reconstruction quality. However, deep learning-based video codecs have very poor error robustness. Once errors such as bit flips or bit erasures occur during transmission, they will have a catastrophic impact on the decoding and reconstruction results.

[0022] Based on this, the present application proposes an auxiliary stream error-resistant network transmission method based on deep learning video codec to improve the robustness performance and the accuracy of reconstruction results.

[0023] In a first aspect of the embodiments of the present application, a method for network transmission of auxiliary streams with robust bit errors based on a deep learning video codec is provided. Figure 2 , Figure 2 1 is a schematic diagram of the steps of a method for error-resistant network transmission of an auxiliary stream based on a deep learning video codec provided in an embodiment of the present application, the method comprising the following steps: Step 201: The decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, thereby obtaining a decoded reconstructed frame APn-1 with slight distortion. Step 202: The decoding end uses the decoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the decoding end, and performs encoding and decoding with all reference contents except the reference image set to None to obtain a correctly decoded reconstructed frame APn.

[0024] In order to clearly describe the auxiliary stream error-resistant network transmission method for deep learning video codec proposed in the embodiment of the present application, Figure 3 For detailed explanation, Figure 3 It is a flow chart of a method for error-resistant network transmission of auxiliary streams based on a deep learning video codec provided in an embodiment of the present application.

[0025] When the specific implementation step 201 is performed, when the video codec finds that an error occurs in the transmission of video frame Pn-1 during the transmission of video frame Pn-1, the decoding end uses the erroneous code stream of video frame Pn-1 for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous code stream is set to all zeros to obtain a slightly distorted decoded reconstructed frame APn-1. The slightly distorted decoded reconstructed frame APn-1 is an erroneously decoded video frame. The core of the auxiliary stream error-resistant network transmission method proposed in the present application is that the reference frames of the decoding end and the encoding end must be synchronized. Once they are not synchronized, catastrophic errors will occur. Since the distortion of the current erroneous frame is not serious, the frame is set as an AP frame (auxiliary frame) in the present application and is used as a reference frame for subsequent encoding and decoding.

[0026] When the step 202 is specifically implemented, when the slightly distorted decoded reconstructed frame APn-1 is used as a reference frame for encoding and decoding, the decoding end uses the decoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the decoding end, sets all reference contents except the reference image to None, and performs encoding and decoding to obtain a correctly decoded reconstructed frame APn. In the present application, the encoding end and the decoding end each include one of the above decoding buffers.

[0027] By using the slightly distorted decoded reconstructed frame APn-1 as an auxiliary frame for encoding and decoding, and using the frame as the next reference frame, cache space is greatly saved. At the same time, since the position of the error is randomly generated during video transmission, and once an error occurs, the decoding result of the corresponding code stream will become random, therefore, when decoding the reconstructed frame APn-1 in the present application, the decoding result corresponding to the erroneous code stream is directly set to all zeros, which can synchronize the encoding end and the decoding end with the highest efficiency, eliminating the additional encoding overhead for the error position.

[0028] In addition, since the reference content used in transmission will be propagated through multiple frames, when an error occurs, the reference content will also be erroneous and distorted, and correcting these distortions will require more bits. Therefore, in this application, the reference content other than the reference image is directly refreshed, and only the reference image is retained as a reference for future inter-frame coding to reduce the bits used to correct the distortion.

[0029] In an optional embodiment of the present application, before the decoding end uses the erroneous bit stream for encoding and decoding, the above method also includes the following process: the encoding end encodes and decodes the original frame Pn-1 to obtain four bit streams of the encoded original frame Pn-1, and sends them to the decoding end; the decoding end confirms through verification that the erroneous bit stream is one of the four bit streams of the encoded original frame Pn-1, sends NACK (Negative Acknowledgement) feedback information, and informs the above encoding end of the error and the erroneous bit stream.

[0030] In an optional embodiment of the present application, after the encoding end receives NACK feedback information from the decoding end, the method also includes: the above-mentioned encoding end sets the erroneous code stream to all zeros, and re-encodes and decodes the code stream after setting to zero, to obtain a slightly distorted coded reconstructed frame APn-1, and uses the slightly distorted coded reconstructed frame APn-1 as an erroneous coded video frame, and is used as an auxiliary frame for subsequent coding and decoding to achieve reference frame synchronization between the encoding end and the decoding end; then, the above-mentioned encoding end uses the slightly distorted coded reconstructed frame APn-1 as a reference image, and refreshes the decoding buffer of the encoding end, and performs coding and decoding with all reference contents except the above-mentioned reference image set to None, to obtain the correctly coded original frame APn.

[0031] Since the distortion of the current error frame is not serious, the frame is set as an AP frame and used as the next reference image frame. This method greatly saves cache space compared to the transmission method of the SP frame.

[0032] At the same time, considering that when the probability of transmission errors is low, the smaller the encoding granularity, the better it can control error propagation, and the deep video codec model has a higher compression rate, the encoding end and the decoding end used in the embodiments of the present application respectively use the deep video codec model as the codec for encoding and decoding. The deep video codec model mainly includes the following processes: context-based conditional encoding and decoding process, the process of optical flow network estimating motion vectors and completing encoding and decoding, and the process of generating context features. In the context-based conditional encoding and decoding process, the frame to be encoded is input. and contextual features To generate the reconstructed frame and Features The optical flow network estimates the motion vector and completes the encoding and decoding process. and decoded frames To generate the reconstructed motion vector The reconstructed motion vector is input into the process of generating context features. , features before the propagated reconstructed frame and decoded frames To generate context features . In the process of generating context features, the deep video codec model introduces an offset prediction mechanism and a cross-group fusion module to further enhance the propagation of context information in the network. In addition, entropy coding based on quadtree partitioning is proposed in this model to estimate the probability distribution more accurately. Compared with previous models, this coding method has finer coding granularity, better parallelism and higher compression rate.

[0033] For the specific encoding and decoding process in the deep video codec model, see Figure 4 and Figure 5 , Figure 4 is a schematic diagram of a process of encoding and decoding performed by an encoder in a deep video codec proposed in an embodiment of the present application, Figure 5 This is a schematic diagram of the process of encoding and decoding in a decoder of a deep video codec proposed in an embodiment of the present application. The processing of video frames in the encoder mainly includes two steps: encoding and decoding. The decoder only needs to decode the original image from the received bit stream. Therefore, when describing a video codec, generally only the encoder is described, and the decoder is a subset of the encoder.

[0034] The encoding and decoding process of the decoder in the above-mentioned deep video codec is as follows: First, extract the last decoded frame from the decoding buffer as the reference frame , and the current video frame to be encoded Perform motion estimation together to generate motion vectors , and encode it to obtain a motion vector code stream. Then, the encoder decodes the encoded motion vector code stream to obtain the motion vector , and combine with the reference frame for motion compensation to generate motion estimation results , and then The frame to be encoded Operation , and get the residual , and encode it into a residual code stream. It includes: motion estimation and motion compensation operations, context feature learning operations, and motion estimation and motion compensation operations are implemented using methods such as deep learning optical flow estimation. The resulting bitstream from context feature learning operations contains two parts: residual and residual super prior. Compared with traditional video coding standards, The residual code stream is a whole and cannot be subdivided. After that, proceed The inverse operation yields the result , and finally post-process to get the decoded frame (Right now Figure 4 Reconstructed frame in ), and send it to the decoding buffer for cache as a reference for the encoding of the next frame. In the deep video encoding and decoding model, reference features, reference motion vectors, etc. are also calculated and sent to the decoding buffer as a reference.

[0035] The encoding and decoding process of the encoder in the above-mentioned deep video codec is as follows: first, the received motion vector bitstream is decoded to obtain the motion vector, and motion compensation is performed in combination with the reference frame to obtain the motion compensation result. The received residual bitstream is decoded to obtain the decoded residual, and the inverse operation is performed together with the motion compensation result to obtain the result. , and finally post-process to get the decoded frame (Right now Figure 5 Reconstructed frame in ).

[0036] In traditional video encoders, the encoding steps generally include DCT transformation, quantization, entropy coding, reordering and other steps. The resulting bitstream is a whole that cannot be further subdivided and only includes two parts: motion vector and residual. In the deep video coding model, the bitstream obtained by encoding and decoding includes the corresponding content of the traditional encoder and its super prior. The four bitstreams transmitted between the above-mentioned encoding end and the above-mentioned decoding end in the deep video coding model include: motion vector super prior bitstream, motion vector bitstream, residual super prior bitstream, and residual bitstream. When the deep learning-based video codec decouples the bitstream of a block into four streams, the coding granularity is reduced. Therefore, when an error occurs in a bitstream, it will not immediately have a large impact.

[0037] In an optional embodiment of the present application, see Figure 6 , Figure 6This is a schematic diagram of refreshing a decoding buffer proposed in an embodiment of the present application. The reference contents of the decoding buffer in the deep video coding model include: reference images, reference features, reference motion features, reference motion vectors, and reference residuals. The decoding buffers used at the encoding end and the decoding end in the embodiment of the present application are all decoding buffers in the deep video coding model. The traditional decoding buffer only contains reference images. Among them, in addition to the reference content other than the reference image, which is unique to the deep video coding model, these contents perform secondary predictions on variables that need to be transmitted, such as motion vectors and residuals, thereby further improving the compression rate. Since the reference content used in transmission will be propagated through multiple frames, when errors occur, the reference content will also be erroneous and distorted. Correcting these distortions will require more bits. Therefore, in the present application, the reference content other than the reference image is directly refreshed, and only the reference image is retained as a reference for future inter-frame coding, which is the same as Figure 1 Compared with the SI frame protocol in the IEEE 802.11ac standard, the method proposed in the present application uses less bits to correct the distortion generated, thereby reducing the additional bit overhead.

[0038] In an optional embodiment of the present application, the four code streams transmitted in the deep video coding model include: a motion vector super-a priori code stream, a motion vector code stream, a residual super-a priori code stream, and a residual code stream. Figure 7 , Figure 7 This is a schematic diagram of obtaining an auxiliary frame when an error occurs, as proposed in an embodiment of the present application. When the erroneous code stream is a residual super-prior code stream among the four code streams, the erroneous code stream is notified to the decoding end, and the decoding end uses the erroneous code stream for normal decoding, that is, the decoding end sets the decoding result corresponding to the erroneous residual super-prior code stream to all zeros, and the remaining code streams are decoded normally, and then the decoding result set to all zeros is used for residual decoding to obtain a slightly distorted decoded reconstructed frame APn-1. If the erroneous code stream is the remaining code stream, the decoding result of the erroneous code stream is set to all zeros, and the decoding of the next code stream is performed.

[0039] In an optional embodiment of the present application, when encoding and decoding the nth frame at the encoding end, the n-2th frame is not cached, and only the n-1th frame is used as a reference for encoding and decoding. Figure 1 Compared with the SP frame protocol in , the method proposed in this application does not need to cache the extra frame (n-2th frame) previously decoded, which greatly saves cache space.

[0040] Exemplarily, the embodiment of the present application uses a deep video codec model as a video codec, performs a packet loss experiment on one of the streams of motion vector super-prior, motion vector, residual super-prior, and residual, and transmits the other streams normally. PSNR (Peak Signal-to-Noise Ratio) is used to measure the reconstruction quality, and bits per pixel (Bits perPixel, BPP) is used to measure the bandwidth occupancy. The changes in these two indicators after packet loss are observed for the transmission method without using auxiliary frames and the transmission method with auxiliary frames. The experiment uses a video with a resolution of 1080p from the UVG dataset, which is randomly cropped to The experimental results are obtained by performing 318 repeated experiments for each quantization step size and each stream.

[0041] The experiment was conducted based on a simulation setting with the encoding and decoding ends separated. The above packet loss means that the stream is discarded after an error occurs and replaced with an all-zero bitstream. The schematic diagram of the actual network transmission environment modeled by separating the encoding and decoding ends is shown in the figure. Figure 8 As shown ( Figure 8 This is a schematic diagram of the actual network transmission environment of the codec-end separation model proposed in this application).

[0042] Taking SI frame compensation as a benchmark, the changes in BPP and PSNR of each stream loss under different quantization step conditions are shown in Table 1. When an error occurs in frame 0, the SI frame compensation scheme will use I frame for encoding, and the bit overhead of I frame is relatively large. In contrast, the average PSNR of the scheme proposed in the present invention decreases by 0.0229% compared with the SI frame scheme within 4 frames after the error occurs, but the average BPP increment decreases by 60.53%, which greatly reduces the number of additional bits used for compensation with almost no impact on the reconstruction quality. The BPP and PSNR outside the 4 frames after the error occurs are basically the same as those without error, and the compensation is basically completed, so it will not be repeated.

[0043] Table 1 Changes in BPP and PSNR of each stream loss under different quantization step sizes

[0044] For a comparison of the results of the transmission method without error-resistant network and the transmission method with auxiliary frames when using the deep video coding model as a video codec, see Fig. 9 , Fig. 9This is a comparison diagram of a non-error-resistant network transmission method and an auxiliary frame transmission method proposed in this application. Frame number 0 represents an error in the transmission of this frame. When an error occurs, a color block will appear on the image and spread rapidly until the entire image cannot be decoded. That is, when the error-resistant network transmission method is not used, once an error occurs, it will have catastrophic consequences. The error will spread rapidly on the video and eventually lead to complete inability to decode. The auxiliary frame compensation method can correct the error immediately and prevent the propagation of the error.

[0045] In the error-resistant network transmission method for auxiliary streams based on deep learning video codecs proposed in the embodiment of the present application, the decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros to obtain a slightly distorted decoded reconstructed frame APn-1; the decoding end uses the decoded reconstructed frame APn-1 as a reference image, and refreshes the decoding buffer of the decoding end, and sets all reference contents except the reference image to None for encoding and decoding to obtain a correctly decoded reconstructed frame APn. This method is a low-error network transmission method that can adapt to dynamic transmission environments, reduce the bandwidth requirements for transmission while ensuring the quality of video reconstruction, and enhance the error-resistant robustness of the deep learning-based video codec.

[0046] In a second aspect of the embodiment of the present application, there is provided an auxiliary stream error-resistant network transmission device based on a deep learning video codec, referring to Fig.10 , Fig.10 1 is a functional module diagram of a device for network transmission of auxiliary streams with robust bit errors based on a deep learning video codec proposed in an embodiment of the present application, the device comprising: The decoding and reconstruction module 1001 is used for the decoding end to use the erroneous bit stream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bit stream is set to all zeros to obtain a slightly distorted decoded and reconstructed frame APn-1; The correct decoding and reconstruction module 1002 is used for the decoding end to use the decoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the decoding end, and perform encoding and decoding with all reference contents except the reference image set to None to obtain the correctly decoded and reconstructed frame APn.

[0047] Wherein, before decoding and reconstructing the module, the device further comprises: The original frame code stream acquisition module is used for encoding and decoding at the encoding end to obtain four code streams of the encoded original frame Pn-1 and send them to the decoding end; The confirmation module is used for the decoding end to confirm through verification that the erroneous code stream is one of the four code streams encoding the original frame Pn-1, and send NACK feedback information to inform the encoding end of the error generation and the erroneous code stream.

[0048] Wherein, after the confirmation module, the device further includes: A coding reconstruction module, used for the coding end to set the erroneous code stream to all zeros, and re-encode and decode the code stream after setting to zero, so as to obtain a coding reconstruction frame APn-1 with slight distortion; The correct encoding and reconstruction module is used for the encoding end to use the slightly distorted encoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the encoding end, set all reference contents except the reference image to None for encoding and decoding, and obtain the correctly encoded original frame APn.

[0049] Wherein, the device further comprises: The codec confirmation module is used for the encoding end and the decoding end to use the deep video codec model as the codec for encoding and decoding respectively. The four code streams transmitted between the encoding end and the decoding end include: motion vector super-prior code stream, motion vector code stream, residual super-prior code stream, and residual code stream.

[0050] Wherein, the device further comprises: A code stream zeroing module, used for setting the corresponding decoding result of the residual super a priori code stream to all zeros when the erroneous code stream is the residual super a priori code stream; The residual decoding module is used to perform residual decoding using the decoding result set to all zeros to obtain a decoded reconstructed frame APn-1 with slight distortion.

[0051] Wherein, the device further comprises: The coding and decoding reference frame confirmation module is used for the coding end and the decoding end to perform coding and decoding based on the SI frame protocol respectively. When coding and decoding the nth frame, the n-2th frame is not cached, and only the n-1th frame is used as a reference for coding and decoding.

[0052] Based on the same application concept, the embodiment of the present application discloses an electronic device in a third aspect. Fig.11 A schematic diagram of an electronic device disclosed in an embodiment of the present application is shown. Fig.11 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory of the electronic device is not less than 12G, the main frequency of the processor is not less than 2.4GHz, the memory 110 and the processor 120 are connected through a bus communication, and a computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement an auxiliary stream error-resistant network transmission method for a deep learning video codec based on an embodiment of the present application.

[0053] Based on the same application concept, the fourth aspect of the embodiment of the present application discloses a computer-readable storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by the processor, an auxiliary stream error-resistant network transmission method based on a deep learning video codec disclosed in the embodiment of the present application is implemented.

[0054] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0055] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0056] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0057] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0058] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0059] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0060] The above is a detailed introduction to the auxiliary stream error-resistant network transmission method based on deep learning video codec provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for error-resistant network transmission of auxiliary streams based on deep learning video codec, characterized in that: The method comprises: The decoding end uses the erroneous bitstream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bitstream is set to all zeros, thereby obtaining a decoded reconstructed frame APn-1 with slight distortion. The decoding end uses the decoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the decoding end, and performs encoding and decoding with all reference contents except the reference image set to None, thereby obtaining a correctly decoded reconstructed frame APn.

2. The method for error-resistant network transmission of auxiliary streams based on deep learning video codec according to claim 1, characterized in that: Before the decoding end uses the erroneous bit stream for encoding and decoding, the method further includes: The encoder performs encoding and decoding to obtain four code streams of the encoded original frame Pn-1 and sends them to the decoder; The decoding end confirms through checking that the erroneous code stream is one of the four code streams encoding the original frame Pn-1, and sends NACK feedback information to inform the encoding end of the error generation and the erroneous code stream.

3. The method for error-resistant network transmission of auxiliary streams based on deep learning video codec according to claim 2, characterized in that: After the encoding end receives the NACK feedback information from the decoding end, the method further includes: The encoding end sets the erroneous code stream to all zeros, and re-encodes and decodes the code stream after setting to zero, to obtain a coded reconstructed frame APn-1 with slight distortion; The encoding end uses the slightly distorted encoded reconstructed frame APn-1 as a reference image, refreshes the decoding buffer of the encoding end, sets all reference contents except the reference image to None for encoding and decoding, and obtains the correctly encoded original frame APn.

4. The method for error-resistant network transmission of auxiliary streams based on deep learning video codec according to claim 1, characterized in that: The encoding end and the decoding end respectively use the deep video encoding and decoding model as the codec for encoding and decoding, and the four code streams transmitted between the encoding end and the decoding end include: motion vector super-prior code stream, motion vector code stream, residual super-prior code stream, and residual code stream.

5. The method for error-resistant network transmission of auxiliary streams based on deep learning video codec according to claim 4, characterized in that: The method further comprises: When the erroneous bitstream is the residual super a priori bitstream, setting the corresponding decoding result of the residual super a priori bitstream to all zeros; Residual decoding is performed using the decoding result set to all zeros to obtain a decoded reconstructed frame APn-1 with slight distortion.

6. The method for error-resistant network transmission of auxiliary streams based on deep learning video codec according to claim 1, characterized in that: The method further comprises: The encoding end and the decoding end perform encoding and decoding based on the SI frame protocol respectively. When encoding and decoding the nth frame, the n-2th frame is not cached, and encoding and decoding is performed only with reference to the n-1th frame.

7. An auxiliary stream error-resistant network transmission device based on deep learning video codec, characterized in that: The device comprises: A decoding and reconstruction module is used for the decoding end to use the erroneous bit stream for encoding and decoding. During the encoding and decoding process, the decoding result corresponding to the erroneous bit stream is set to all zeros to obtain a slightly distorted decoded and reconstructed frame APn-1; The correct decoding and reconstruction module is used for the decoding end to use the decoded and reconstructed frame APn-1 as a reference image, refresh the decoding buffer of the decoding end, and perform encoding and decoding with all reference contents except the reference image set to None to obtain the correctly decoded and reconstructed frame APn.

8. The auxiliary stream error-resistant network transmission device for deep learning video codec according to claim 7, characterized in that: Before decoding and reconstructing the module, the device further comprises: The original frame code stream acquisition module is used for encoding and decoding at the encoding end to obtain four code streams of the encoded original frame Pn-1 and send them to the decoding end; The confirmation module is used for the decoding end to confirm through verification that the erroneous code stream is one of the four code streams encoding the original frame Pn-1, and send NACK feedback information to inform the encoding end of the error generation and the erroneous code stream.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the auxiliary stream error-resistant network transmission method based on deep learning video codec according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the auxiliary stream error-resistant network transmission method based on a deep learning video codec described in any one of claims 1-6 is implemented.

Images