Video super-resolution-based scalable video coding and decoding method, equipment and medium
By building a super-resolution algorithm model, ultra-high-definition videos are encoded and decoded, and the video quality and resolution are dynamically adjusted, which solves the problem of high computing complexity and bandwidth requirements in traditional technology, and realizes the High-quality video playback.
Patent Information
- Application Number
- CN202410981814.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional encoding and decoding technology increases computational complexity, high bandwidth requirements and serious video quality losses when processing ultra-high-definition videos, and cannot dynamically adjust the video resolution according to the playback object.
By building a super-resolution algorithm model, the target video is encoded and decoded, and the high-resolution frame is downsampled to obtain low-resolution frames, and then the basic layer and enhancement layer are encoded and coded, combining video super-resolution technology and Scalable codec technology to dynamically adjust video quality and resolution.
Significantly improve video visual quality, reduce bandwidth requirements, ensure that videos are played at the best quality under different devices and network conditions, and make up for the quality loss of traditional encoding and codec technology at low bit rates.
Smart Images

Figure CN118972616A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video processing, and in particular to a scalable video encoding and decoding method, device and medium based on video super-resolution. Background Art
[0002] Ultra-high-definition video, with its extremely high resolution and fine image quality, is becoming a hallmark of the modern audio-visual experience. In order to effectively process and transmit this data, effective encoding algorithms are needed to compress this data into a smaller volume while maintaining high-quality visual effects as much as possible.
[0003] In online conference and live broadcast scenarios, scalable video coding can effectively reduce redundancy by exploiting the correlation between videos of different resolutions or qualities, rather than transmitting each video stream independently. However, the implementation of scalable codec technology is complex and has certain requirements for network stability. In addition, factors such as video quality, transmission efficiency, and real-time performance need to be balanced in the design of the codec. When processing ultra-high-definition video, it can make up for the quality loss that traditional codec technology may cause at low bit rates.
[0004] Therefore, traditional codec technology faces several key problems when processing ultra-high-definition videos: 1) Increased computational complexity: Processing high-resolution videos requires more computing resources, and the time and energy consumption of encoding and decoding increase; 2) High bandwidth requirements: Ultra-high-definition video transmission requires larger bandwidth, which is a challenge for network transmission and mobile devices; 3) Quality loss: At low bit rates, traditional codec technology may cause obvious loss of video quality, such as compression artifacts or blurred details. Summary of the invention
[0005] The technical problem to be solved by the present invention is that the video quality is low, the bandwidth requirement is high, the quality loss is large, and the resolution of the video cannot be dynamically adjusted according to the playback object. The purpose is to provide a scalable video encoding and decoding method, device and medium based on video super-resolution, and encode and decode the target video by constructing a super-resolution algorithm model; before encoding, the HR frame is downsampled to obtain the original low-resolution LRP, and then the base layer BL encoding and the base layer BL enhancement layer EL encoding and decoding are performed, combining the image reconstruction capability of the video super-resolution technology and the dynamic adjustment capability of the scalable video encoding and decoding technology, the quality and resolution of the video can be dynamically adjusted according to the capabilities of the receiving device and the network conditions, and by retaining multiple resolution and quality versions, the video can be played with the best quality on different bandwidths and devices.
[0006] The present invention is achieved through the following technical solutions:
[0007] A first aspect of the present invention provides a scalable video encoding and decoding method based on video super-resolution, comprising the following specific steps:
[0008] Build a super-resolution algorithm model to encode and decode the target video;
[0009] The encoding process includes:
[0010] Get the high-resolution HR frame of the target video, downsample the HR frame to get the original low-resolution LRP;
[0011] Based on the high-resolution HR frame and the original low-resolution LRP, base layer BL encoding, base layer BL super-resolution enhancement and enhancement layer EL encoding are performed to obtain BL bit stream and EL bit stream respectively;
[0012] The decoding process includes:
[0013] Decode the BL bitstream to obtain a low-resolution LR frame;
[0014] The EL bitstream is decoded based on the low-resolution LR frames to obtain high-resolution HR frames.
[0015] The present invention encodes and decodes the target video by constructing a super-resolution algorithm model; before encoding, the HR frame is downsampled to obtain the original low-resolution LRP, and then the base layer BL encoding and the base layer BL enhancement layer EL encoding and decoding are performed. By combining the image reconstruction capability of the video super-resolution technology and the dynamic adjustment capability of the scalable video encoding and decoding technology, the quality and resolution of the video can be dynamically adjusted according to the capabilities of the receiving device and the network conditions. By retaining multiple versions of resolution and quality, the video can be played with the best quality on different bandwidths and devices.
[0016] Furthermore, the super-resolution algorithm model includes a shallow feature extraction module, a deep feature extraction module and an image reconstruction module.
[0017] Furthermore, in the shallow feature extraction module, the LR frame is obtained, and the LR frame is convolved to obtain the shallow features;
[0018] The shallow features are input into the deep feature extraction module. The shallow features pass through k VSTBs and a convolution layer to obtain deep features.
[0019] In the image reconstruction module, a reconstruction function is used to aggregate shallow features and deep features, and the features are upsampled through sub-pixel convolution to obtain a reconstructed high-resolution image.
[0020] Furthermore, the base layer BL super-resolution enhancement is performed, and the specific steps include:
[0021] The encoded BL frame is used for inter-layer prediction, and the compressed low-resolution frame LRC obtained from BL encoding is used as the input of the audio-visual speech recognition VSR model, and the output is the video super-resolution inter-layer reference picture ILR_VSR with the same bit rate as the EL frame.
[0022] Further, enhancement layer EL coding is performed, and the specific steps include:
[0023] Get the codec and use the reconstructed picture ILR_VSR configured from BL as the inter-layer reference ILR;
[0024] The ILR_VSR frame is combined with the HR frame to encode the enhancement layer EL to obtain the EL bit stream.
[0025] Furthermore, after obtaining the low-resolution LR frame, the method further includes decoding the EL bit stream using the gain of the neural super-resolution:
[0026] The LR frames obtained after decoding the BL bitstream are extracted, and the LR frames are input into the VSR model to be enhanced into ILR_VSR features and stored.
[0027] Furthermore, in the process of obtaining the ILR_VSR feature according to the LR frame, the low-frequency information is transmitted to the image reconstruction module through SwinVSR to identify and retain the features containing the high-frequency information.
[0028] Furthermore, the EL bit stream decoding specifically comprises the following steps:
[0029] The stored ILR_VSR features are extracted, and the EL bit stream is decoded and restored to obtain the HR frame based on the ILR_VSR features. The recovery process includes:
[0030] fetching a reference frame from the cache using the reference index;
[0031] The codec scales the motion vectors to transfer the target block from the reference frame to the current frame;
[0032] The target block is accumulated into the transmitted block by decoding and upscaling the residual via lightweight bilinear interpolation, and a high-resolution block is output.
[0033] A second aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a scalable video encoding and decoding method based on video super-resolution is implemented.
[0034] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a scalable video encoding and decoding method based on video super-resolution.
[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0036] The target video is encoded and decoded by building a super-resolution algorithm model; the HR frame is downsampled before encoding to obtain the original low-resolution LRP, and then the base layer BL encoding and the base layer BL enhancement layer EL encoding and decoding are performed. Combining the image reconstruction capability of video super-resolution technology and the dynamic adjustment capability of scalable video codec technology, the quality and resolution of the video can be dynamically adjusted according to the capabilities of the receiving device and network conditions. By retaining multiple resolution and quality versions, the video can be played with the best quality on different bandwidths and devices.
[0037] It can significantly improve the visual quality of videos, especially when processing ultra-high-definition videos, and can make up for the quality loss that traditional coding and decoding technologies may cause at low bit rates. Video super-resolution technology uses the spatial and temporal correlations within the image to generate a higher-resolution version that is clearer and more detailed than the original low-resolution video through complex interpolation and reconstruction algorithms. Even at low bit rates or with limited bandwidth, users can still watch clearer and more detailed videos;
[0038] Reduced bandwidth requirements: Super-resolution technology can effectively reduce the bandwidth required to transmit ultra-high-definition video. By upscaling the video to high resolution before encoding and then selecting the appropriate resolution and quality version during transmission, the video transmission requirements can be dynamically adjusted according to network conditions and device capabilities, thereby saving bandwidth resources;
[0039] Combined with video super-resolution technology, users can get a consistent and high-quality viewing experience on different devices. Whether watching on a large-screen TV, tablet or smartphone, they can enjoy clearer and more vivid video content;
[0040] Scalable video coding and decoding technologies face challenges due to their complexity and requirements for network stability. The introduction of video super-resolution technology can partially make up for the shortcomings of these technologies by improving the processing power and quality reproduction capabilities of the decoding end, optimizing the stability and real-time performance of video transmission, and thus improving the overall viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:
[0042] Figure 1 The following is a schematic diagram of the method of the present invention in an embodiment of the present invention:
[0043] Figure 2 The super-resolution model in the embodiment of the present invention is: SwinVSR;
[0044] Figure 3 VSTB architecture in the embodiment of the present invention;
[0045] Figure 4 The figure is a flow chart of encoding and decoding in an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.
[0047] As a possible implementation, Figure 1 As shown, this embodiment provides a scalable video encoding and decoding method based on video super-resolution. This embodiment encodes and decodes the target video by constructing a super-resolution algorithm model; before encoding, the HR frame is downsampled to obtain the original low resolution (Pristine LR, LRP), and then the base layer (Base Layer, BL) encoding and decoding and the enhancement layer (Enhancement Layer, EL) encoding and decoding are performed. Combining the image reconstruction capability of the video super-resolution technology and the dynamic adjustment capability of the scalable video encoding and decoding technology, the quality and resolution of the video can be dynamically adjusted according to the capabilities of the receiving device and the network conditions. By retaining multiple versions of resolution and quality, the video can be played with the best quality on different bandwidths and devices. By combining the image reconstruction capability of the video super-resolution technology and the dynamic adjustment capability of the scalable video encoding and decoding technology, future video streaming media and communication systems can better cope with challenges under different network environments and device conditions, thereby achieving more flexible and efficient video content transmission.
[0048] The specific implementation steps of this embodiment are as follows:
[0049] S1: Super-resolution algorithm model.
[0050] The VSR model is built based on Swin Transformer, and the model is improved on this basis to better achieve super-resolution amplification of LR frames. The improved model is SwinVSR. Figure 2 As shown in the figure, SwinVSR consists of three modules: shallow feature extraction module, deep feature extraction module and high-resolution image reconstruction module.
[0051] S11: Shallow feature extraction. Given LR frame input (H, W and C in are the height, width and number of input channels of the input image respectively), we use a 3×3 convolutional layer H SFE (·) Extract shallow features
[0052] F SF =H SFE (I LR ) (1)
[0053] Where C is the characteristic channel number.
[0054] S12: Deep feature extraction. We start from F SF Extracting deep features
[0055] F DF =H DFE (F SF ) (2)
[0056] Among them, H DFE (·) is a deep feature extraction module, which contains K VSTBs (Video Swin Transformerblocks, VSTB) and a 3×3 convolutional layer. More specifically, the intermediate features F1, F2, ..., F k And output deep features F DF is extracted block by block as:
[0057]
[0058] F DF =H conv (F K ) (4)
[0059] in, represents the i-th VSTB, H conv (·) indicates the last convolutional layer. The convolutional layer is used at the end of feature extraction to introduce the inductive bias of the convolution operation into the transformer-based network, laying a better foundation for the subsequent shallow and deep feature aggregation.
[0060] S13: High-resolution image reconstruction. Aggregate shallow and deep features to reconstruct high-resolution image I CHR :
[0061] I CHR =H REC (F0+F DF ) (5)
[0062] Among them, H REC (·) is the function of the reconstruction module. Shallow features mainly contain low frequencies, while deep features focus on recovering lost high frequencies. Through longer connections, SwinVSR can pass low-frequency information directly to the reconstruction module, helping the deep feature extraction module to focus on high-frequency information and stabilize training. For the implementation of the reconstruction module, we use sub-pixel convolutional layers to upsample the features.
[0063] S14: The main component of the VSR model architecture used in this embodiment is VSTB. Figure 3 As shown in the figure, its construction method is to replace the Multihead Self-Attention (MSA) module in the standard Transformer layer with a 3D shift window-based MSA, and keep other components unchanged. Specifically, VSTB consists of a 3D shift window-based MSA module followed by a feed-forward network (FFN). In particular, a 2-layer multi-layer perceptron (MLP) with a Gaussian Error Linear Unit (GELU) is used as its nonlinear activation function. Layer Normalization (LN) is performed before each MSA module and FFN, and a residual connection is performed after each module.
[0064] like Figure 4 As shown in the figure, the specific process of encoding and decoding is as follows:
[0065] S2: Encoding process at the sender
[0066] S21: Downsampling the target video. Get the target video, downsample the high-resolution HR frame of the target video, and after downsampling, use a compliant SVC (Scalable Video Coding) codec or a non-SVC external codec (such as AVC (Advanced Video Coding)) to encode the downsampled low-resolution LRP frame into a BL bitstream. By downsampling the HR frame to obtain the original low-resolution LRP frame, we use the X265 codec as the base layer BL encoder, encode the LRP frame as the input of the BL encoder, and obtain the BL bitstream.
[0067] S22: Base layer BL super-resolution enhancement. To encode the EL frame image, the encoded BL video from the low reference layer is used for inter-layer prediction to improve the EL coding efficiency. In SVC, the reconstructed video from the reference layer can be used for inter-layer prediction. Therefore, we use the compressed low-resolution frame (Compressed LR, LRC) obtained from the BL encoder as the input of the VSR model, and the output is the video super-resolution inter-layer reference picture (Inter-layerReference_VSR, ILR_VSR) with the same bit rate as the EL frame.
[0068] S23: Enhanced layer EL encoding. We use the reconstructed picture ILR_VSR configured from BL as ILR. Due to the reference structure of SVC, the size of the scalable bitstream is much smaller than the sum of two independent bitstreams of different resolutions, which can save storage costs. Therefore, this embodiment uses the X265 codec to input the HR frame and the ILR_VSR frame into the codec for encoding to obtain the HL bitstream.
[0069] S3: Decoding process at the receiving end
[0070] S31: BL bitstream decoding: First, the received BL bitstream is decoded using the X265 codec, and then the decoded LR frame is input into the VSR model and enhanced into ILR_VSR.
[0071] S32: ILR_VSR feature extraction and caching. Since the target video is a compressed video, which usually contains rich information about frame dependencies, we want to use the gain of neural super-resolution to help the decoding of EL bitstream. First, extract the ILR_VSR features obtained from decoding, and then cache the ILR_VSR features for subsequent EL decoding.
[0072] S33: EL bitstream decoding. When decoding the EL bitstream, we reuse the decoded ILR_VSR features from the buffer to recover the HR frame. First, the codec uses the reference index to reference the frame from the buffer. Next, the codec scales the motion vector. Using the motion vector, the codec transfers the target block from the reference frame to the current frame. Finally, the codec decodes and upscales the residual block using lightweight bilinear interpolation and accumulates it into the transmitted block to output a high-resolution block.
[0073] As a possible implementation, this embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a scalable video encoding and decoding method based on video super-resolution is implemented.
[0074] As a possible implementation, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a scalable video encoding and decoding method based on video super-resolution is implemented.
[0075] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A scalable video encoding and decoding method based on video super-resolution, characterized in that: The specific steps include: Build a super-resolution algorithm model to encode and decode the target video; The encoding process includes: Get the high-resolution HR frame of the target video, downsample the HR frame to get the original low-resolution LRP; Based on the high-resolution HR frame and the original low-resolution LRP, base layer BL encoding, base layer BL super-resolution enhancement and enhancement layer EL encoding are performed to obtain BL bit stream and EL bit stream respectively; The decoding process includes: Decode the BL bitstream to obtain a low-resolution LR frame; The EL bitstream is decoded based on the low-resolution LR frames to obtain high-resolution HR frames.
2. The scalable video encoding and decoding method based on video super-resolution according to claim 1, characterized in that: The super-resolution algorithm model includes a shallow feature extraction module, a deep feature extraction module and an image reconstruction module.
3. The scalable video encoding and decoding method based on video super-resolution according to claim 2, characterized in that: In the shallow feature extraction module, the LR frame is obtained, and the LR frame is convolved to obtain the shallow features; The shallow features are input into the deep feature extraction module. The shallow features pass through k VSTBs and a convolution layer to obtain deep features. In the image reconstruction module, a reconstruction function is used to aggregate shallow features and deep features, and the features are upsampled through sub-pixel convolution to obtain a reconstructed high-resolution image.
4. The scalable video encoding and decoding method based on video super-resolution according to claim 1, characterized in that: Perform base layer BL super-resolution enhancement, the specific steps include: The encoded BL frame is used for inter-layer prediction, and the compressed low-resolution frame LRC obtained from BL encoding is used as the input of the audio-visual speech recognition VSR model, and the output is the video super-resolution inter-layer reference picture ILR_VSR with the same bit rate as the EL frame.
5. The scalable video encoding and decoding method based on video super-resolution according to claim 1, characterized in that: Perform enhancement layer EL coding, the specific steps include: Get the codec and use the reconstructed picture ILR_VSR configured from BL as the inter-layer reference ILR; The ILR_VSR frame is combined with the HR frame to encode the enhancement layer EL to obtain the EL bit stream.
6. The scalable video encoding and decoding method based on video super-resolution according to claim 1, characterized in that: After obtaining the low-resolution LR frame, the EL bit stream is decoded using the gain of the neural super-resolution: The LR frames obtained after decoding the BL bitstream are extracted, and are input into the VSR model according to the LR frames to enhance them into ILR_VSR features and store them.
7. The scalable video encoding and decoding method based on video super-resolution according to claim 1, characterized in that: The process of obtaining the ILR_VSR feature according to the LR frame also includes passing the low-frequency information to the image reconstruction module through SwinVSR to identify and retain the features containing high-frequency information.
8. The scalable video encoding and decoding method based on video super-resolution according to claim 1, characterized in that: The EL bit stream decoding comprises the following specific steps: The stored ILR_VSR features are extracted, and the EL bit stream is decoded and restored to obtain the HR frame based on the ILR_VSR features. The recovery process includes: fetching a reference frame from the cache using the reference index; The codec scales the motion vectors to transfer the target block from the reference frame to the current frame; The target block is accumulated into the transmitted block by decoding and upscaling the residual via lightweight bilinear interpolation, and a high-resolution block is output.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the scalable video encoding and decoding method based on video super-resolution is implemented as described in any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the scalable video encoding and decoding method based on video super-resolution as described in any one of claims 1 to 8 is implemented.
