Video processing method, device, apparatus, decoder, system and storage medium
A super-resolution network model addresses the challenges of image quality loss and frame rate reduction in video compression by simultaneously improving both, enhancing video resolution and frame rate while reducing data and bit rates.
Patent Information
- Application Number
- JP2022581541
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-07-14
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2040-07-14
AI Technical Summary
Current video compression technologies face challenges in maintaining image quality and frame rate, with existing algorithms either having low complexity and minimal quality improvement or high complexity and significant computational overhead.
A super-resolution network model comprising a first sub-network model for improving video resolution and a second sub-network model for enhancing image quality is applied to video sequences, allowing simultaneous improvement of both resolution and frame rate.
The super-resolution network model effectively enhances video image quality and frame rate, reducing the amount of data to be encoded and lowering bit rates, thereby improving codec efficiency.
Smart Images

Figure 0007588163000002 
Figure 0007588163000003 
Figure 0007588163000004
Abstract
Description
[Technical field]
[0001] TECHNICAL FIELD Embodiments of the present application relate to the field of video codec technology, and in particular to a video processing method, apparatus, device, decoder, system and storage medium. [Background technology]
[0002] With the continuous development of the video industry chain and the continuous breakthrough of computer technology, the information dissemination method using video as a medium has been widely applied. The amount of data contained in video is relatively larger than that of ordinary text and images, and with the rapid development of imaging devices, in some scenarios using ultra-high definition video, the amount of data contained in ultra-high definition video is very large, so when storing or transmitting video, due to the limitations of memory capacity and network bandwidth, the video needs to be compressed. However, the image quality of the video after video compression is usually reduced to a certain extent.
[0003] In order to alleviate the discrepancy between data volume and image quality, the improvement of compression algorithms and post-processing of compressed video are important directions in video technology research. However, in the current related technical solutions, if the technical solution has a low algorithmic complexity, the quality improvement effect is not obvious, or if the technical solution has an obvious quality improvement effect, the algorithmic complexity is too high. This is disadvantageous to solving problems such as image quality loss and video frame rate reduction that occur in the video processing and compression process. Summary of the Invention
[0004] The present embodiments provide a video processing method, apparatus, device, decoder, system and storage medium that can improve not only video quality and frame rate but also codec efficiency.
[0005] The technical solution of the embodiment of the present application can be realized as follows.
[0006] In a first aspect, embodiments of the present application provide a video processing method for application to a video device, the method comprising: obtaining a video sequence at a first resolution, the video sequence including at least one video frame; inputting the video sequence into a super-resolution network model to obtain a target video sequence at a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to enhance the resolution of the video sequence, and the second sub-network model is used to enhance the image quality of at least one frame in the output result of the first sub-network model.
[0007] In a second aspect, an embodiment of the present application provides a video processing device for application to a video device, the video processing device comprising: an acquisition unit; and a processing unit; The acquisition unit is configured to acquire a video sequence at a first resolution, the video sequence including at least one video frame; The processing unit is configured to input the video sequence to a super-resolution network model to obtain a target video sequence at a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to enhance the resolution of the video sequence, and the second sub-network model is used to enhance the image quality of at least one frame in the output result of the first sub-network model.
[0008] In a third aspect, embodiments of the present application provide a video device comprising a memory and a processor, the memory is used to store a computer program executable by a processor; The processor, when the computer program is executed, is adapted to carry out the method according to the first aspect.
[0009] In a fourth aspect, embodiments of the present invention provide a computer storage medium having stored thereon a computer program which, when executed by a first processor, implements the method of the first aspect.
[0010] In a fifth aspect, embodiments of the present invention provide a decoder comprising a general decoder and a video processing device, the normal decoder is used to parse the bitstream to obtain a first resolution video sequence; the video processing device is configured to input the video sequence to a super-resolution network model to obtain a target video sequence at a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to adjust the resolution of the video sequence, and the second sub-network model is used to adjust the image quality of at least one frame in the output result of the first sub-network model.
[0011] In a sixth aspect, the present application provides a video system comprising an encoder and a decoder, the encoder comprising a pre-processing device and a normal encoder, the decoder comprising a normal decoder and a video processing device, The pre-processing device is configured to receive an input video sequence at a second resolution and then pre-process the received input video sequence to obtain a video sequence at a first resolution; the normal encoder is configured to perform video compression on the video sequence to generate a bitstream and transmit the bitstream to the standard decoder; the normal decoder is configured to parse the bitstream to obtain the first resolution video sequence; the video processing device is configured to input the video sequence to a super-resolution network model to obtain a target video sequence at a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to enhance the resolution of the video sequence, and the second sub-network model is used to enhance the image quality of at least one frame in the output result of the first sub-network model.
[0012] The present embodiment provides a video processing method, an apparatus, a device, a decoder, a system, and a storage medium, by obtaining a video sequence of a first resolution, inputting the video sequence into a super-resolution network model to obtain a target video sequence of a second resolution, the video sequence including at least one video frame, where the second resolution is higher than the first resolution, and the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to improve the resolution and frame rate of the video sequence, and the second sub-network model is used to improve the image quality of each frame of the output result of the first sub-network model. Thus, the technical solution of the present application adopts a super-resolution network design that can solve problems such as image quality loss and video frame rate reduction occurring in the current video compression process, and also uses one network (i.e., a super-resolution network model) to simultaneously improve the resolution and frame rate of a video image. Therefore, when the super-resolution network model is applied to the processing process of video compression, the image quality of the compressed video can be significantly improved, and the resolution and frame rate of the video image can also be significantly improved. It is also possible to perform a downsampling process on the video sequence before encoding the compressed video, thereby reducing the amount of video data to be encoded, and then perform a corresponding upsampling after decoding when employing the super-resolution network model, thereby reducing the bit rate, reducing the transmitted bitstream, and improving the codec efficiency. [Brief description of the drawings]
[0013] [Figure 1] 4 is a flow chart of a video processing process according to an embodiment of the present invention. [Diagram 2] 4 is a flow chart of another video processing process according to an embodiment of the present invention. [Diagram 3] 2 is a flowchart of a video processing method according to an embodiment of the present invention. [Figure 4]FIG. 2 is a schematic structural diagram of a super-resolution network model according to an embodiment of the present application; [Diagram 5] FIG. 2 is a schematic structural diagram for extracting an EPI image according to an embodiment of the present invention. [Figure 6] 1 is a schematic diagram of a detailed network structure of a branch-and-merge super-resolution network model according to an embodiment of the present application; [Figure 7A] FIG. 2 is a schematic structural diagram of a branching module according to an embodiment of the present application. [Figure 7B] FIG. 2 is a schematic structural diagram of a ResDB module according to an embodiment of the present application. [Figure 8] FIG. 2 is a schematic diagram of a detailed network structure of an interpolation frame reinforcement network model according to an embodiment of the present application; [Figure 9A] FIG. 2 is a schematic structural diagram of an attention mechanism module according to an embodiment of the present application; [Figure 9B] FIG. 2 is a schematic structural diagram of a channel attention module according to an embodiment of the present application; [Figure 9C] FIG. 2 is a schematic structural diagram of a spatial attention module according to an embodiment of the present application; [Figure 10] 4 is a flow chart of yet another video processing process according to an embodiment of the present invention. [Figure 11] 4 is a flow chart of yet another video processing process according to an embodiment of the present invention. [Figure 12] 1 is a schematic structural diagram of a video processing device according to an embodiment of the present application; [Figure 13] 1 is a schematic structural diagram of hardware of a video processing device according to an embodiment of the present application; [Figure 14] 1 is a schematic structural diagram of a video device according to an embodiment of the present invention; [Figure 15] FIG. 2 is a schematic structural diagram of a decoder according to an embodiment of the present invention; [Figure 16] 1 is a schematic structural diagram of a video system according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] In order to more fully understand the features and technical contents of the present embodiment, the implementation of the present embodiment will be described in detail below with reference to the drawings, and the accompanying drawings are for reference purposes only and are not intended to limit the present embodiment.
[0015] With the social needs and the progress of science and technology, the multimedia information industry is developing rapidly, and the requirements for video quality and number are becoming higher and higher, which leads to a rapid increase in video data. Because the amount of uncompressed video data is very large, it is essential to compress the video efficiently due to the limitations of storage and transmission networks. However, the image quality of the video after video compression is usually reduced to a certain extent. In order to alleviate the contradiction between data volume and image quality, the improvement of compression algorithms and post-processing of compressed video are two independent methods commonly used at present, and both are important directions of video technology research.
[0016] Referring to Fig. 1, Fig. 1 is a flow chart of a video processing process according to an embodiment of the present application. As shown in Fig. 1, first, data is collected by a video data collection module 101 to obtain an input video sequence, then the input video sequence is processed by a video pre-processing module 102, and then encoded by a video encoder 103 to generate a bitstream, which is transmitted to a video decoder 105 by a channel transmission module 104, which analyzes the bitstream to obtain a video sequence, and the video sequence is processed by a video post-processing module 106 to restore the original input video sequence, and the video display player 107 plays the video. That is, Fig. 1 shows the position of the video post-processing module 106 in the entire video processing process and video compression processing scheme. Here, when video pre-processing is performed by the video pre-processing module 102, due to limitations such as bandwidth, bit rate, etc., the input video sequence may be adjusted, for example, frames may be dropped, thereby reducing the frame rate of the input video sequence, and at the same time, the image quality may be reduced during video compression encoding. At this time, the video post-processing module 106 is an important step in improving the video image quality.
[0017] Referring to FIG. 2, FIG. 2 is a flow chart of another video processing process according to an embodiment of the present application. As shown in FIG. 2, the video processing process is applied to an application scenario of Low Complexity Enhancement Video Coding (LCEVC). Here, the application scenario already has a basic video encoder and a video decoder, but it is desired to use a lower complexity for quality enhancement, and the platform is a hardware platform. The basic idea of LCEVC is multi-layered, first, there is a basic bitstream, and the bitstream can be decoded by adopting any standardized video decoder, thereby obtaining a video with basic quality, and then, by using more layers, a high-quality bitstream can be obtained. For example, the L1 layer, i.e., a bitstream of a residual layer, may be added to the basic bitstream, and then an upsampling process may be performed. Then, the L0 layer, i.e., a bitstream of a high-quality residual layer having more detailed time domain prediction information, is added. Thus, by overlapping these layers, a high-quality video after reconstruction can be obtained.
[0018] It should be understood that, with regard to image quality loss, current video post-processing techniques are mainly classified into image enhancement-based post-processing algorithms and image restoration-based post-processing algorithms. Here, image enhancement-based post-processing algorithms focus on enhancing the subjective quality of the image. A typical algorithm uses a filter to perform a filtering process on the compressed image to improve the visual effect of the image, which is mainly achieved by smoothing the distortion caused by compression rather than restoring the sample values of the image itself. Such algorithms generally have low complexity but not high improvement effects. Image restoration-based post-processing algorithms focus on removing the effects caused by compression, and build a compression degradation model based on prior knowledge and the observation values on the video decoder side, and then select a target function and solve the target function using optimal methods such as maximum a posteriori probability, sparse representation, etc. Such algorithms generally have high complexity, but the improvement effect is obvious.
[0019] Regarding the loss of frame rate, video post-processing techniques aim at enhancing the frame rate. Frame rate enhancement is mainly classified into block matching-based frame rate enhancement algorithms and non-block matching-based frame rate enhancement algorithms. Here, the non-block matching-based frame rate enhancement algorithm does not consider the motion of the object in the image, but only considers the linear interpolation of neighbors, and the advantage is that the computational complexity is low, but serious jitter and blur phenomena occur in the video frames. The block matching-based frame rate enhancement algorithm can perform interpolation on the motion trajectory of the object by estimating the motion vector of the object, and enhance the frame rate of the video. In this way, the quality of the interpolated video frame is obviously improved, but the complexity is also increased.
[0020] Based on this, the embodiment of the present application provides a video processing method, by obtaining a video sequence of a first resolution, inputting the video sequence into a super-resolution network model to obtain a target video sequence of a second resolution, the video sequence including at least one video frame, where the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to improve the resolution of the video sequence, and the second sub-network model is used to improve the image quality of at least one frame in the output result of the first sub-network model. Thus, the technical solution of the present application adopts a super-resolution network design that can solve problems such as image quality loss and video frame rate reduction occurring in the current video compression process, and also uses the same network (i.e., the super-resolution network model) to simultaneously improve the resolution and frame rate of video images. Therefore, when the super-resolution network model is applied to the processing process of video compression, the image quality of the compressed video can be greatly improved, and the resolution and frame rate of the video image can also be significantly improved. It is also possible to perform a downsampling process on the video sequence before encoding the compressed video, thereby reducing the amount of video data to be encoded, and then perform a corresponding upsampling after decoding when employing the super-resolution network model, thereby reducing the bit rate, reducing the transmitted bitstream, and improving the codec efficiency.
[0021] Hereinafter, each embodiment of the present application will be described in detail with reference to the drawings.
[0022] In one embodiment of the present application, please refer to Fig. 3, which is a flowchart of a video processing method according to the embodiment of the present application. As shown in Fig. 3, the method may include steps S301 to S302.
[0023] In step S301, a video sequence in a first resolution is obtained, where the video sequence includes at least one video frame.
[0024] In step S302, the video sequence is input into a super-resolution network model to obtain a target video sequence in a second resolution.
[0025] Here, the super-resolution network model includes at least a first subnetwork model and a second subnetwork model, the first subnetwork model is used to enhance the resolution of the video sequence, and the second subnetwork model is used to enhance the image quality of at least one frame in the output result of the first subnetwork model.
[0026] It should be noted that the video processing method is applied to a video processing device or a video device integrated with the video processing device. After obtaining a video sequence with a first resolution, a super-resolution network model can be used to improve the resolution to obtain a target video sequence with a second resolution. That is, the super-resolution network model can reconstruct a low-resolution, low-frame-rate video sequence into a high-resolution, high-frame-rate video sequence.
[0027] It should be further explained that the resolution of the video sequence includes at least one of a spatial resolution of the video sequence and a temporal resolution of the video sequence.
[0028] In the present embodiment, the spatial resolution of a video sequence may also be referred to as the image resolution of the video sequence, and the temporal resolution of a video sequence may also be referred to as the frame rate of the video sequence. That is, in terms of resolution, the super-resolution network model can be used not only to improve the image resolution of a video sequence, but also to improve the frame rate of a video sequence, and can also be used to simultaneously improve the image resolution and frame rate of a video sequence, and the present embodiment is not limited thereto.
[0029] Specifically, for the first sub-network model, when the first sub-network model is used to only enhance the image resolution of a video sequence, in some embodiments, the super-resolution network inputs the video sequence into the super-resolution network model to obtain a target video sequence of a second resolution, performing a resolution enhancement process on the video sequence by the first sub-network model to obtain an initial video sequence of a second resolution, where the second resolution is higher than the first resolution; and performing a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model to obtain the target video sequence.
[0030] For a first sub-network model, if the first sub-network model is used to enhance only the frame rate of a video sequence, in some embodiments, inputting the video sequence into a super-resolution network model to obtain a target video sequence in a second resolution includes: performing a resolution enhancement process on the video sequence by the first sub-network model to obtain an initial video sequence at a second resolution, where the initial video sequence includes the at least one video frame and at least one interpolated frame; and performing a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model to obtain the target video sequence.
[0031] For a first sub-network model, when the first sub-network model is used to simultaneously enhance the image resolution and frame rate of a video sequence, in some embodiments, inputting the video sequence into a super-resolution network model to obtain a target video sequence in a second resolution includes: performing a resolution enhancement process on the video sequence by the first sub-network model to obtain an initial video sequence of a second resolution, where the second resolution is higher than the first resolution, and the initial video sequence includes the at least one video frame and at least one interpolated frame; and performing a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model to obtain the target video sequence.
[0032] It should be noted that the super-resolution network model may include at least a first sub-network model and a second sub-network model, where the roles of the first sub-network model and the second sub-network model are different. Here, the first sub-network model may be a Branch Fusion Super Resolution Net (BFSRNet) model, which is mainly used to perform image resolution and / or frame rate enhancement on a video sequence, so that the second resolution is higher than the first resolution, and at least one interpolated frame is newly added to the initial video sequence so that the number of frames corresponding to the obtained initial video sequence is greater than the number of frames corresponding to the video sequence of the first resolution, i.e., to enhance the frame rate. The second sub-network model may be a Quality Enhancement Net (QENet) model, which is mainly used to perform quality enhancement on at least one frame of a video sequence.
[0033] In addition, when the first sub-network model is used to improve the frame rate of a video sequence, the initial video sequence may also include an interpolated frame, and if the image quality of the interpolated frame is still poor after the quality enhancement process is performed on the interpolated frame using the second sub-network model, the quality enhancement can be performed again on only the interpolated frame using the third sub-network model.
[0034] In some embodiments, the second sub-network model may further include a third sub-network model, and the video processing method further comprises: performing a quality enhancement process on an interpolated frame in the target video sequence by the third sub-network model to obtain a target interpolated frame; updating the target video sequence based on the target interpolated frame.
[0035] Here, the third sub-network model is an Interpolate Frame Enhancement Net (IFENet) model, which is mainly used to perform further quality enhancement on the interpolated frames in a video sequence that are obtained by interpolation.
[0036] That is, the core of the present embodiment is to design a branch-and-fuse neural network model (i.e., a super-resolution network model) to simultaneously improve the image resolution and frame rate of a video sequence, i.e., spatial and temporal super-resolution.
[0037] Here, the overall structure of the super-resolution network model may be as shown in FIG. 4, and the super-resolution network model is composed of two sub-network models, a first sub-network model 401 (e.g., a branch-and-fuse super-resolution network model) and a second sub-network model 402 (e.g., a quality enhancement network model), and the second sub-network model 402 may be composed of a normal enhancement network model 4021 and an interpolated frame enhancement network model 4022. Here, the first sub-network model 401 mainly performs super-resolution on a video sequence simultaneously in three different dimensions (including two dimensions in resolution and one dimension in frame rate), and realizes fusion by weighted average to obtain a super-resolution output result. The second sub-network model 402 mainly performs quality enhancement of at least one frame on the super-resolution output result, specifically, the normal enhancement network model 4021 can perform quality enhancement on a frame-by-frame basis on the super-resolution output result, and the interpolated frame enhancement network model 4022 can further perform quality enhancement on an interpolated frame obtained by interpolation.
[0038] Regarding the first sub-network model, the first sub-network model is the core of realizing the super-resolution function, and the first sub-network model may include at least one branching module and one fusion module. Here, in some embodiments, processing the video sequence by the first sub-network model to obtain an initial video sequence of a second resolution includes: constructing an initial epipolar plane image EPI image set of at least one dimension based on the video sequence, where each initial EPI image set of dimension corresponds to one branching module; correspondingly inputting the at least one dimensional initial EPI image set into the at least one branching module to obtain at least one target EPI image set; performing, by the fusion module, a fusion process on the at least one target EPI image set to obtain the initial video sequence; Here, the resolution of each EPI image in the initial EPI image set is a first resolution, and the resolution of each EPI image in the target EPI image set is a second resolution.
[0039] It should be explained that, take the first sub-network model as an example: a branch-fusion super-resolution network model. The branch-fusion super-resolution network model is the core of realizing the super-resolution function. A network with three branches can enhance the resolution from different dimensions of the video sequence, and then perform fusion to obtain the final super-resolution output result.
[0040] Here, the design philosophy of the three branches refers to extracting Epipolar Plane Image (EPI) images. In some embodiments, constructing an initial set of EPI images of at least one dimension based on the video sequence includes: forming a stereoscopic video collection by juxtaposing the video sequences, wherein dimensions of the stereoscopic video collection include a horizontal resolution dimension, a vertical resolution dimension, and a frame rate dimension; extracting at least one EPI image after performing at least one slicing operation on the stereoscopic video set in the one dimension of the stereoscopic video set according to one dimension of the stereoscopic video set, and constructing an initial EPI image set in the one dimension of the stereoscopic video set from the at least one EPI image; and obtaining the at least one dimensional initial EPI image set based on at least one dimension of the stereoscopic video set.
[0041] That is, take FIG. 5 as an example, FIG. 5 is a schematic structural diagram for extracting EPI images according to an embodiment of the present application. As shown in FIG. 5, (a) shows a video sequence including a plurality of video frames. By stacking a plurality of the video sequences with a certain order, a stereoscopic video set represented by V(x,y,n) as shown in (b) can be formed, where the image resolution of each video frame is (H,W) and the number of video frames is N. Then, along the direction of any axis, an arbitrary row is selected as a slice point, and a slice operation is performed on the stereoscopic video set, that is, by extracting pixels of the same height of all images in the set, a series of images having linear characteristics, that is, EPI images, is obtained. As shown in (c), along the direction of the y axis, the i-th row is selected as a slice point, and a slice operation is performed to obtain a single EPI image as shown in (d). In this way, a series of EPI images can be obtained by performing a slice operation on all rows in one axis, thereby forming one initial EPI image set in this axis. Here, one axis represents one dimension.
[0042] It can be understood that the detailed network configuration of the branch-fusion super-resolution network model is as shown in Fig. 6, and the network model can simultaneously realize the effects of image resolution and frame interpolation of video frames. In Fig. 6, the overall framework of the network model includes a first branch (represented as B1_SRNet) module 601, a second branch (represented as B2_SRNet) module 602, a third branch (represented as B3_SRNet) module 603, and a fusion (represented as Fusion) module 604. Here, the three branch modules of the first branch module 601, the second branch module 602, and the third branch module 603 consider three dimensions in the stereoscopic video set, and each branch module can be regarded as an operation on the initial EPI image set of different dimensions in the stereoscopic video set. In addition, in the present embodiment, the three branching modules adopt similar network structures, except that only the parameters of the ConvTranspose3d layer in the 3D convolution module are different, where the ConvTranspose3d layer can be called a transposed 3D convolution layer, or can also be called a 3D deconvolution layer, a 3D deconvolution layer, etc. In this way, the three branching modules are input to a fusion module 604 after completing their respective super-resolution. In some embodiments, the fusion module performs a fusion process on the at least one target EPI image set to obtain the initial video sequence: The fusion module may perform a weighted average process on the at least one target EPI image set to obtain the initial video sequence.
[0043] That is, in the embodiment of the present application, a simple weighted average fusion method can be adopted, and the final output result of the branch-and-fusion super-resolution network model is obtained after the fusion.
[0044] It should be further explained that for each branching module, in some embodiments, the branching module may include an upsampling module and a convolution calculation module, and the video processing method includes: an initial EPI image set in one dimension of the stereoscopic video set is subjected to an upsampling module to increase the resolution of the initial EPI image set to obtain a first EPI image set; The method may further include performing a convolution operation on the first EPI image set by the convolution calculation module to obtain a target EPI image set in one dimension of the stereoscopic video set.
[0045] Here, the convolution calculation module may include a two-dimensional convolution (Conv2d) module, a three-dimensional convolution (Conv3d) module, and a corrected convolution (ResDB) module, and these convolution modules can perform a convolution operation on the first EPI image set to obtain a target EPI image set in one dimension of the stereoscopic video set.
[0046] Specifically, as shown in FIG. 7A, FIG. 7A is a schematic structural diagram of a branching module according to an embodiment of the present application. In FIG. 7A, the branch-fusion super-resolution network model first improves the resolution of the initial EPI image set of the current dimension by an upsampling module (i.e., a simple upsampling operator). Then, it is processed by a convolution calculation module, which includes shallow feature extraction by two Conv2d layers and deep feature extraction by a series of ResDB modules (i.e., ResDB 1 module, ..., ResDB d module, ..., ResDB D module, etc.), where each ResDB module itself uses a residual learning method, and the output of multiple ResDB modules performs feature concatenation by a concat layer, and then uses a 1×1 Conv2d layer to reduce the number of feature channels. In addition, the residual reconstruction also uses one Conv2d layer to obtain an image by jump-type connection of the upsampling module, and further obtains a super-resolution EPI image set. Finally, the 3D deconvolution of the ConvTranspose3d layer must be used to enhance the three dimensional resolution of the stereoscopic video ensemble, and then the Conv3d layer must be used to perform super-resolution reconstruction at the image resolution and frame rate of the stereoscopic video ensemble, where the ConvTranspose3d layer further includes a Leaky Rectified Linear Unit (Leaky ReLU) function.
[0047] Also, as shown in FIG. 7B, the configuration structure of the ResDB module can be composed of a Conv2d layer with three activation functions and one 1*1 Conv2d layer. Here, the inside of each ResDB module all adopts dense connections, and the outputs of these three Conv2d layers are combined by Concat and reduced in dimension by a 1*1 Conv2d layer. In addition, a jump-type connection is used between ResDB modules, that is, the output of the previous block (i.e., ResDB d-1 module) is overlapped with the output of the current block (i.e., ResDB d module), and then the sum is used as the input of the next block (i.e., ResDB d+1 module).
[0048] It should be noted that the activation function can be a rectified linear unit (ReLU), also known as a rectified linear unit, which is a commonly used activation function in artificial neural networks, and usually refers to nonlinear functions such as ramp functions and their derivatives. The leaky ReLU function is a variant of the classic (and widely used) ReLu function. When the input value of the ReLU function is negative, the output is always 0, so its first derivative is also always 0. To solve this shortcoming of the ReLU function, a leaky value is introduced in the negative half interval of the ReLU function, i.e., the leaky ReLU function.
[0049] Furthermore, after the first sub-network model enhances the image resolution and frame rate of the video sequence, the image quality of the video frames needs to be further enhanced, in which case a second sub-network model, i.e., a quality enhancement network model, is added, where the quality enhancement network model may enhance the image quality of each frame or enhance the image quality of a part of the frame. That is, the quality enhancement network model is not fixed, and the second sub-network model usually includes a normal enhancement network model (i.e., the QENet model described in the embodiment of the present application), and the QENet model can adopt any existing image video quality enhancement network model, such as a Super-Resolution Convolutional Neural Network (SRCNN) model, an Artifacts Reduction Convolutional Neural Network (ARCNN) model, a Very Deep convolutional networks for Super-Resolution (VDSR) model, a Recurrent Back-Projection Network for Video Super-Resolution (RBPN) model, and a Video Restoration with Enhanced Deformable Convolutional Networks (EDVR) model, etc. Due to the need for complexity in the design of the video processing device, it is recommended to select an effective and low-complexity network, where the embodiment of the present application preferably selects the ARCNN model.
[0050] However, due to the large motion of some objects in the video sequence, the quality of the interpolated frames when the frame rate is increased is poor, although they meet the existing motion trend. In this case, the image quality of the interpolated frames can be further improved by one simple network (i.e., the third sub-network model).
[0051] In some embodiments, performing a quality enhancement process on an interpolated frame in the target video sequence by the third sub-network model to obtain a target interpolated frame includes: obtaining a pending interpolated frame and one or more predetermined video frames; and inputting the interpolated frame and the predetermined one or more video frames into a third sub-network model to obtain a target interpolated frame corresponding to the interpolated frame.
[0052] Here, the predetermined one or more video frames may be one or more video frames adjacent to the interpolated frame, i.e., obtain the to-be-processed interpolated frame and the predetermined one or more adjacent video frames, and then input the obtained interpolated frame and the predetermined one or more adjacent video frames into a third sub-network model to obtain a target interpolated frame corresponding to the interpolated frame.
[0053] More specifically, the predetermined one or more video frames may be two adjacent video frames, i.e., a previous video frame and a next video frame adjacent to the interpolated frame. In this case, in some embodiments, performing a quality enhancement process on the interpolated frame in the target video sequence by the third sub-network model to obtain a target interpolated frame includes: Obtaining a pending interpolated frame and a previous video frame and a next video frame adjacent to the interpolated frame according to a playback order of the target video sequence; inputting the interpolated frame, the previous video frame, and the next video frame into the third sub-network model to obtain a target interpolated frame corresponding to the interpolated frame.
[0054] That is, after obtaining the pending interpolation frame, a predetermined one or more video frames may be combined, or one or more adjacent video frames may be specifically combined, and further, the adjacent previous video frame and the next video frame may be specifically combined, and then the corresponding target interpolation frame is obtained by the third sub-network model. In the present embodiment, for the expression "previous and next", one possible embodiment is the displaying order corresponding to the video sequence, or another possible embodiment is the image order (i.e., output order) of the decoded video sequence output by a conventional decoder, such as an H.266 / Versatile Video Coding (VVC) decoder, an H.265 / High Efficiency Video Coding (HEVC) decoder, etc., for the decoded video. Here, the displaying order of the target video sequence may be the order defined as "previous and next", but the present embodiment is not limited thereto.
[0055] It should also be noted that the third sub-network model includes a first convolution module, an attention mechanism module, a second convolution module, and an addition module, where the first convolution module includes an activation function, and the second convolution module does not include an activation function.
[0056] Further, in some embodiments, the number of the first convolution modules is 1 or more, the number of the attention mechanism modules is 1 or more, the number of the second convolution modules is 1, and the number of the summation modules is 1. In this case, in some embodiments, inputting the interpolated frame and the one or more predetermined video frames into a third sub-network model to obtain a target interpolated frame corresponding to the interpolated frame includes: performing feature extraction on the interpolated frame and the one or more predetermined video frames by the first convolution module and the attention mechanism module to obtain a feature image; The second convolution module performs a convolution operation on the feature image to obtain a residual image; and overlapping, by the summing module, the enhanced interpolated frame and the residual image to obtain a target interpolated frame corresponding to the interpolated frame.
[0057] In some embodiments, inputting the interpolated frame, the previous video frame, and the next video frame into the third sub-network model to obtain the target interpolated frame includes: Performing feature extraction on the interpolated frame, the previous video frame, and the next video frame by the first convolution module and the attention mechanism module to obtain a feature image; The second convolution module performs a convolution operation on the feature image to obtain a residual image; and overlapping, by the summation module, the enhanced interpolated frame and the residual image to obtain the target interpolated frame.
[0058] That is, the second sub-network model may include a third sub-network model for enhancing image quality. Here, taking the third sub-network model as an example of an interpolated frame enhancing network model, the network structure is as shown in FIG. 8. In FIG. 8, the input of the network model is an interpolated frame whose quality needs to be improved and two video frames before and after it, a total of three frames of images. The network structure is composed of seven Conv layers, the first six Conv layers are used for feature extraction, and the last Conv layer is used for residual reconstruction. Except for the last Conv layer, the remaining six Conv layers are added with activation functions (i.e., ReLU functions filled in gray). The final interpolated frame can be obtained by superimposing the interpolated frame that needs to be improved and the residual image of the network output. In addition, an attention mechanism module (CBAM: Convolutional Block Attention Module) is added to the network structure of FIG. 8, and the attention mechanism module also has an activation function, so that the extracted features can be more accurate.
[0059] As shown in Figure 9A, Figure 9A provides an example of the structure of an attention mechanism module. Here, the attention mechanism module may include a channel attention module (CA) 901 and a spatial attention module (SA) 902. After the input features are sequentially passed through the channel attention module 901 and the spatial attention module 902, the extracted features can be more accurate.
[0060] Here, the channel attention module 901 is specifically shown in Fig. 9B, and its core idea is to use the inter-channel relationship of features to generate a channel attention map. This step may include: performing Max Pool and Avg Pool on the input features (represented by F) to aggregate spatial information to obtain two C-dimensional pooled feature images represented by F_max and F_avg, respectively; feeding F_max and F_avg into a multilayer perceptron (MLP) with one hidden layer to obtain two 1x1xC channel attention maps; and adding and activating corresponding elements to the two channel attention maps obtained by the MLP to obtain a final channel attention map (represented by MC).
[0061] The spatial attention module is specifically shown in Fig. 9C, and its core idea is to generate a spatial attention map by using the spatial relationship between features. This step includes: first, for the features (represented by F') subdivided by the channel attention module 901, performing maximum pooling and average pooling along the channel direction to obtain two 2D feature maps (represented by F_avg and F_max) with attributes of 1 x H x W; dimensionally concatenating the obtained two 2D feature maps to obtain a concatenated feature map; and using a convolution layer of 7 x 7 size for the concatenated feature map to generate a spatial attention map (represented by Ms).
[0062] Referring to the network structure shown in Fig. 8, the specific steps of the interpolation frame reinforcement network model include: for the video frame reinforced by the normal reinforcement network model, the combination of the frame that needs to be reinforced with the interpolation frame and the two frames before and after it, a total of three frames, is input to the interpolation frame reinforcement network model, and a residual image is obtained through a Conv layer having a feature extraction and one residual reconstruction, which is composed of a plurality of Conv layers with activation functions and an attention mechanism module, and the residual image is superimposed with the original interpolation frame image that needs to be reinforced with the interpolation frame, and finally a quality-enhanced target interpolation frame is obtained. That is, the input of the network structure is a total of three frames, including the two frames before and after the interpolation frame and the interpolation frame, and only one frame of the interpolation frame after quality enhancement is output.
[0063] In this way, the super-resolution network model of the present embodiment can be obtained based on the first sub-network model and the second sub-network model. determining model parameters of the super-resolution network model; and The method may further include constructing the super-resolution network model based on the determined model parameters.
[0064] Moreover, in some embodiments, determining model parameters of the super-resolution network model comprises: obtaining a training sample, wherein the training sample includes at least a plurality of video frames at a first resolution and a plurality of video frames at a second resolution; and training a neural network-based super-resolution network model using the training samples to obtain the model parameters.
[0065] Moreover, in some embodiments, determining model parameters of the super-resolution network model comprises: The method may include obtaining the model parameters by parsing a bitstream.
[0066] It should be noted that the training sample includes at least a plurality of video frames in a first resolution and a plurality of video frames in a second resolution, or a plurality of video images in a first resolution and a plurality of video images in a second resolution, and is used to perform model parameter training and obtain model parameters of the super-resolution network model.
[0067] In other words, the model parameters of the super-resolution network model can be obtained, on the one hand, by training the model parameters based on training samples, and on the other hand, the model parameters can be trained by the encoder side, and then the trained model parameters can be written into a bitstream, and the decoder side can directly obtain the model parameters by analyzing the bitstream, and this is not limited to the embodiments of the present application.
[0068] In summary, the present embodiment primarily solves problems such as image quality loss and video frame rate degradation that occur in current video compression processes, while also solving the problem of low current codec efficiency by downsampling (i.e., downsampling processing) during pre-processing and then restoring reconstruction during video post-processing.
[0069] The present embodiment mainly bases on the super-resolution network model and applies it to the video processing process.
[0070] In the following, we describe the application of the super-resolution network model in several application scenarios.
[0071] (1) Applied to the post-processing step of video compression. As shown in Fig. 1, after data collection and video pre-processing, the video encoder 103 performs encoding to generate a bitstream, and then the video decoder 105 analyzes the bitstream to reconstruct the video at the decoding side. Since the loss of image quality and frame rate occurs during the entire video processing process, the super-resolution network model in the embodiment of the present application can be applied to the post-processing of video, thereby improving the image quality of the video, and at the same time, the frame rate can be simultaneously improved during post-processing to meet the frame rate requirements of the decoding side.
[0072] (2) Applied to LCEVC video compression scheme. As shown in Fig. 2, in the current LCEVC scheme, after obtaining a basic bitstream, high quality video is obtained through multiple layers. In the present embodiment, multiple layers can be changed into one layer, that is, after decoding the basic bitstream to obtain the basic quality video, the super-resolution network model in the present embodiment can be directly adopted to obtain the high quality video.
[0073] (3) Applied to low bitrate video compression scheme. Most of the current video sequences are high resolution and high frame rate, and directly compressing the entire video sequence will result in low coding efficiency and large bitstream. To solve this problem, a low bitrate compression scheme can be designed. As shown in FIG. 10, for the original input video sequence at the encoder side, a downsampling module 1001 performs downsampling processing to obtain a low resolution, low frame rate video sequence, and then a video encoder 103 encodes the downsampled video sequence to generate a bitstream, for example, 01010001011101010, and transmits the bitstream to the decoder side, and a video decoder 105 analyzes the bitstream to obtain a decoded low resolution, low frame rate video sequence, and then a super-resolution reconstruction module 1002 can obtain a video sequence with the same high resolution and high frame rate as the input video sequence. At this time, the super-resolution reconstruction module 1002 can realize super-resolution reconstruction by using the super-resolution network model in the embodiment of the present application. That is, the core of the super-resolution network model is to super-resolution reconstruct a decoded low-resolution, low-frame-rate video sequence into a video sequence with the same high resolution and high frame rate as the input video sequence.
[0074] (4) Applied to a scalable video coding (SVC) scheme. It can be selected according to the actual application environment, and can select whether to divide the video sequence into different parts, downsampling coding, how many times downsampling coding, and whether the batch coding is adapted to several channels or terminals. Here, the transmission of video is limited by the bandwidth. When the bandwidth is insufficient, it is necessary to transmit and decode the bitstreams of video coding with different downsampling multiples, and at this time, the image quality of the decoded video is not high, and after decoding, a super-resolution network model can be adopted to improve the image quality. When the bandwidth gradually increases, the bitstream of the input video sequence can be transmitted and decoded to directly obtain a high-quality video. Specifically, as shown in FIG. 11, the process can include the following steps:
[0075] In step S1101, an input video sequence is obtained.
[0076] In step S1102, it is determined whether the bandwidth meets the requirement.
[0077] In step S1103, if the determination result is "YES", a video encoder is used to perform encoding to generate a bitstream.
[0078] If the determination result is "No" in step S1104, downsampling processing is performed, and step S1103 is executed.
[0079] In step S1105, the bitstream is transmitted.
[0080] In step S1106, a video decoder is utilized to parse the bitstream to obtain a parsed video sequence.
[0081] In step S1107, it is determined whether the target quality requirement is met.
[0082] In step S1108, if the determination result is "YES", the analyzed video sequence is determined as the target video sequence.
[0083] In step S1109, if the determination result is "NO", the analyzed video sequence is input to the super-resolution network model for processing, and the output of the super-resolution network model is determined as the target video sequence.
[0084] That is, on the video encoder side, based on the currently detected bandwidth environment, judge whether downsampling is required for the input video sequence, and if necessary, select a corresponding downsampling multiple to downsample the input video sequence at a spatial resolution and frame rate, and then use an existing video encoder to encode it to obtain a bitstream; on the video decoder side, adopt a corresponding video decoder to encode it, and then input the super-resolution network model, thereby obtaining a target video sequence with a required resolution and frame rate.
[0085] It should be understood that the embodiment of the present application proposes a branch-fusion neural network structure that can simultaneously improve the image resolution and frame rate of a video, i.e., super-resolution in space and time. The entire network structure can be composed of two sub-network models, namely, a branch-fusion super-resolution network model and a quality enhancement network model, where the quality enhancement network model can include a normal enhancement network model and an interpolated frame enhancement network model. That is, the entire network structure can be composed of three sub-network models, namely, a branch-fusion super-resolution network model, a normal enhancement network model, and an interpolated frame enhancement network model. Specifically, the branch-fusion super-resolution network model adopts a network structure of a network with three branches, and each branch network adopts a densely coupled residual learning method to improve the performance of the network, and improves the image resolution and frame rate of a video sequence from different dimensions of a stereoscopic video set. Meanwhile, the interpolated frame enhancement network model takes an interpolated frame and two adjacent frames before and after it as input, and can further improve the quality of the interpolated frame by combining with an attention mechanism model.
[0086] It should be noted that there are three possible application scenarios for the super-resolution network model in the embodiments of the present application in the video compression process: the first is simple video compression post-processing, in which the compressed and reconstructed video is super-resolved with video images to improve the frame rate; the second is a low-bitrate video compression scheme designed based on the above, in which the video is first downsampled spatially and temporally before encoding, and then the corresponding super-resolution enhancement is performed after decoding, so that the bit rate can be significantly reduced; the third is applying the super-resolution network model to the inter-frame prediction part of the video encoding standard, which can reduce the number of reference frames and improve the prediction accuracy.
[0087] In addition, the network structure of the super-resolution network model in the embodiment of the present application can be modified. Specifically, the three branches of the branch-fusion super-resolution network model can be appropriately deleted to meet the needs of different scenarios and different computational capabilities. In the practical application of the network structure of the ordinary reinforcement network model, the ARCNN model is usually used, but is not limited to this, as long as it meets the image quality enhancement effect of the video frame. The number of Conv layers and the number of attention mechanism modules of the interpolation frame reinforcement network model are also not limited and can be appropriately increased or decreased. All modifications may cause gaps in the final image quality of the video sequence.
[0088] In this way, the low bit rate compression scheme proposed in the embodiment of the present application greatly improves the coding efficiency. Specifically, by downsampling the video in space and time before compression coding, the amount of video data that needs to be coded is greatly reduced, and the super-resolution network model is adopted to perform corresponding upsampling after decoding to restore the original video. Overall, the bit rate can be greatly reduced, the coding efficiency can be greatly improved, and the transmission bit stream can be reduced. In addition, the super-resolution network design adopted in the embodiment of the present application greatly improves the image quality and frame rate of the video, and the proposed super-resolution network model can be applied to the processing process of video compression to greatly improve the quality of the compressed video, and also has a great effect on the super-resolution and frame rate improvement of the video image. The super-resolution network model proposed in the embodiment of the present application mainly uses one network to simultaneously achieve two effects of super-resolution of the video frame image and improvement of the frame rate.
[0089] As an example, the super-resolution network model can be implemented on the Nvidia GTX 1080Ti GPU using the PyTorch platform, and the training and test sets of the experiment both use Vimeo-90K. Assuming that 4x super-resolution is achieved in the video frame image and the frame rate is improved by 2x, the specific experimental results are shown in Table 1. Here, the first row shows the experimental results using only the first sub-network model (i.e., the BFSRNet model), the second row shows the experimental results of adding the QENet model in the second sub-network model in addition to the BFSRNet model, and the last row shows the experimental results of further using the IFENet model in addition to the BFSRNet model and the QENet model. In Table 1, the Peak Signal to Noise Ratio (PSNR) and Structural SIMilarity (SSIM) are the average results of all frames of one video sequence, the PSNR / SSIM of Super-Resolution (SR) corresponds to the average results of all frames that are simply super-resolutioned, and the PSNR / SSIM of Video Frame Interpolation (VFI) corresponds to the average results of all interpolated frames. As can be seen from the results of Table 1, the normal quality enhancement network model and the interpolated frame enhancement network model added in the embodiment of the present application both have a certain effect, and both the PSNR and SSIM can be improved. [Table 1] The present embodiment provides a video processing method, which includes obtaining a video sequence of a first resolution, inputting the video sequence into a super-resolution network model to obtain a target video sequence of a second resolution, the video sequence including at least one video frame, where the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to improve the resolution of the video sequence, and the second sub-network model is used to improve the image quality of at least one frame in the output result of the first sub-network model. Thus, the technical solution of the present application adopts a super-resolution network design that can solve problems such as image quality loss and video frame rate reduction occurring in the current video compression process, and also uses one network (i.e., a super-resolution network model) to simultaneously improve the resolution and frame rate of a video image. Therefore, when the super-resolution network model is applied to the processing process of video compression, the image quality of the compressed video can be greatly improved, and the resolution and frame rate of the video image can also be significantly improved. It is also possible to perform a downsampling process on the video sequence before encoding the compressed video, thereby reducing the amount of video data to be encoded, and then perform a corresponding upsampling after decoding when employing the super-resolution network model, thereby reducing the bit rate, reducing the transmitted bitstream, and improving the codec efficiency.
[0090] In another embodiment of the present application, based on the same technical concept as the above embodiment, refer to Fig. 12, which is a schematic structural diagram of a video processing device 120 according to the embodiment of the present application. As shown in Fig. 12, the video processing device 120 may include an acquisition unit 1201 and a processing unit 1202; The acquisition unit 1201 is configured to acquire a video sequence in a first resolution, where the video sequence includes at least one video frame; The processing unit 1202 is configured to input the video sequence into a super-resolution network model to obtain a target video sequence in a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to enhance the resolution of the video sequence, and the second sub-network model is used to enhance the image quality of at least one frame in the output result of the first sub-network model.
[0091] In some embodiments, the resolution of the video sequence includes at least one of a spatial resolution of the video sequence and a temporal resolution of the video sequence.
[0092] In some embodiments, the processing unit 1202 is specifically configured to perform a resolution enhancement process on the video sequence by the first sub-network model to obtain an initial video sequence of a second resolution, and to perform a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model to obtain the target video sequence, where the second resolution is higher than the first resolution.
[0093] In some embodiments, the processing unit 1202 is specifically configured to: perform a resolution enhancement process on the video sequence by the first sub-network model to obtain an initial video sequence of a second resolution, the initial video sequence including the at least one video frame and at least one interpolated frame; and perform a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model to obtain the target video sequence.
[0094] In some embodiments, the first sub-network model is a branch-and-fuse super-resolution network model and the second sub-network model is a quality enhancement network model.
[0095] In some embodiments, the second sub-network model further comprises a third sub-network model; The processing unit 1202 is further configured to perform a quality enhancement process on an interpolated frame in the target video sequence by the third sub-network model to obtain a target interpolated frame, and update the target video sequence based on the target interpolated frame.
[0096] In some embodiments, the third sub-network model is an interpolated frame reinforced network model.
[0097] In some embodiments, the first sub-network model includes at least one branching module and one fusion module; The processing unit 1202 is further configured to construct an initial epipolar plane image EPI image set of at least one dimension based on the video sequence, where each dimensional initial EPI image set corresponds to one branching module, correspondingly input the initial EPI image set of at least one dimension to the at least one branching module to obtain at least one target EPI image set, and perform a fusion process on the at least one target EPI image set by the fusion module to obtain the initial video sequence, where a resolution of each EPI image in the initial EPI image set is a first resolution and a resolution of each EPI image in the target EPI image set is a second resolution.
[0098] In some embodiments, the processing unit 1202 is further configured to form a stereoscopic video set by stacking the video sequences, the dimensions of the stereoscopic video set including a horizontal resolution dimension, a vertical resolution dimension, and a frame rate dimension, extract at least one EPI image after performing at least one slicing operation on the stereoscopic video set in the one dimension of the stereoscopic video set based on the one dimension of the stereoscopic video set, construct an initial EPI image set in the one dimension of the stereoscopic video set from the at least one EPI image, and obtain an initial EPI image set of the at least one dimension based on the at least one dimension of the stereoscopic video set.
[0099] In some embodiments, the branching module includes an upsampling module and a convolution calculation module; The processing unit 1202 is further configured to target an initial EPI image set in one dimension of the stereoscopic video set, increase the resolution of the initial EPI image set by the upsampling module to obtain a first EPI image set, and perform a convolution operation on the first EPI image set by the convolution calculation module to obtain a target EPI image set in one dimension of the stereoscopic video set.
[0100] In some embodiments, the processing unit 1202 is further configured to perform, by the fusion module, a weighted average process on the at least one target EPI image set to obtain the initial video sequence.
[0101] In some embodiments, the acquisition unit 1201 is further configured to acquire the pending interpolated frame and the predetermined one or more video frames; The processing unit 1202 is further configured to input the interpolated frame and the given one or more video frames to a third sub-network model to obtain a target interpolated frame corresponding to the interpolated frame.
[0102] In some embodiments, the given one or more video frames include two adjacent video frames; The acquisition unit 1201 is further configured to acquire, according to a playback order of the target video sequence, a pending interpolated frame and an adjacent previous video frame and a next video frame to the interpolated frame; The processing unit 1202 is further configured to input the interpolated frame, the previous video frame, and the next video frame into the third sub-network model to obtain a target interpolated frame corresponding to the interpolated frame.
[0103] In some embodiments, the third sub-network model includes a first convolution module, an attention mechanism module, a second convolution module, and a summation module, wherein the first convolution module includes an activation function and the second convolution module does not include an activation function.
[0104] In some embodiments, the number of the first convolution modules is one or more, the number of the attention mechanism modules is one or more, the number of the second convolution modules is one, and the number of the summation modules is one; The processing unit 1202 is further configured to: perform feature extraction on the interpolated frame and the predetermined one or more video frames by the first convolution module and the attention mechanism module to obtain a feature image; perform a convolution operation on the feature image by the second convolution module to obtain a residual image; and overlay the enhanced interpolated frame and the residual image by the addition module to obtain a target interpolated frame corresponding to the interpolated frame.
[0105] In some embodiments, processing unit 1202 is further configured to determine model parameters of the super-resolution network model and construct the super-resolution network model based on the determined model parameters.
[0106] In some embodiments, the acquiring unit 1201 is further configured to acquire training samples, where the training samples include at least a plurality of video frames at a first resolution and a plurality of video frames at a second resolution; The processing unit 1202 is further configured to utilize the training samples to train a neural network-based super-resolution network model to obtain the model parameters.
[0107] In some embodiments, the obtaining unit 1201 is further configured to obtain the model parameters by parsing a bitstream.
[0108] As can be understood, in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program, or software, and may of course be a module or a non-modular type. Also, each component in this embodiment may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated unit may be realized in the form of hardware or in the form of a software functional module.
[0109] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in one computer-readable storage medium. Based on this understanding, the essential part of the technical solution of this embodiment, i.e., the part that contributes to the prior art, or all or part of the solution of the technology can be embodied in the form of a software product, and the computer software product is stored in one storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a U disk, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0110] In yet another embodiment of the present application, the embodiment provides a computer storage medium applied to a video processing device 120, the computer storage medium storing a computer program, the computer program, when executed by at least one processor, realizing a method according to any of the previously described embodiments.
[0111] Based on the above-mentioned configuration of the video processing device 120 and the computer storage medium, refer to FIG. 13. FIG. 13 shows an example of a specific hardware structure of the video processing device 120 according to an embodiment of the present application, which may include a communication interface 1301, a memory 1302, and a processor 1303, and each component is coupled by a bus system 1304. It can be understood that the bus system 1304 is used to realize the connection communication between these components. In addition to the data bus, the bus system 1304 includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all the various buses are represented as the bus system 1304 in FIG. 7. Here, the communication interface 1301 is configured to transmit and receive signals in the process of transmitting and receiving information to and from other external network elements.
[0112] The memory 1302 is configured to store a computer program executable by the processor 1303 .
[0113] When executing the computer program, the processor 1303: obtaining a video sequence at a first resolution, the video sequence including at least one video frame; inputting the video sequence into a super-resolution network model to obtain a target video sequence at a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to enhance the resolution of the video sequence, and the second sub-network model is used to enhance the image quality of at least one frame in the output result of the first sub-network model.
[0114] As can be appreciated, memory 1302 of the present embodiment may be volatile or non-volatile memory, or may include both volatile and non-volatile memory, where non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory, and volatile memory may be random access memory (RAM) used as an external cache. By way of illustrative, but non-limiting example, many forms of RAM are available, such as, for example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM), etc. The memory 1302 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0115] The processor 1303 may be an integrated circuit chip with signal processing functions. In the realization process, each step of the method can be completed by an instruction in the form of a hardware integrated logic circuit or software in the processor 1303. The above processor 1303 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., and can realize or execute each method, step, and logic block disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly performed by a hardware decoding processor, or may be performed by a combination of hardware and software modules in the decoding processor. The software modules may be located in conventional storage media such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is disposed in the memory 1302, and the processor 1303 reads the information in the memory 1302 and completes the steps of the above-mentioned method in combination with its hardware.
[0116] It can be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For a hardware implementation, the processing unit is implemented in one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processing (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field-Programmable Gate Arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units configured to perform the functions described herein, or a combination thereof.
[0117] In the case of a software implementation, the technical solutions described herein can be realized by executing modules of functionality (e.g., processes, functions, etc.) described herein. The software codes can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.
[0118] Illustratively, as another embodiment, the processor 1303 is further configured to, when executing the computer program, perform the method according to any of the previously described embodiments.
[0119] In some further embodiments of the present application, the video device of the present application may be a video playback device, an encoder, a decoder, etc. Referring to Fig. 14, Fig. 14 is a schematic structural diagram of a video device according to an embodiment of the present application. As shown in Fig. 14, the video device 140 may include at least the video processing device 120 according to any of the above-mentioned embodiments.
[0120] In some further embodiments of the present application, please refer to Fig. 15, which is a schematic structural diagram of a decoder according to an embodiment of the present application. As shown in Fig. 15, the decoder 150 may include at least a normal decoder 1501 and a video processing device 1200 according to any of the above-mentioned embodiments. Here, the normal decoder 1501 is the video decoder 105 according to the above-mentioned embodiments, i.e., also an H.266 / VVC decoder, an H.265 / HEVC decoder, etc.
[0121] In some embodiments, a generic decoder 1501 is used to parse the bitstream to obtain a first resolution video sequence; the video processing unit 120 is configured to input the video sequence into a super-resolution network model to obtain a target video sequence in a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to adjust the resolution of the video sequence, and the second sub-network model is used to adjust the image quality of at least one frame in the output result of the first sub-network model.
[0122] In some embodiments, the normal decoder 1501 is further configured to parse the bitstream to obtain a first indication flag bit, the first indication flag bit being used to indicate that the first resolution video sequence needs to have its spatial resolution enhanced by the super-resolution network model; The video processing device 120 is further configured to perform a resolution enhancement process on the video sequence by the first sub-network model based on the first instruction flag bit to obtain an initial video sequence of a second resolution, where the second resolution is higher than the first resolution.
[0123] In some embodiments, the normal decoder 1501 is further configured to parse the bitstream to obtain the second indication flag bit, the second indication flag bit being used to indicate that the first resolution video sequence needs to be enhanced in temporal resolution by the super-resolution network model; The video processing device 120 is further configured to perform a resolution enhancement process on the video sequence by the first sub-network model based on the second instruction flag bit to obtain an initial video sequence of a second resolution, the initial video sequence including the at least one video frame and at least one interpolated frame.
[0124] In some embodiments, the normal decoder 1501 is further configured to parse the bitstream to obtain the third indication flag bit, the third indication flag bit being used to indicate that the first resolution video sequence needs to be enhanced by the super-resolution network model; The video processing unit 120 is further configured to perform a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model based on the third indication flag bit to obtain the target video sequence.
[0125] In some embodiments, the normal decoder 1501 is further configured to analyze the bitstream to obtain model parameters of the super-resolution network model; The video processor 120 is further configured to construct the super-resolution network model based on the model parameters.
[0126] It should be noted that the functions of the video processing device 120 in the above embodiment can also be applied to the decoder 150. In this way, the decoder 150 can utilize the super-resolution network design to solve problems such as image quality loss and video frame rate reduction occurring in the current video compression process, and also use one network (i.e., the super-resolution network model) to simultaneously improve the resolution and frame rate of the video image. Therefore, when the super-resolution network model is applied to the processing process of video compression, the image quality of the compressed video can be greatly improved, and the resolution and frame rate of the video image can also be greatly improved. In addition, the video sequence can be downsampled before the compressed video is encoded, so that the amount of video data to be encoded can be reduced, and then the corresponding upsampling can be performed when the super-resolution network model is adopted after decoding, thereby reducing the bit rate, reducing the transmission bit stream, and improving the codec efficiency.
[0127] In some further embodiments of the present application, reference is made to FIG. 16, which is a schematic structural diagram of a video system according to an embodiment of the present application. As shown in FIG. 16, the video system 160 may include an encoder 161 and a decoder 150. Here, the encoder 161 may include a pre-processing device 1611 and a normal encoder 1612, and the decoder 150 may include a normal decoder 1501 and a video processing device 120 according to any of the above-mentioned embodiments. Here, the normal encoder 1612 is the video encoder 103 according to the above-mentioned embodiments, which may be an H.266 / VVC encoder, an H.265 / HEVC encoder, etc., and the normal decoder 1501 is the video decoder 105 according to the above-mentioned embodiments, which may be an H.266 / VVC decoder, an H.265 / HEVC decoder, etc.
[0128] In some embodiments, the pre-processing device 1611 is configured to, after receiving an input video sequence at a second resolution, pre-process the received input video sequence to obtain a video sequence at a first resolution; a standard encoder 1612 configured to perform video compression on the video sequence, generate a bitstream, and transmit the bitstream to the standard decoder; A normal decoder 1501 is configured to parse the bitstream to obtain a video sequence in the first resolution.
[0129] the video processing unit 120 is configured to input the video sequence into a super-resolution network model to obtain a target video sequence in a second resolution; Here, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to enhance the resolution of the video sequence, and the second sub-network model is used to enhance the image quality of at least one frame in the output result of the first sub-network model.
[0130] In some embodiments, the pre-processing unit 1611 is configured to perform a down-sampling process on the received input video sequence to obtain a first resolution video sequence, and set a first indication flag bit, the first indication flag bit being used to indicate that the first resolution video sequence needs to be enhanced in spatial resolution by the super-resolution network model; The regular encoder 1612 is further configured to write the first indication flag bit into the bitstream.
[0131] In some embodiments, the pre-processing unit 1611 is configured to perform a frame rate adjustment process on the received input video sequence to obtain a first resolution video sequence, and set a second indication flag bit, the second indication flag bit being used to indicate that the first resolution video sequence needs to have a temporal resolution of the video sequence enhanced by the super-resolution network model; The regular encoder 1612 is further configured to write the second indication flag bit into the bitstream.
[0132] In some embodiments, the normal encoder 1612 is further configured to, after performing video compression on the video sequence, determine whether the image quality of the compressed video sequence meets a predetermined image quality criterion, set a third indication flag bit, and write the third indication flag bit into a bitstream, where the third indication flag bit is used to indicate that the first resolution video sequence needs to have the image quality of the video sequence enhanced by the super-resolution network model.
[0133] In some embodiments, the preprocessing unit 1611 is further configured to obtain training samples, and use the training samples to train a neural network-based super-resolution network model to obtain model parameters of the super-resolution network model, where the training samples include at least a plurality of video frames at a first resolution and a plurality of video frames at a second resolution; The generic encoder 1612 is further configured to write the model parameters into a bitstream.
[0134] In some embodiments, the general decoder 1501 is further configured to parse the bitstream to obtain the first indication flag bit; The video processing device 120 is further configured to perform a resolution enhancement process on the video sequence by the first sub-network model based on the first instruction flag bit to obtain an initial video sequence of a second resolution, where the second resolution is higher than the first resolution.
[0135] In some embodiments, the normal decoder 1501 is further configured to parse the bitstream to obtain the second indication flag bit; The video processing device 120 is further configured to perform a resolution enhancement process on the video sequence by the first sub-network model based on the second instruction flag bit to obtain an initial video sequence of a second resolution, the initial video sequence including the at least one video frame and at least one interpolated frame.
[0136] In some embodiments, the normal decoder 1501 is further configured to parse the bitstream to obtain the third indication flag bit; The video processing unit 120 is further configured to perform a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model based on the third indication flag bit to obtain the target video sequence.
[0137] In some embodiments, the normal decoder 1501 is further configured to analyze the bitstream to obtain model parameters of the super-resolution network model; The video processor 120 is further configured to construct the super-resolution network model based on the model parameters.
[0138] It should be noted that the function of the video processing device 1200 in the above embodiment can also be applied to the video system 160. In this way, the video system 160 can utilize the super-resolution network design to solve problems such as image quality loss and video frame rate reduction occurring in the current video compression process, and also use one network (i.e., super-resolution network model) to simultaneously improve the resolution and frame rate of the video image. Therefore, when the super-resolution network model is applied to the processing process of video compression, the image quality of the compressed video can be greatly improved, and the resolution and frame rate of the video image can also be greatly improved. In addition, the video sequence can be downsampled before the compressed video is encoded, so that the amount of video data to be encoded can be reduced, and then the corresponding upsampling can be performed when the super-resolution network model is adopted after decoding, thereby reducing the bit rate, reducing the transmission bit stream, and improving the codec efficiency.
[0139] It should also be explained that in this application, the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a set of elements includes not only those elements, but also other elements not expressly listed or inherent elements of such process, method, article, or apparatus. Unless specifically limited, an element defined with the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0140] The numbers of the above embodiments of the present invention are merely for the purpose of explanation and do not indicate superiority or inferiority of the embodiments.
[0141] The methods disclosed in the several method embodiments provided herein may be combined in any manner without conflict to obtain new method embodiments.
[0142] Features disclosed in the several product embodiments provided herein may be combined in any non-conflicting manner to obtain new product embodiments.
[0143] The features disclosed in any method or apparatus embodiment provided herein may be combined in any non-conflicting manner to obtain new method or apparatus embodiments.
[0144] The above content is only a specific embodiment of the present application, and the protection scope of the present application is not limited thereto. Any modifications or replacements that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims. [Industrial Applicability]
[0145] In the present embodiment, the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to improve the resolution of the video sequence, and the second sub-network model is used to improve the image quality of at least one frame in the output result of the first sub-network model. In this way, the super-resolution network design can be used to solve problems such as image quality loss and video frame rate reduction occurring in the current video compression process, and one network (i.e., the super-resolution network model) is used to simultaneously improve the resolution and frame rate of the video image. Therefore, when the super-resolution network model is applied to the processing process of video compression, the image quality of the compressed video can be greatly improved, and the resolution and frame rate of the video image can also be greatly improved. In addition, the video sequence can be downsampled before encoding the compressed video, so that the amount of video data to be encoded can be reduced, and then the corresponding upsampling can be performed when the super-resolution network model is adopted after decoding, thereby reducing the bit rate, reducing the transmission bit stream, and improving the codec efficiency.
Claims
1. A video processing method applied to a video device, comprising: obtaining a video sequence at a first resolution, the video sequence including at least one video frame; inputting the video sequence into a super-resolution network model to obtain a target video sequence in a second resolution; the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to enhance the resolution of the video sequence, and the second sub-network model is used to enhance the image quality of at least one frame in an output result of the first sub-network model; The second sub-network model further includes a third sub-network model, and the video processing method further comprises: performing a resolution enhancement process on the video sequence by the first sub-network model to obtain an initial video sequence at a second resolution, the initial video sequence including the at least one video frame and at least one interpolated frame; performing a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model to obtain the target video sequence; performing a quality enhancement process on an interpolated frame in the target video sequence by the third sub-network model to obtain a target interpolated frame; updating the target video sequence based on the target interpolated frame. Video processing methods.
2. the resolution of the video sequence comprises at least one of a spatial resolution of the video sequence and a temporal resolution of the video sequence.
2. The video processing method of claim 1.
3. The second resolution is higher than the first resolution.
2. The video processing method of claim 1.
4. the first sub-network model is a branch-and-fuse super-resolution network model, and the second sub-network model is a quality enhancement network model; 2. The video processing method of claim 1.
5. the third sub-network model is an interpolated frame reinforcement network model; 2. The video processing method of claim 1.
6. The first sub-network model includes at least one branching module and one fusion module; performing a resolution enhancement process on the video sequence with the first sub-network model to obtain an initial video sequence at a second resolution, constructing an initial epipolar plane image (EPI) image set of at least one dimension based on the video sequence, where each dimensional initial EPI image set corresponds to one branching module; correspondingly inputting the at least one dimensional initial EPI image set to the at least one branching module to obtain at least one target EPI image set; performing, by the fusion module, a fusion process on the at least one target EPI image set to obtain the initial video sequence; a resolution of each EPI image in the initial EPI image set is a first resolution, and a resolution of each EPI image in the target EPI image set is a second resolution.
4. A video processing method according to claim 3.
7. Constructing an initial EPI image set of at least one dimension based on the video sequence includes: forming a stereoscopic video collection by stacking the video sequences, the dimensions of the stereoscopic video collection including a horizontal resolution dimension, a vertical resolution dimension, and a frame rate dimension; extracting at least one EPI image after performing at least one slicing operation on the stereoscopic video set in the one dimension of the stereoscopic video set based on one dimension of the stereoscopic video set, and constructing an initial EPI image set in the one dimension of the stereoscopic video set from the at least one EPI image; obtaining the at least one dimensional initial EPI image set based on at least one dimension of the stereoscopic video set.
7. A video processing method according to claim 6.
8. A computer storage medium having stored thereon a computer program which, when executed by a first processor, performs the method according to any one of claims 1 to 7.
9. A decoder comprising a normal decoder and a video processing device, the normal decoder is used to parse the bitstream to obtain a first resolution video sequence; the video processing device is configured to input the video sequence to a super-resolution network model to obtain a target video sequence in a second resolution; the super-resolution network model includes at least a first sub-network model and a second sub-network model, the first sub-network model is used to adjust the resolution of the video sequence, and the second sub-network model is used to adjust the image quality of at least one frame in an output result of the first sub-network model; The second sub-network model further includes a third sub-network model, and the video processing device further comprises: performing a resolution enhancement process on the video sequence by the first sub-network model to obtain an initial video sequence at a second resolution, the initial video sequence including the at least one video frame and at least one interpolated frame; performing a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model to obtain the target video sequence; performing a quality enhancement process on an interpolated frame in the target video sequence by the third sub-network model to obtain a target interpolated frame; updating the target video sequence based on the target interpolated frame. decoder.
10. The normal decoder is further configured to analyze the bitstream to obtain a first indication flag bit, the first indication flag bit being used to indicate that the first resolution video sequence needs to have its spatial resolution enhanced by the super-resolution network model; The video processing device is further configured to perform a resolution enhancement process on the video sequence by the first sub-network model based on the first indication flag bit to obtain the initial video sequence in a second resolution, the second resolution being higher than the first resolution. A decoder according to claim 9.
11. The normal decoder is further configured to parse the bitstream to obtain the second indication flag bit, the second indication flag bit being used to indicate that the first resolution video sequence needs to be enhanced in temporal resolution of the video sequence by the super-resolution network model; The video processing device is further configured to perform a resolution enhancement process on the video sequence by the first sub-network model based on the second indication flag bit to obtain the initial video sequence in a second resolution. A decoder according to claim 9.
12. The normal decoder is further configured to analyze the bitstream to obtain the third indication flag bit, the third indication flag bit being used to indicate that the first resolution video sequence needs to be enhanced by the super-resolution network model; The video processing device is further configured to perform a quality enhancement process on at least one frame in the initial video sequence by the second sub-network model based on the third indication flag bit to obtain the target video sequence. A decoder according to claim 9.
13. The normal decoder is further configured to analyze the bitstream to obtain model parameters of the super-resolution network model; the video processing device is further configured to construct the super-resolution network model based on the model parameters. A decoder according to claim 9.
Citation Information
Patent Citations
Image super-resolution method based on adaptive convolutional neural network
CN111161152A
Signal processing apparatus, signal processing method and program
JP2019164618A
