Method and system for enhancing the perceptual quality of compressed video based on multi-scale bidirectional convolution

Abstract

The application discloses a kind of based on multiscale bidirectional convolution's compressed video sensory quality enhancement method and system, comprising: the input multiple video frames are motion compensated, obtain the feature after alignment;The input multiple video frames include reference frame and the front and rear several adjacent frames of reference frame;Extract the timing information of feature after alignment, obtain the feature after space-time fusion;Extract the direction feature of feature after space-time fusion, obtain horizontal feature and vertical feature;Based on reference frame and adjacent frame to guide the recovery of horizontal feature and vertical feature, obtain the feature after detail attention mechanism processing;The feature after detail attention mechanism processing is enhanced, obtain enhanced direction feature;Enhanced direction feature is added with reference frame element by element, and enhanced frame is obtained.The application can improve the sensory quality of reconstructed video frame, and the reconstructed frame recovered is more in line with human visual system, and can improve the quality decline problem caused by compression.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video coding, in particular to a compressed video sensory quality enhancement method and system based on multi-scale bidirectional convolution. BACKGROUND

[0002] Nowadays, video content has become the main part of digital network traffic and is still growing. In order to transmit video under limited bandwidth, it is necessary to compress the video to significantly reduce the bit rate. However, the mainstream compression methods such as H.264 / AVC and H.265 / HEVC often introduce various artifacts in the compressed video, especially in the case of low bit rate, the existence of these artifacts greatly affects people's visual sense and some image processing tasks such as image recognition, target detection, etc. Therefore, it is necessary to enhance the compressed video.

[0003] The main processing method for the loss of coding process information in the video coding standard is loop filtering processing. However, the traditional loop filtering processing is limited by the computational complexity and cannot well improve the quality of the finally reconstructed video. In recent years, people have carried out post-processing on the decompressed low-quality video based on deep learning to improve the video quality without changing the coding framework. Unfortunately, the existing deep learning-based enhancement method mainly enhances the objective quality of the compressed video, ignoring the sensory quality. In fact, the human visual system (HVS) is the ultimate purpose of the compressed video, so the enhancement of the sensory quality is more important than the objective quality. Therefore, how to design a compressed video enhancement network that meets the human visual system has important significance and application value. SUMMARY

[0004] The main purpose of the present application is to overcome the above-mentioned defects in the prior art, and to provide a compressed video sensory quality enhancement method and system based on multi-scale bidirectional convolution, which can reconstruct high-quality video frames that meet the human visual system from low-quality decompressed video.

[0005] The present application adopts the following technical solutions:

[0006] On the one hand, a compressed video sensory quality enhancement method based on multi-scale bidirectional convolution, comprising:

[0007] motion compensation is performed on the input multiple video frames to obtain aligned features; the input multiple video frames include a reference frame and several adjacent frames before and after the reference frame;

[0008] extracting the temporal information of the aligned features to obtain spatio-temporal fused features;

[0009] extracting the directional features of the spatio-temporal fused features to obtain horizontal features and vertical features;

[0010] The recovery of horizontal features and vertical features is guided based on the reference frame and the adjacent frame, to obtain features processed by a detail attention mechanism;

[0011] The features processed by the detail attention mechanism are enhanced to obtain enhanced directional features; and the enhanced directional features are element-wise added to the reference frame to obtain an enhanced frame.

[0012] Preferably, the input multiple video frames are motion compensated to obtain aligned features, specifically including:

[0013] The input reference frame V t and the adjacent frame of the reference frame wherein N is a natural number, representing the number of front / back adjacent frames;

[0014] The multiple video frames are converted into multiple high-dimensional feature vectors by a U-net network, each dimension in the vector representing a feature or attribute of the video frame, and the U-net network outputs a motion offset offset;

[0015] The input frame V t and The concatenated information is input into a deformable convolution network along with the motion offset offset.

[0016] The deformable convolution network fuses the motion offset offset and the multiple video frames, so that the pixels of the multiple video frames move towards the corresponding offset offset, to obtain aligned multiple frame information.

[0017] Preferably, the motion offset offset is O, the fused and aligned features are F, and the motion compensation process is represented as follows:

[0018]

[0019]

[0020] wherein [·,·] represents concatenation in the channel dimension, u(·) represents a U-net operation, and DCN(·) represents a deformable convolution operation.

[0021] Preferably, the time sequence information of the aligned features is extracted to obtain spatio-temporally fused features, specifically including:

[0022] The spatio-temporal information of the alignment feature is extracted using two 3D convolution layers; each convolution layer includes multiple convolution kernels, each of which slides in the spatio-temporal dimension to extract spatio-temporal information of multiple video frames; in the first 3D convolution layer, each convolution kernel extracts features from multiple video frames, and after the features are averaged or summarized in the spatio-temporal dimension, a new set of features is generated; the new features are sent to the next 3D convolution layer for further extraction of spatio-temporal information;

[0023] Specifically, the spatio-temporally fused feature is represented as:

[0024] F 3d =Con3d(Conv3d(F))

[0025] Wherein, F 3d represents the spatio-temporally fused feature; F represents the fused alignment feature; Conv3(·) represents a 3D convolution operation.

[0026] Preferably, the direction feature of the spatio-temporally fused feature is extracted to obtain horizontal and vertical features, specifically including:

[0027] Taking the video frame including the spatio-temporally fused feature as input;

[0028] The horizontal and vertical direction features are extracted by bidirectional convolution; the bidirectional convolution includes mx1 and 1xm convolution; wherein the mx1 convolution extracts the horizontal direction feature, and the 1xm convolution extracts the vertical direction feature; the mx1 convolution and the 1xm convolution are stacked together to extract the horizontal and vertical direction features of the input video frame;

[0029] The horizontal direction feature extracted by the mx1 convolution and the vertical direction feature extracted by the 1xm convolution are spliced in the channel dimension, and the spliced information is sent to the multi-scale convolution to capture the geometric changes of the video frame using different size convolution kernels, and the direction feature is output.

[0030] Preferably, the reference frame and the adjacent frame are used to guide the recovery of the horizontal and vertical features to obtain the features processed by the detail attention mechanism, specifically including:

[0031] The reference frame, adjacent frame, horizontal direction feature and vertical direction feature are spliced in the channel dimension by the detail attention mechanism;

[0032] The maximum value and the sum of the spliced information in the channel dimension are taken respectively;

[0033] The feature map after taking the maximum value and the feature map after summing are spliced in the channel dimension, and feature fusion is performed by 1x1 convolution;

[0034] The fused feature map is output through a Sigmoid activation function, and finally the output result is element-wise added to the direction feature to obtain a feature processed through the detail attention mechanism.

[0035] Preferably, the horizontal feature, the vertical feature, the direction feature fused through the multi-scale convolution, and the feature processed through the detail attention mechanism are respectively represented as follows:

[0036] F H = c H (F 3d )

[0037] F V = c V (F 3d )

[0038] F f = M([F H , F V ])

[0039]

[0040] In the formula, F 3d represents the spatio-temporal fused feature; F H represents the horizontal feature; F V represents the vertical feature; F V represents the direction feature fused through the multi-scale convolution; F A represents the feature processed through the detail attention mechanism; c H (·) represents a 1xm convolution operation, c V (·) represents a mx1 convolution operation, M(·) represents a multi-scale convolution operation, DFA(·) represents a detail attention mechanism operation; V t represents a reference frame; represents a neighboring frame of the reference frame.

[0041] Preferably, the feature processed through the detail attention mechanism is enhanced to obtain an enhanced direction feature; the enhanced direction feature is element-wise added to the reference frame to obtain an enhanced frame, which specifically includes:

[0042] The feature processed through the detail attention mechanism is input into a network including a plurality of dense residual blocks to output the enhanced direction feature;

[0043] The enhanced direction feature is element-wise added to the reference frame to obtain the enhanced frame;

[0044] The enhanced frame is represented as follows:

[0045] F E = Res(F A)+V t

[0046] wherein, F E denotes an enhanced frame; F A denotes a feature processed by a detail attention mechanism; V t denotes a reference frame; Res(·) denotes a dense residual convolution operation.

[0047] In another aspect, a compressed video sensory quality enhancement system based on a multi-scale bidirectional convolution includes a generator and a discriminator based on a generative adversarial network; the generator includes a motion compensation network, a space-time fusion network, a multi-scale bidirectional network and a dense residual reconstruction network;

[0048] The motion compensation network is configured to perform motion compensation on inputted multiple video frames to obtain aligned features; the inputted multiple video frames include a reference frame and several adjacent frames before and after the reference frame;

[0049] The space-time fusion network is configured to extract time sequence information of the aligned features to obtain space-time fused features;

[0050] The multi-scale bidirectional network is configured to extract directional features of the space-time fused features to obtain horizontal features and vertical features; and is further configured to guide recovery of the horizontal features and the vertical features based on the reference frame and the adjacent frames to obtain features processed by a detail attention mechanism;

[0051] The dense residual reconstruction network is configured to enhance the features processed by the detail attention mechanism to obtain enhanced directional features; and is configured to perform element-wise addition of the enhanced directional features and the reference frame to obtain an enhanced frame.

[0052] Compared with the prior art, the present application has the following advantages:

[0053] (1) The present application can improve the sensory quality of reconstructed video frames; compared with traditional compressed video objective quality enhancement methods, the reconstructed frames recovered by the network are more consistent with the human visual system, and the quality degradation problem caused by compression can be significantly improved;

[0054] (2) The motion compensation process of the present application includes a U-net network and a deformable convolution network, which can make the motion between video frames smoother and reduce the influence of compressed video noise and artifacts, thereby improving the video quality;

[0055] (3) The two 3D convolutions of the present application can well capture the time sequence information between video frames and frames;

[0056] (4) The bidirectional convolution of the present application can effectively extract the horizontal and vertical features of the video frame, and the texture features of the bidirectional convolution can be controlled by using multiple convolution kernels with different lengths and widths, so as to obtain more detailed texture information; the multiscale convolution combined with the bidirectional convolution can capture the geometric changes of the video frame by using different size convolution kernels. Specifically, using a smaller convolution kernel can help capture some local geometric changes, such as the subtle movement or shape change of an object, while using a larger convolution kernel can help capture more extensive geometric changes, such as the overall movement or scaling of an object, and by combining convolution kernels of different sizes, the geometric changes in the video can be more comprehensively captured, thereby better understanding the motion and changes in the video;

[0057] (5) The present application sends the features processed by the detail attention mechanism into a network including multiple dense residual blocks, can output enhanced directional features, and adds the enhanced directional features and the reference frame element by element to reconstruct high-quality video frames; specifically, the dense residual refers to inputting the reconstruction error of each pixel as a residual into the next layer network when reconstructing the video frame, which can better preserve the details and texture information of the image; compared with the method without using dense residual, the dense residual can better reconstruct high-quality video frames, thereby improving the visual effect and viewing quality of the video. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The flowchart of the compression video sensory quality enhancement method based on multiscale bidirectional convolution of the embodiment of the present application is shown in the figure.

[0059] Figure 2 The network structure block diagram of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0060] The present application will be further described in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that after reading the content taught by the present application, those skilled in the art can make various modifications or changes to the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0061] In order to solve the problem of quality decline of reconstructed video caused by information loss in video coding process, the present embodiment proposes a compression video sensory quality enhancement method based on multiscale bidirectional convolution, which is used to reconstruct high-quality video frames conforming to human visual system from low-quality decompressed video.

[0062] Specifically, referring to Figure 1 The compression video sensory quality enhancement method based on multiscale bidirectional convolution, as shown in the figure, comprises:

[0063] S101, motion compensation is performed on the input multiple video frames to obtain aligned features; the input multiple video frames include a reference frame and several adjacent frames before and after the reference frame;

[0064] S102, time sequence information of the aligned features is extracted to obtain spatio-temporal fused features;

[0065] S103, directional features of the spatio-temporal fused features are extracted to obtain horizontal features and vertical features;

[0066] S104, the horizontal features and the vertical features are recovered based on the reference frame and the adjacent frames to obtain features processed by a detail attention mechanism;

[0067] S105, the features processed by the detail attention mechanism are enhanced to obtain enhanced directional features; and the enhanced directional features are added to the reference frame element by element to obtain an enhanced frame.

[0068] Referring to Figure 2 In the embodiment, the training set production, network model construction and network processing of the compressed video sensory quality enhancement method based on the multi-scale bidirectional convolution are as follows.

[0069] Step 1, data set production. The collected video is converted into YUV420 format supported by HEVC encoding, and then encoded on the standard test platform HM-16.5 of H.256 / HEVC in a low delay (LDP) configuration mode. In the encoding process, the quantization levels are set to QP=27, 32, 37 and 42 respectively, and four low-quality reconstructed video sequence training sets with different quantization levels are obtained.

[0070] Step 2, network model construction. The model of the embodiment adopts a generative adversarial network, including a generator and a discriminator. The generator, as the core of the whole method, is composed of four networks, namely a motion compensation network, a spatio-temporal fusion network, a multi-scale bidirectional network and a dense residual reconstruction network.

[0071] Step 3, motion compensation network processing.

[0072] Specifically, the reference frame V t and the adjacent frame V are taken as the input of the network. First, the multiple video frames are converted into multiple high-dimensional feature vectors by the U-net network. Each dimension in the vector represents a feature or attribute of the video frame. In this way, each video frame is represented as a point in a high-dimensional space, which makes them easier to compare and fuse. The output of the U-net is the motion offset offset. Then, the input frame V t and The splicing is performed in the channel dimension, and the spliced information and the motion offset offset are sent into a deformable convolution network. The network can fuse the motion offset offset and the multiple video frames, so that the pixels of the multiple video frames move towards the corresponding offset offset to obtain aligned multiple frame information. This process can make the motion between video frames smoother and reduce the influence of compressed video frame noise and artifacts, thereby improving the video quality. The motion offset offset is O, and the fused and aligned feature is F, which can be represented as:

[0073]

[0074]

[0075] Where [·,·] represents splicing in the channel dimension, u(·) represents the U-net operation, and DCN(·) represents the deformable convolution operation.

[0076] Referring to Figure 2 In the embodiment, the reference frame and the low-quality frames before and after the reference frame are taken as the input of the network. The five frames are mapped to a high-dimensional space through the U-net network, and then the motion offset is predicted through the deformable convolution. Finally, the motion offset and the five video frames are fused to obtain the aligned five frames.

[0077] Step 4, spatio-temporal fusion network processing. The temporal information of the video frames is fully considered, and two 3D convolutions are used to extract the spatio-temporal information of the aligned features.

[0078] Specifically, the aligned features F of the motion compensation network are taken as the input of the network, and then two 3D convolution layers are used. Each convolution layer includes multiple convolution kernels, and each convolution kernel slides in the spatio-temporal dimension to extract the spatio-temporal information of the multiple video frames. The spatio-temporal features are extracted through the convolution operation. In each convolution layer, each convolution kernel will extract features from multiple video frames, and these features will be averaged or summarized in the spatio-temporal dimension to generate a new set of features. These features will be sent to the next convolution layer for further extraction of spatio-temporal information. The spatio-temporal fused features are F 3d , which can be represented as:

[0079] F 3d = Con3d(Conv3d(F))

[0080] Where Conv3(·) represents the 3D convolution operation.

[0081] Step 5, multi-scale bidirectional network processing.

[0082] Specifically, the spatio-temporally fused features F 3dAs the input of the network. First, through the bidirectional convolution, the convolution is processed by using different long and wide convolution kernels, such as mx1 and 1xm convolution. The mx1 can analyze the horizontal direction features of the input video frame, and the 1xm convolution can analyze the vertical direction features of the input video frame. Each convolution kernel of the mx1 convolution kernel is an m row 1 column matrix, when it is convolved on the video frame, it will extract the features of each column in the video frame, so it can extract the horizontal direction features of the video frame. Similarly, each convolution kernel of the 1xm convolution kernel is a 1 row m column matrix, when it is convolved on the video frame, it will extract the features of each row in the video frame, so it can extract the vertical direction features of the video frame. The convolution kernel of mx1 and 1xm is stacked together, which can analyze the horizontal and vertical direction features of the input video frame. By controlling the size of n, the texture features of the bidirectional convolution are controlled, so as to obtain more fine high frequency information. Finally, the horizontal direction features extracted by mx1 convolution and the vertical direction features extracted by 1xm convolution are spliced in the channel dimension, and then the spliced information is sent into the multi-scale convolution to capture the geometric changes of the video frame, such as the distortion content in the low quality compressed video and the change of small objects (basketball in motion, etc.), and then the direction features are output. Finally, through the detail attention mechanism, the reference frame, the adjacent frame, the horizontal direction feature and the vertical direction feature are spliced in the channel dimension, and then the spliced information is respectively taken maximum value and summed in the channel dimension, and then the feature map after taking the maximum value and the feature map after summing are spliced in the channel dimension, and then the feature fusion is carried out through the 1x1 convolution, and then the fused feature map is output through the Sigmoid activation function, and finally the output result is added with the direction feature element by element. Let the horizontal feature and the vertical feature be F H and F V , the direction feature fused through the multi-scale convolution is F f , and the feature F A processed through the detail attention mechanism can be respectively represented as:

[0083] F H =c H (F 3d )

[0084] F V =c V (F 3d )

[0085] F f =M([F H ,F V ])

[0086]

[0087] Wherein, cH (·) represents a convolution operation of 1xm, c V (·) represents a convolution operation of mx1, M(·) represents a multi-scale convolution operation, DFA(·) represents a detail attention mechanism operation.

[0088] Step 6, target frame reconstruction.

[0089] Specifically, the residual information of the above enhancement network is added to the original reference frame to obtain the final reconstructed video frame. Specifically, the input frame is first passed through the motion compensation network to obtain the aligned feature F, then the aligned feature is sent into the space-time fusion network to obtain the fused information F 3d , then the fused information is sent into the multi-scale bidirectional convolution to extract the directional feature F f , then F f is sent into the detail attention mechanism to output F A , then F A is sent into the dense residual reconstruction network to output the enhanced directional feature, and finally the enhanced directional feature is element-wise added to the reference frame to obtain the enhanced frame. Let the enhanced frame be F E , which can be represented as:

[0090] F E = Res(F A ) + V t

[0091] Wherein, Res(·) represents a dense residual convolution operation.

[0092] The above is only a specific embodiment of the present application, but the design concept of the present application is not limited thereto, and any non-essential modification of the present application using this concept shall be deemed as an infringement of the protection scope of the present application.

Claims

1. A method for enhancing the perceptual quality of compressed video based on multi-scale bi-directional convolution, characterized in that, The method comprises the following steps: motion compensation is performed on the input multiple video frames to obtain aligned features; The input multiple video frames include a reference frame and several adjacent frames before and after the reference frame; time sequence information of the aligned features is extracted to obtain spatio-temporal fused features; directional features of the spatio-temporal fused features are extracted to obtain horizontal features and vertical features; a detail attention mechanism is used to guide the recovery of the horizontal features and the vertical features based on the reference frame and the adjacent frames to obtain features processed by the detail attention mechanism; the features processed by the detail attention mechanism are enhanced to obtain enhanced directional features; the enhanced directional features are added to the reference frame element by element to obtain an enhanced frame; the directional features of the spatio-temporal fused features are extracted to obtain horizontal features and vertical features, which specifically include: a video frame including the spatio-temporal fused features is taken as input; horizontal directional features and vertical directional features are extracted through bidirectional convolution; the bidirectional convolution includes mx1 and 1xm convolution; the mx1 convolution extracts the horizontal directional features, and the 1xm convolution extracts the vertical directional features; the mx1 convolution and the 1xm convolution are stacked together to extract the horizontal directional features and the vertical directional features of the input video frame; the horizontal directional features extracted by the mx1 convolution and the vertical directional features extracted by the 1xm convolution are spliced in the channel dimension, and the spliced information is input into a multi-scale convolution to capture the geometric changes of the video frame by using different size convolution kernels to output directional features; the detail attention mechanism is used to splice the reference frame, the adjacent frame, the horizontal directional features and the vertical directional features in the channel dimension; the maximum value and the sum of the spliced information in the channel dimension are respectively taken; the feature map after taking the maximum value and the feature map after summing are spliced in the channel dimension, and feature fusion is performed through 1x1 convolution; the fused feature map is output through a Sigmoid activation function, and finally the output result is added to the directional features element by element to obtain the features processed by the detail attention mechanism; the horizontal features, the vertical features, the directional features fused by the multi-scale convolution and the features processed by the detail attention mechanism are represented as follows: motion compensation is performed on the input multiple video frames to obtain aligned features, which specifically include: ； ； ； ； In the formula, denotes the spatio-temporal fused feature; denotes the horizontal feature; denotes the horizontal feature; denotes the direction feature fused by multi-scale convolution; denotes the feature processed by the detail attention mechanism; denotes the convolution operation of 1xm, denotes the convolution operation of mx1, denotes the multi-scale convolution operation, denotes the detail attention mechanism operation; denotes the reference frame; denotes the adjacent frame of the reference frame.

2. The method of claim 1, wherein the method is a method of enhancing the perceptual quality of compressed video based on multi-scale bi-directional convolution. a plurality of video frames are converted into a plurality of high-dimensional feature vectors by a U-net network, each dimension in the vector represents a feature or attribute of the video frame, and the U-net network outputs a motion offset; input reference frame and neighboring frames of the reference frame wherein, is a natural number, indicating the number of previous / succeeding neighboring frames; the deformable convolution network fuses the motion offset and the multiple video frames, so that the pixels of the multiple video frames move towards the corresponding offset to obtain aligned multiple frame information. concatenate the input frame and concatenate in channel dimension and feed the concatenated information with motion offset into the deformable convolution network; time sequence information of the aligned features is extracted to obtain spatio-temporal fused features, which specifically include:

3. The method for enhancing the sensory quality of compressed video based on multi-scale bidirectional convolution according to claim 2, characterized in that, let... The motion offset offset is , the aligned features after fusion are , and the process of motion compensation is represented as follows: ； ； wherein, denotes concatenation in the channel dimension, denotes a U-net operation, denotes a deformable convolution operation.

4. The method of claim 1, wherein the method is a method of enhancing the perceptual quality of compressed video based on multi-scale bi-directional convolution. ​ The spatio-temporal information of the alignment features is extracted using two 3D convolution layers; each convolution layer includes multiple convolution kernels, each of which slides in the spatio-temporal dimension to extract spatio-temporal information of multiple video frames; in the first 3D convolution layer, each convolution kernel extracts features from multiple video frames, and after the features are averaged or summarized in the spatio-temporal dimension, a new set of features is generated; the new features are sent to the next 3D convolution layer for further extraction of spatio-temporal information; Specifically, the spatio-temporally fused feature is represented as: ； wherein, represents the spatio-temporally fused features; represents the aligned fused features; represents a 3D convolution operation.

5. The method of claim 1, wherein, The feature processed by the detail attention mechanism is enhanced to obtain the enhanced directional feature; The enhanced directional feature is element-wise added to the reference frame to obtain the enhanced frame, specifically including: The feature processed by the detail attention mechanism is sent to a network including multiple dense residual blocks to output the enhanced directional feature; The enhanced directional feature is element-wise added to the reference frame to obtain the enhanced frame. The enhanced frame is represented as: ； wherein, denotes an enhanced frame; denotes a feature processed by a detailed attention mechanism; denotes a reference frame; denotes a dense residual convolution operation.

6. A multi-scale bidirectional convolution based compressed video sensory quality enhancement system based on the method of any one of claims 1-5. Based on the generative adversarial network, including a generator and a discriminator; the generator includes a motion compensation network, a spatio-temporal fusion network, a multi-scale bidirectional network and a dense residual reconstruction network; The motion compensation network is configured to perform motion compensation on the input multiple video frames to obtain aligned features; the input multiple video frames include a reference frame and several adjacent frames before and after the reference frame; The spatio-temporal fusion network is configured to extract the time sequence information of the aligned features to obtain spatio-temporally fused features; The multi-scale bidirectional network is configured to extract directional features of the spatio-temporally fused features to obtain horizontal features and vertical features; and is further configured to guide the recovery of the horizontal features and the vertical features based on the reference frame and the adjacent frames to obtain the feature processed by the detail attention mechanism; The dense residual reconstruction network is configured to enhance the feature processed by the detail attention mechanism to obtain the enhanced directional feature; The enhanced directional feature is element-wise added to the reference frame to obtain the enhanced frame.

Images