Video quality enhancement method, device, equipment and storage medium

Abstract

The present application discloses a video quality enhancement method, apparatus, device and storage medium, the video quality enhancement method comprising: obtaining a first video sequence to be enhanced; inputting the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module; based on the adjacent frame information fusion module, performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence to obtain a second video sequence after quality enhancement. The present application uses a pre-trained quality enhancement model to perform offset prediction and adjacent frame information fusion on each first video frame to be enhanced in the first video sequence to more accurately align the time domain features of each video frame with the adjacent frames, thereby performing subjective quality enhancement on the compressed video, which can reduce encoding costs and reduce encoding pressure while achieving the same encoding effect.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video quality enhancement method, apparatus, device and storage medium. Background Art

[0002] In the field of video image processing, captured video is often compressed for efficient signal transmission. However, this can lead to various distortions in the compressed video. Related technologies have proposed video quality enhancement algorithms to address this distortion. These algorithms are data-driven and perform the enhancement task by learning the mapping between the compressed video and the source video.

[0003] The video quality enhancement algorithm proposed in the related art does not take the motion of objects in the video into consideration when performing video frame alignment, resulting in poor video enhancement effect. Summary of the Invention

[0004] The main purpose of this application is to provide a video quality enhancement method, device, equipment and storage medium, aiming to solve the technical problem of low quality after video compression in the prior art.

[0005] To achieve the above objectives, the present application provides a video quality enhancement method, which includes:

[0006] Acquire a first video sequence to be enhanced;

[0007] Inputting the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module;

[0008] Based on the adjacent frame information fusion module, object offset prediction and adjacent frame information fusion processing are performed on each first video frame to be enhanced in the first video sequence to obtain a second video sequence with enhanced quality.

[0009] Optionally, the step of performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence with enhanced quality includes:

[0010] Determining, based on the adjacent frame information fusion module, a preset number of adjacent video frames for each first video frame to be enhanced in the first video sequence;

[0011] Performing object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences for the first video frame and adjacent video frames of the first video frame;

[0012] Based on the target prediction offset and the target offset confidence, deformable convolution is performed on the first video frame and the adjacent video frames of the first video frame to achieve information fusion of the first video frame and the adjacent video frames of the first video frame to obtain a second video sequence with enhanced quality.

[0013] Optionally, the step of performing object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences of the first video frame and adjacent video frames of the first video frame includes:

[0014] Using a preset multi-scale fully convolutional network to perform offset prediction on the first video frame and adjacent video frames of the first video frame according to classification, to obtain an initial predicted offset and an initial offset confidence for each category;

[0015] The initial prediction offset and the initial offset confidence of each category are fused to obtain the target prediction offset and the target offset confidence of each of the first video frames.

[0016] Optionally, the category is a motion state category of the object, including stillness, low-speed motion, and high-speed motion.

[0017] Optionally, the quality enhancement model further includes a spatial feature fusion module. After the step of performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence after quality enhancement, the method includes:

[0018] Based on the spatial feature fusion module, feature extraction is performed on the second video frame in the second video sequence to obtain feature information of the second video frame;

[0019] Performing feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features;

[0020] Based on the enhanced low-frequency features and high-frequency features, the feature information is reconstructed to obtain enhanced feature information, and based on the enhanced feature information, a third video sequence with enhanced quality is determined.

[0021] Optionally, the step of performing feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features includes:

[0022] Performing global average pooling on the feature information to obtain a global feature vector;

[0023] Performing feature adjustment on the global feature vector to obtain an adjusted global feature vector;

[0024] Extracting coarse-grained features of the feature information, and fusing the coarse-grained features with the adjusted global feature vector to obtain enhanced low-frequency features;

[0025] Subtracting the feature information from the low-frequency feature to obtain a high-frequency feature;

[0026] Asymmetric convolution is performed on the high-frequency features to obtain enhanced high-frequency features.

[0027] Optionally, the step of reconstructing the feature information based on the enhanced low-frequency features and high-frequency features to obtain enhanced feature information, and determining a third video sequence with enhanced quality based on the enhanced feature information includes:

[0028] Adding the enhanced low-frequency features and high-frequency features to obtain enhanced feature information;

[0029] Determining a prediction residual based on the enhanced feature information;

[0030] The prediction residual is added to the second video frame to obtain a third video frame with enhanced quality, and a third video sequence with enhanced quality is composed based on the third video frame.

[0031] The present application also provides a video quality enhancement device, the video quality enhancement device comprising:

[0032] An acquisition module, configured to acquire a first video sequence to be enhanced;

[0033] An input module, configured to input the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module;

[0034] The enhancement module is used to perform object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence with enhanced quality.

[0035] The present application also provides a video quality enhancement device, the video quality enhancement device comprising: a memory, a processor, and a program stored in the memory for implementing the video quality enhancement method.

[0036] The memory is used to store a program for implementing a video quality enhancement method;

[0037] The processor is configured to execute a program for implementing the video quality enhancement method, so as to implement the steps of the video quality enhancement method.

[0038] The present application also provides a storage medium, on which is stored a program for implementing the video quality enhancement method, and the program for implementing the video quality enhancement method is executed by a processor to implement the steps of the video quality enhancement method.

[0039] The present application provides a video quality enhancement method, apparatus, device and storage medium. Compared with the related art in which the motion of objects in the video is not taken into account when aligning video frames, resulting in poor video enhancement effect, in the present application, a first video sequence to be enhanced is obtained; the first video sequence is input into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module; based on the adjacent frame information fusion module, the offset of the object and the adjacent frame information fusion processing are performed on each first video frame to be enhanced in the first video sequence to obtain a second video sequence after quality enhancement. That is, in the present application, the offset of the object and the adjacent frame information fusion are performed on each first video frame to be enhanced in the first video sequence to be enhanced through the pre-trained quality enhancement model, so as to more accurately achieve the temporal feature alignment of each video frame with the adjacent frames, thereby performing subjective quality enhancement on the compressed video. Under the premise of achieving the same encoding effect, the encoding cost can be reduced and the encoding pressure can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the embodiments or the description of the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without inventive work.

[0041] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application;

[0042] Figure 2 This is a flowchart of the first embodiment of the video quality enhancement method of the present application;

[0043] Figure 3 This is a module diagram of the video quality enhancement device of this application;

[0044] Figure 4 This is a flow chart of the adjacent frame information fusion module in the video quality enhancement method of this application;

[0045] Figure 5This is a flow chart of the spatial feature fusion module of the second embodiment of the video quality enhancement method of this application;

[0046] Figure 6 This is a flow chart of the overall solution of the second embodiment of the video quality enhancement method of this application.

[0047] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0049] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present application.

[0050] The terminal in the embodiment of the present application can be a PC, or it can be a smart phone, tablet computer, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3) player, MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) player, portable computer and other portable terminal devices with display function.

[0051] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0052] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal can also be configured with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0053] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0054] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating device, a network communication module, a user interface module, and a video quality enhancement program.

[0055] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the video quality enhancement program stored in the memory 1005.

[0056] Reference Figure 2 , an embodiment of the present application provides a video quality enhancement method, the video quality enhancement method comprising:

[0057] Step S100, obtaining a first video sequence to be enhanced;

[0058] Step S200: inputting the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module;

[0059] Step S300 : Based on the adjacent frame information fusion module, object offset prediction and adjacent frame information fusion processing are performed on each first video frame to be enhanced in the first video sequence to obtain a second video sequence with enhanced quality.

[0060] In this embodiment, the application scenarios are:

[0061] As an example, a scenario for video quality enhancement may be that after video compression, the video is distorted and the compressed video needs to be quality enhanced. In the related art, a video quality enhancement algorithm is proposed to solve the distortion problem of compressed video, wherein the video quality enhancement algorithm is data-driven and performs the enhancement task by learning the mapping between the compressed video and the source video. The video quality enhancement algorithm proposed in the related art does not take into account the movement of objects in the video when aligning the video frames, resulting in poor video enhancement effect. For this scenario, the video quality enhancement method of this embodiment uses a pre-trained quality enhancement model to predict the offset of the object and fuse the information of adjacent frames for each first video frame to be enhanced in the first video sequence to be enhanced, so as to more accurately align the time domain features of each video frame with the adjacent frames, thereby performing subjective quality enhancement on the compressed video. Under the premise of achieving the same encoding effect, the encoding cost can be reduced and the encoding pressure can be reduced.

[0062] As an example, the application scenario of video quality enhancement is not only the above-mentioned quality enhancement processing of compressed video, but also includes various video quality enhancement scenarios, which are not specifically limited here.

[0063] This embodiment aims to improve the quality of compressed video.

[0064] In this embodiment, the video quality enhancement method is applied to a video quality enhancement device.

[0065] The specific steps are as follows:

[0066] Step S100, obtaining a first video sequence to be enhanced;

[0067] In this embodiment, the first video sequence to be enhanced is a compressed video sequence consisting of multiple video frames, and may also be other video sequences requiring video enhancement, which is not specifically limited here.

[0068] In this embodiment, the apparatus may obtain the first video sequence to be enhanced by receiving a compressed video (video sequence) uploaded by a user.

[0069] Step S200: inputting the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module;

[0070] In this embodiment, the quality enhancement model is a convolutional neural network model that includes an adjacent frame information fusion module. This model is obtained through deep learning training. The term "neural network" (NN) refers to an artificial neural network, not a biological neural network. A neural network is a computational model composed of a large number of interconnected nodes (or neurons). In an artificial neural network, neuronal processing units can represent different objects, such as features, letters, concepts, or meaningful abstract patterns. Processing units in the network are divided into three types: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the system's processing results; and hidden units are located between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information are reflected in the connections between the network processing units. Artificial neural networks are a non-programmed, brain-like information processing method. Their essence is to achieve parallel and distributed information processing capabilities through network transformations and dynamic behavior, mimicking the information processing functions of the human brain to varying degrees and levels. Currently, in the field of video processing, commonly used neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), fully connected networks, etc.

[0071] Furthermore, the Convolutional Neural Network (CNN) is a feedforward neural network and one of the most representative network structures in deep learning technology. Its artificial neurons can respond to surrounding units within a certain coverage area, and it performs well in large-scale image processing. The basic structure of a CNN consists of two layers. The first is the feature extraction layer (also known as the convolutional layer). The input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local feature is extracted, its positional relationship with other features is also determined. The second is the feature mapping layer (also known as the activation layer). Each computational layer of the network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, because neurons on a mapping surface share weights, the number of free parameters of the network is reduced. One of the advantages of CNN over traditional image processing algorithms is that it avoids complex pre-processing of images (such as extracting artificial features) and can directly input raw images for end-to-end learning. One of the advantages of CNN over traditional neural networks is that traditional neural networks use a fully connected approach, meaning that all neurons from the input layer to the hidden layer are fully connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNN avoids this difficulty through methods such as local connections and weight sharing.

[0072] In this embodiment, the device inputs the first video sequence into a preset quality enhancement model, and performs quality enhancement processing on it based on the quality enhancement model and the adjacent frame information fusion module in the model, wherein the adjacent frame information fusion module refers to aligning the features of the video frame to be enhanced (i.e., each video frame in the first video sequence) with its adjacent video frames, increasing the amount of information of the video frame to be enhanced, thereby performing subjective quality enhancement on the compressed video.

[0073] Step S300 : Based on the adjacent frame information fusion module, object offset prediction and adjacent frame information fusion processing are performed on each first video frame to be enhanced in the first video sequence to obtain a second video sequence with enhanced quality.

[0074] In this embodiment, the device performs object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence with enhanced quality. That is, through the pre-trained quality enhancement model, the object offset prediction and adjacent frame information fusion are performed on each first video frame to be enhanced in the first video sequence to be enhanced, and the time domain features of each video frame and the adjacent frames are more accurately aligned, so as to perform subjective quality enhancement on the compressed video. Under the premise of achieving the same encoding effect, the encoding cost can be reduced and the encoding pressure can be reduced.

[0075] Specifically, the step S300 includes the following steps S310-S330:

[0076] Step S310: determining a preset number of adjacent video frames for each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module;

[0077] In this embodiment, the device determines a preset number of adjacent video frames for each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module. Specifically, for the first video sequence, the enhancement of each frame requires the information of the current frame and a preset number of video frames before and after the current frame. If the number of frames before or after the current frame does not meet the preset number (such as the first video frame or the last video frame), the first frame or the last frame needs to be reused to ensure the number of input frames. For example, if the preset number is 6, the device determines the three video frames before and after each first video frame as adjacent frames. Further, the first video sequence includes 7 first video frames to be enhanced, namely x1-x7. When x4 is enhanced, the adjacent video frames of x4 are x1-x3 and x5-x7; when x2 is enhanced, the adjacent video frames of x2 are 3 x1 and x3-x5; when x3 is enhanced, the adjacent video frames of x3 are 2 x1, x2 and x4-x6.

[0078] Step S320: performing object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences for the first video frame and adjacent video frames of the first video frame.

[0079] In this embodiment, the device predicts the offset of the object for the first video frame and the adjacent video frames of the first video frame, and obtains the target prediction offset offset and the target offset confidence mask for the first video frame and the adjacent video frames of the first video frame, that is, by predicting the object offset of each video frame and the corresponding adjacent frames, the time domain feature alignment of each video frame and the adjacent frames is achieved more accurately, thereby increasing the amount of information of each first video frame, that is, enhancing the quality of the first video frame, wherein the offset prediction method includes but is not limited to the following methods: (1) extracting features from the input video sequence and sending it to the multi-scale U-Net network structure for offset prediction; (2) using the multi-scale U-N et++ structure network to predict the offset and mask of deformable convolution; (3) send the input sequence to multiple U-Net structure networks of different depths, and predict different offsets and masks on different U-Net structures; (4) send the input sequence to a single U-Net structure network to predict the offset and mask; (5) send the input sequence to multiple U-Net++ structure networks of different depths, and predict different offsets and masks on different U-Net structures; (6) send the input sequence to a single U-Net++ structure network to predict the offset and mask.

[0080] Specifically, the step S320 includes the following steps S321-S322:

[0081] Step S321: Using a preset multi-scale fully convolutional network, perform offset prediction on the first video frame and adjacent video frames of the first video frame according to classification, and obtain an initial predicted offset and initial offset confidence for each category;

[0082] In this embodiment, the categories are motion state categories of the object. Specifically, the motion state categories include stationary, low-speed motion, and high-speed motion.

[0083] In this embodiment, the apparatus uses a preset multi-scale fully convolutional network to perform classification offset prediction on the first video frame and the adjacent video frames of the first video frame, and obtains the initial predicted offset and initial offset confidence of each category. Specifically, referring to Figure 4, the device uses a multi-scale network structure to extract corresponding features on network branches of different depths, and finally obtains the motion offset and mask related to the receptive field. Since the receptive fields of network branches of different depths are different, shallow networks are more likely to obtain the offsets of static and small moving objects, while deep networks are more likely to obtain the offsets of large moving objects. For example, the offset obtained on the shallow network branch is only extracted by a convolution operation with a step size of 1, while the offset predicted on the deep network branch requires a convolution with a step size of 1, a convolution with a step size of 2, bilinear upsampling and channel splicing operations to be processed together. In this embodiment, the motion state of the object is divided into three categories: static, low-speed motion and fast motion, and three types of offsets and masks are predicted accordingly, that is, three scales (or depths) of network branches are used to classify the offset predictions of the first video frame and the adjacent video frames of the first video frame. The specific network structure MCDConv is expressed as follows:

[0084] MCDConv(I)=DConv(I,offset mc (I), mask mc (I))

[0085] offset mc (I)=fuse(offset1,offset2,offset3)

[0086] mask mc (I)=fuse(mask1,mask2,mask3)

[0087] Where I is the first video frame sequence of the input, DConv stands for deformable convolution v2, fuse is 1×1 convolution, offset mc It is the target offset obtained by fusion of three types of offsets, mask mc It is the target offset confidence obtained by fusing the three types of offset confidence masks, offseti and maski are the predicted initial offset offset and initial offset confidence mask.

[0088] Step S322 : Fusing the initial prediction offset and initial offset confidence of each category to obtain a target prediction offset and target offset confidence of each first video frame.

[0089] In this embodiment, the device fuses the initial prediction offset and initial offset confidence of each category to obtain the target prediction offset and target offset confidence of each first video frame, specifically referring to the offset in the above MCDConv. mc (I) and maskmc (I).

[0090] Step S330: Based on the target prediction offset and the target offset confidence, deformable convolution is performed on the first video frame and the adjacent video frames of the first video frame to achieve information fusion of the first video frame and the adjacent video frames of the first video frame, so as to obtain a second video sequence with enhanced quality.

[0091] In this embodiment, the device performs deformable convolution on the first video frame and the adjacent video frames of the first video frame based on the target prediction offset and the target offset confidence to achieve information fusion of the first video frame and the adjacent video frames of the first video frame, and obtain a second video sequence with enhanced quality. Specifically, after obtaining three types of offsets and masks, deformable convolution v2 is used to convolve the input sequence to fuse the information of adjacent frames, thereby achieving more accurate alignment of the features of the same object in different frames, increasing the amount of information in the video frames, and thereby achieving subjective quality enhancement of the compressed video.

[0092] The present application provides a video quality enhancement method. Compared with the related art in which the motion of objects in the video is not taken into account when aligning video frames, resulting in poor video enhancement effect, in the present application, a first video sequence to be enhanced is obtained; the first video sequence is input into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module; based on the adjacent frame information fusion module, the offset of the object is predicted and the adjacent frame information is fused for each first video frame to be enhanced in the first video sequence to obtain a second video sequence after quality enhancement. That is, in the present application, the offset of the object is predicted and the adjacent frame information is fused for each first video frame to be enhanced in the first video sequence to be enhanced through a pre-trained quality enhancement model, so as to more accurately achieve the alignment of the time domain features of each video frame with the adjacent frames, thereby performing subjective quality enhancement on the compressed video. Under the premise of achieving the same encoding effect, the encoding cost can be reduced and the encoding pressure can be reduced.

[0093] Based on the above first embodiment, the present application also provides another embodiment, wherein the video quality enhancement method includes:

[0094] After the step S300 of performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence with enhanced quality, the method includes the following steps A100-A300:

[0095] Step A100: extracting features from a second video frame in the second video sequence based on the spatial feature fusion module to obtain feature information of the second video frame;

[0096] In this embodiment, the quality enhancement model also includes a spatial feature fusion module, wherein the spatial feature fusion module is used to further perform feature fusion of video frames, increase the amount of information of each video frame, and thereby achieve subjective quality enhancement of the compressed video.

[0097] Step A200, performing feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features;

[0098] In this embodiment, the device performs feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features, wherein feature separation is to separate the low-frequency features and high-frequency features in the feature information, and then perform corresponding feature enhancement on features of different categories, so as to more effectively restore the details of the image (or the distorted area), increase the amount of information in the video frame, and thereby achieve the enhancement of the subjective quality of the compressed video.

[0099] In this embodiment, the feature separation method includes but is not limited to the following methods: (1) using low-pass filtering to obtain low-frequency features by fusing features of adjacent frame information, and obtaining high-frequency features by feature subtraction; (2) using high-pass filtering to obtain high-frequency features by fusing features of adjacent frame information, and obtaining low-frequency features by feature subtraction; (3) using pooling and convolution operations to obtain low-frequency features by fusing features of adjacent frame information, and obtaining high-frequency features by feature subtraction; (4) using convolution operations to obtain high-frequency features by fusing features of adjacent frame information, and obtaining low-frequency features by feature subtraction; (5) using Octave convolution operations to obtain low-frequency features and high-frequency features by fusing features of adjacent frame information.

[0100] In this embodiment, the feature enhancement method is: performing feature adjustment of the global feature vector on the low-frequency features and performing asymmetric convolution on the high-frequency features.

[0101] Specifically, the step A200 includes the following steps A210-A250:

[0102] Step A210, performing global average pooling on the feature information to obtain a global feature vector;

[0103] In this embodiment, the device performs global average pooling on the feature information to obtain a global feature vector. Specifically, the device first performs a pooling operation on the feature information to obtain the global feature information, that is, uses global average pooling on the feature information to obtain a one-dimensional global feature vector.

[0104] Step A220, performing feature adjustment on the global feature vector to obtain an adjusted global feature vector;

[0105] In this embodiment, the device performs feature adjustment on the global feature vector to obtain an adjusted global feature vector. Specifically, the device performs 1×1 convolution on the global feature vector to achieve the purpose of global feature adjustment (low-frequency feature enhancement). After adjustment, its feature size is expanded to the original size to facilitate the feature fusion operation of the subsequent step A230.

[0106] Step A230: extracting coarse-grained features of the feature information, and fusing the coarse-grained features with the adjusted global feature vector to obtain enhanced low-frequency features;

[0107] In this embodiment, the device extracts the coarse-grained features of the feature information, and fuses the coarse-grained features with the adjusted global feature vector to obtain enhanced low-frequency features. Specifically, the device uses two 3×3 convolutions to extract the coarse-grained features of the input features, and fuses them with the adjusted global feature vector. This is equivalent to adding a layer of coarse texture to the global features, and the fused result is regarded as the enhanced low-frequency feature.

[0108] Step A240, subtracting the feature information from the low-frequency feature to obtain a high-frequency feature;

[0109] In this embodiment, the feature information is divided into low-frequency features and high-frequency features, so the device subtracts the feature information from the low-frequency features to obtain the high-frequency features.

[0110] Step A250: performing asymmetric convolution on the high-frequency features to obtain enhanced high-frequency features.

[0111] In this embodiment, the device performs asymmetric convolution (ACB) on the high-frequency features to obtain enhanced high-frequency features, which is beneficial to the enhancement and restoration of details.

[0112] Step A300: reconstructing the feature information based on the enhanced low-frequency features and high-frequency features to obtain enhanced feature information, and determining a third video sequence with enhanced quality based on the enhanced feature information.

[0113] In this embodiment, the device reconstructs the feature information based on the enhanced low-frequency features and high-frequency features to obtain enhanced feature information, and determines a third video sequence with enhanced quality based on the enhanced feature information, that is, the enhanced low-frequency features and high-frequency features are added to reconstruct the feature information, and the feature separation, feature enhancement and reconstruction operations are repeated a preset number of times in the spatial feature fusion module to enhance the processing effect, wherein the preset number of times is preferably 8 times.

[0114] In this embodiment, the device separates the low-frequency features and high-frequency features of the video frame, and then performs corresponding feature enhancement on the separated features, thereby effectively restoring the details of the image, further increasing the amount of information in the video frame, and thereby enhancing the subjective quality of the compressed video. In this way, while achieving the same encoding effect, the encoding cost can be reduced and the encoding pressure can be reduced.

[0115] Specifically, the step A300 includes the following steps A310 to A330:

[0116] Step A310, adding the enhanced low-frequency features and high-frequency features to obtain enhanced feature information;

[0117] Step A320: determining a prediction residual based on the enhanced feature information;

[0118] Step A330: Add the prediction residual to the second video frame to obtain a third video frame with enhanced quality, and compose a third video sequence with enhanced quality based on the third video frame.

[0119] In this embodiment, the prediction residual refers to the residual between the video frame before enhancement and the video frame after enhancement, that is, the difference in the amount of information of the video frames. The prediction residual is added to the second video frame to obtain a third video frame with enhanced quality. After the quality of each video frame is enhanced, a third video sequence with enhanced quality is formed.

[0120] In this embodiment, the network structure of the spatial feature fusion module is as follows:

[0121] F l =conv1([expand(conv1(pool(F in ))),conv3(F in )])

[0122] F h =ACB(F in -F l )

[0123] F out =F l +F h

[0124] Among them, conv1 and conv3 represent 1×1 convolution and 3×3 convolution, pool represents global average pooling, expand is the feature map size expansion function, ACB is asymmetric convolution, F in is the total feature information, F l is the low-frequency feature, F h is the high frequency feature, Fout It is the enhanced feature information.

[0125] In the spatial feature fusion module, 8 SFDM cascades and a convolution with a stride of 1 are used to obtain the prediction residual of the current frame (i.e., feature separation, feature enhancement and reconstruction operations are repeated 8 times), and the quality of a single frame is enhanced by adding it to the intermediate frame of the input sequence.

[0126] Reference Figure 5 For spatial feature fusion, step 1, use global average pooling and convolution operations to adjust the global features of the fused features of adjacent frame information; step 2, use convolution operations to extract coarse-grained features of the fused features of adjacent frame information; step 3, scale the adjusted global features and fuse them with the coarse-grained features to obtain low-frequency features; step 4, use the features before processing to subtract the low-frequency features obtained in step 3 to obtain high-frequency features; step 5, use ACB to process the high-frequency features; step 6, add the processed low-frequency features and high-frequency features to reconstruct the features; step 7, repeat steps 1-6 7 times to obtain the final target reconstructed features; step 8, convolve the final target reconstructed features to obtain the current frame prediction residual; step 9, add the residual to the input current frame to obtain the enhanced result of the current frame.

[0127] In this embodiment, a spatial frequency domain decomposition module is designed in the process of spatial domain feature fusion of video images to effectively decompose the features into low-frequency features and high-frequency features, and perform corresponding feature processing on them, so that the algorithm can more effectively remove compression artifacts and restore image details, thereby achieving the subjective quality enhancement of the compressed video. While achieving the same encoding effect, the encoding cost can be reduced and the encoding pressure can be reduced.

[0128] In this embodiment, the adjacent frame information fusion module and the spatial feature fusion module are combined, referring to Figure 6 The input of the network is a video sequence, which consists of the current frame to be enhanced and the three adjacent frames before and after it (a total of 7 frames). The output is the enhanced result of the intermediate frame. In the adjacent frame information fusion module, we propose a motion classification deformable convolution (MCDConv) algorithm to more accurately fuse the features of objects with different motion speeds in adjacent frames. After time domain processing, the features of the fused multi-frame information are input into the spatial feature fusion module for further processing. In order to decompose the features into low-frequency features and high-frequency features for different levels of processing, the spatial frequency domain decomposition module (SFDM) is proposed as the basic processing unit in the spatial domain. Finally, the processed features are fused to obtain the final prediction residual, which is then added to the input intermediate frame to reconstruct a high-quality intermediate frame, thereby achieving quality enhancement of the video frame.

[0129] The present application also provides a video quality enhancement device, the video quality enhancement device comprising:

[0130] An acquisition module 10 is configured to acquire a first video sequence to be enhanced;

[0131] An input module 20, configured to input the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module;

[0132] The enhancement module 30 is configured to perform object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence with enhanced quality.

[0133] Optionally, the enhancement module includes:

[0134] an adjacent frame determination module, configured to determine, based on the adjacent frame information fusion module, a preset number of adjacent video frames for each first video frame to be enhanced in the first video sequence; a prediction module, configured to perform object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences for the first video frame and adjacent video frames of the first video frame; and a convolution module, configured to perform deformable convolution on the first video frame and adjacent video frames of the first video frame based on the target prediction offsets and target offset confidences to achieve information fusion of the first video frame and adjacent video frames of the first video frame, thereby obtaining a second video sequence with enhanced quality;

[0135] Optionally, the prediction module includes: a classification prediction module, configured to use a preset multi-scale fully convolutional network to perform offset prediction on the first video frame and adjacent video frames of the first video frame according to classification, to obtain an initial predicted offset and an initial offset confidence of each category; an offset fusion module, configured to fuse the initial predicted offset and the initial offset confidence of each category, to obtain a target predicted offset and a target offset confidence of each first video frame;

[0136] Optionally, the video quality enhancement device further includes: a feature extraction module, configured to perform feature extraction on the second video frame in the second video sequence based on the spatial feature fusion module to obtain feature information of the second video frame; a feature separation module, configured to perform feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features; and a spatial feature fusion module, configured to reconstruct the feature information based on the enhanced low-frequency features and high-frequency features to obtain enhanced feature information, and determine a third video sequence with enhanced quality based on the enhanced feature information.

[0137] Optionally, the feature separation module includes: a pooling module for performing global average pooling on the feature information to obtain a global feature vector; a feature adjustment module for performing feature adjustment on the global feature vector to obtain an adjusted global feature vector; a coarse-grained feature extraction module for extracting coarse-grained features of the feature information and fusing the coarse-grained features with the adjusted global feature vector to obtain enhanced low-frequency features; a high-frequency feature determination module for subtracting the feature information from the low-frequency features to obtain high-frequency features; and an asymmetric convolution module for performing asymmetric convolution on the high-frequency features to obtain enhanced high-frequency features.

[0138] Optionally, the spatial feature fusion module includes: a reconstruction module, used to add the enhanced low-frequency features and high-frequency features to obtain enhanced feature information; a residual prediction module, used to determine the prediction residual based on the enhanced feature information; a quality enhancement module, used to add the prediction residual to the second video frame to obtain a quality-enhanced third video frame, and based on the third video frame, form a quality-enhanced third video sequence.

[0139] The specific implementation of the video quality enhancement device of the present application is basically the same as the embodiments of the above-mentioned video quality enhancement method, and will not be repeated here.

[0140] Reference Figure 1 , Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present application.

[0141] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0142] Optionally, the video quality enhancement device may further include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. The rectangular user interface may include a display and an input submodule such as a keyboard. Optionally, the rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WiFi interface).

[0143] Those skilled in the art will understand that Figure 1 The structure of the video quality enhancement device shown in the figure does not constitute a limitation to the video quality enhancement device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0144] like Figure 1 As shown, memory 1005, which serves as a storage medium, may include an operating system, a network communication module, and a video quality enhancement program. The operating system is a program that manages and controls the hardware and software resources of the video quality enhancement device and supports the operation of the video quality enhancement program and other software and / or programs. The network communication module is used to enable communication between the various components within memory 1005, as well as communication with other hardware and software in the video quality enhancement system.

[0145] exist Figure 1 In the video quality enhancement device shown, the processor 1001 is configured to execute the video quality enhancement program stored in the memory 1005 to implement the steps of any one of the above-mentioned video quality enhancement methods.

[0146] The specific implementation of the video quality enhancement device of the present application is basically the same as the embodiments of the above-mentioned video quality enhancement method, and will not be repeated here.

[0147] The present application further provides a storage medium, wherein a program for implementing a video quality enhancement method is stored on the storage medium, and the program for implementing a video quality enhancement method is executed by a processor to implement the following video quality enhancement method:

[0148] Acquire a first video sequence to be enhanced;

[0149] Inputting the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module;

[0150] Based on the adjacent frame information fusion module, object offset prediction and adjacent frame information fusion processing are performed on each first video frame to be enhanced in the first video sequence to obtain a second video sequence with enhanced quality.

[0151] Optionally, the step of performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence with enhanced quality includes:

[0152] Determining, based on the adjacent frame information fusion module, a preset number of adjacent video frames for each first video frame to be enhanced in the first video sequence;

[0153] Performing object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences for the first video frame and adjacent video frames of the first video frame;

[0154] Based on the target prediction offset and the target offset confidence, deformable convolution is performed on the first video frame and the adjacent video frames of the first video frame to achieve information fusion of the first video frame and the adjacent video frames of the first video frame to obtain a second video sequence with enhanced quality.

[0155] Optionally, the step of performing object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences of the first video frame and adjacent video frames of the first video frame includes:

[0156] Using a preset multi-scale fully convolutional network to perform offset prediction on the first video frame and adjacent video frames of the first video frame according to classification, to obtain an initial predicted offset and an initial offset confidence for each category;

[0157] The initial prediction offset and the initial offset confidence of each category are fused to obtain the target prediction offset and the target offset confidence of each of the first video frames.

[0158] Optionally, the category is a motion state category of the object, including stillness, low-speed motion, and high-speed motion.

[0159] Optionally, the quality enhancement model further includes a spatial feature fusion module. After the step of performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence after quality enhancement, the method includes:

[0160] Based on the spatial feature fusion module, feature extraction is performed on the second video frame in the second video sequence to obtain feature information of the second video frame;

[0161] Performing feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features;

[0162] Based on the enhanced low-frequency features and high-frequency features, the feature information is reconstructed to obtain enhanced feature information, and based on the enhanced feature information, a third video sequence with enhanced quality is determined.

[0163] Optionally, the step of performing feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features includes:

[0164] Performing global average pooling on the feature information to obtain a global feature vector;

[0165] Performing feature adjustment on the global feature vector to obtain an adjusted global feature vector;

[0166] Extracting coarse-grained features of the feature information, and fusing the coarse-grained features with the adjusted global feature vector to obtain enhanced low-frequency features;

[0167] Subtracting the feature information from the low-frequency feature to obtain a high-frequency feature;

[0168] Asymmetric convolution is performed on the high-frequency features to obtain enhanced high-frequency features.

[0169] Optionally, the step of reconstructing the feature information based on the enhanced low-frequency features and high-frequency features to obtain enhanced feature information, and determining a third video sequence with enhanced quality based on the enhanced feature information includes:

[0170] Adding the enhanced low-frequency features and high-frequency features to obtain enhanced feature information;

[0171] Determining a prediction residual based on the enhanced feature information;

[0172] The prediction residual is added to the second video frame to obtain a third video frame with enhanced quality, and a third video sequence with enhanced quality is composed based on the third video frame.

[0173] The specific implementation of the storage medium of the present application is basically the same as the embodiments of the above-mentioned video quality enhancement method, and will not be repeated here.

[0174] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video quality enhancement method when executed by a processor.

[0175] The specific implementation of the computer program product of the present application is basically the same as the embodiments of the above-mentioned video quality enhancement method, and will not be repeated here.

[0176] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0177] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0178] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0179] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video quality enhancement method, characterized in that: The video quality enhancement method comprises: Acquire a first video sequence to be enhanced; Inputting the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module; Based on the adjacent frame information fusion module, perform object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence to obtain a second video sequence with enhanced quality; The step of predicting the offset of an object for each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module includes: Determining, based on the adjacent frame information fusion module, a preset number of adjacent video frames for each first video frame to be enhanced in the first video sequence; Performing object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences for the first video frame and adjacent video frames of the first video frame; The step of predicting the offset of the object for the first video frame and adjacent video frames of the first video frame includes: A preset multi-scale fully convolutional network is used to perform offset prediction on the first video frame and adjacent video frames of the first video frame according to classification, to obtain an initial predicted offset and an initial offset confidence for each category.

2. The video quality enhancement method according to claim 1, wherein: The step of performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence with enhanced quality includes: Based on the target prediction offset and the target offset confidence, deformable convolution is performed on the first video frame and the adjacent video frames of the first video frame to achieve information fusion of the first video frame and the adjacent video frames of the first video frame to obtain a second video sequence with enhanced quality.

3. The video quality enhancement method according to claim 1, wherein: The step of performing object offset prediction on the first video frame and adjacent video frames of the first video frame to obtain target prediction offsets and target offset confidences for the first video frame and adjacent video frames of the first video frame further includes: The initial prediction offset and the initial offset confidence of each category are fused to obtain the target prediction offset and the target offset confidence of each of the first video frames.

4. The video quality enhancement method according to claim 1, wherein: The categories are categories of motion states of objects, including stationary, low-speed motion, and high-speed motion.

5. The video quality enhancement method according to claim 1, wherein: The quality enhancement model further includes a spatial feature fusion module. After the step of performing object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module to obtain a second video sequence after quality enhancement, the method includes: Based on the spatial feature fusion module, feature extraction is performed on the second video frame in the second video sequence to obtain feature information of the second video frame; Performing feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features; Based on the enhanced low-frequency features and high-frequency features, the feature information is reconstructed to obtain enhanced feature information, and based on the enhanced feature information, a third video sequence with enhanced quality is determined.

6. The video quality enhancement method according to claim 5, wherein: The step of performing feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features includes: Performing global average pooling on the feature information to obtain a global feature vector; Performing feature adjustment on the global feature vector to obtain an adjusted global feature vector; Extracting coarse-grained features of the feature information, and fusing the coarse-grained features with the adjusted global feature vector to obtain enhanced low-frequency features; Subtracting the feature information from the low-frequency feature to obtain a high-frequency feature; Asymmetric convolution is performed on the high-frequency features to obtain enhanced high-frequency features.

7. The video quality enhancement method according to claim 5, wherein: The step of reconstructing the feature information based on the enhanced low-frequency features and high-frequency features to obtain enhanced feature information, and determining a third video sequence with enhanced quality based on the enhanced feature information includes: Adding the enhanced low-frequency features and high-frequency features to obtain enhanced feature information; Determining a prediction residual based on the enhanced feature information; The prediction residual is added to the second video frame to obtain a third video frame with enhanced quality, and a third video sequence with enhanced quality is composed based on the third video frame.

8. A video quality enhancement device, characterized in that: The video quality enhancement device comprises: An acquisition module, configured to acquire a first video sequence to be enhanced; An input module, configured to input the first video sequence into a preset quality enhancement model, wherein the quality enhancement model is a convolutional neural network model including an adjacent frame information fusion module; an enhancement module, configured to perform object offset prediction and adjacent frame information fusion processing on each first video frame to be enhanced in the first video sequence based on the adjacent frame information fusion module, to obtain a second video sequence with enhanced quality; an adjacent frame determining module, configured to determine, based on the adjacent frame information fusion module, a preset number of adjacent video frames for each first video frame to be enhanced in the first video sequence; a prediction module, performing object offset prediction on the first video frame and adjacent video frames of the first video frame, to obtain target prediction offsets and target offset confidences of the first video frame and adjacent video frames of the first video frame; A classification prediction module is used to use a preset multi-scale fully convolutional network to predict the offset of the first video frame and the adjacent video frames of the first video frame according to the classification category, and obtain an initial predicted offset and an initial offset confidence for each category.

9. The video quality enhancement device according to claim 8, wherein: The quality enhancement model further includes a spatial feature fusion module, and the enhancement module further includes: a convolution module, configured to perform a deformable convolution on the first video frame and adjacent video frames of the first video frame based on the target prediction offset and the target offset confidence, so as to achieve information fusion of the first video frame and the adjacent video frames of the first video frame, thereby obtaining a second video sequence with enhanced quality; And / or the prediction module includes: an offset fusion module, configured to fuse the initial prediction offset and the initial offset confidence of each category to obtain a target prediction offset and a target offset confidence of each first video frame; And / or the video quality enhancement device further includes: a feature extraction module, configured to perform feature extraction on the second video frame in the second video sequence based on the spatial feature fusion module to obtain feature information of the second video frame; a feature separation module, configured to perform feature separation and feature enhancement processing on the feature information to obtain enhanced low-frequency features and high-frequency features; a spatial feature fusion module, configured to reconstruct the feature information based on the enhanced low-frequency features and high-frequency features to obtain enhanced feature information, and determine a third video sequence with enhanced quality based on the enhanced feature information; And / or the feature separation module includes: a pooling module for performing global average pooling on the feature information to obtain a global feature vector; a feature adjustment module for performing feature adjustment on the global feature vector to obtain an adjusted global feature vector; a coarse-grained feature extraction module for extracting coarse-grained features of the feature information and fusing the coarse-grained features with the adjusted global feature vector to obtain enhanced low-frequency features; a high-frequency feature determination module for subtracting the feature information from the low-frequency features to obtain high-frequency features; and an asymmetric convolution module for performing asymmetric convolution on the high-frequency features to obtain enhanced high-frequency features. And / or the spatial feature fusion module includes: a reconstruction module, used to add the enhanced low-frequency features and high-frequency features to obtain enhanced feature information; a residual prediction module, used to determine the prediction residual based on the enhanced feature information; a quality enhancement module, used to add the prediction residual to the second video frame to obtain a third video frame with enhanced quality, and based on the third video frame, form a third video sequence with enhanced quality.

10. A video quality enhancement device, characterized in that: The video quality enhancement device includes: a memory, a processor, and a program stored in the memory for implementing the video quality enhancement method. The memory is used to store a program for implementing a video quality enhancement method; The processor is configured to execute a program for implementing the video quality enhancement method, so as to implement the steps of the video quality enhancement method according to any one of claims 1 to 7.

11. A storage medium, characterized in that: The storage medium stores a program for implementing the video quality enhancement method, and the program for implementing the video quality enhancement method is executed by a processor to implement the steps of the video quality enhancement method according to any one of claims 1 to 7.

Images