Deep learning-based compressed video quality enhancement method
By combining multi-scale attention networks and deformable convolutions, the problems of high computational overhead and poor feature fusion in existing compressed video quality enhancement methods are solved, achieving efficient compressed video quality enhancement and improving video clarity and integrity.
Patent Information
- Application Number
- CN202510631693.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-09-19
AI Technical Summary
Existing compressed video quality enhancement methods have high computational overhead and inaccurate flow estimation when utilizing cross-frame information. In addition, the limited receptive field hinders the network's ability to focus on key areas and feature fusion, resulting in a decrease in compressed video quality.
A multi-scale attention network and a multi-category parallel attention mechanism are used, combined with deformable convolution. The offset and modulation coefficient of the compressed video frame are extracted through the training model, and feature fusion and residual prediction are performed to generate high-quality video frames.
Effectively remove compression artifacts, improve video quality, enhance feature extraction capabilities, and enhance image clarity and integrity.
Smart Images
Figure CN120672875A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a method for enhancing the quality of compressed videos based on deep learning. Background Art
[0002] Video files occupy a large amount of storage space. To save storage space and improve video file transmission efficiency, lossy video compression algorithms (such as H.264 / AVC, H.265 / HEVC, AV1, and H.266 / VVC) can be used to reduce video storage and transmission costs within limited bandwidth. However, the quantization in lossy video coding inevitably introduces compression artifacts in the compressed video, which leads to a decrease in video quality and is detrimental to the user's viewing experience.
[0003] In addition, compression artifacts can also create difficulties for downstream video-oriented tasks such as object detection, action recognition, video summarization, and facial recognition. For example, when performing object detection, compression artifacts may change or blur key details in the image, making it difficult for object detection and recognition algorithms to accurately locate and classify the target object.
[0004] To improve user experience, image processing and deep learning techniques can be used to restore or enhance the original visual quality of compressed videos. Currently, single-frame-based image processing methods are specifically designed for JPEG quality enhancement. This approach inherently ignores the temporal correlations in video sequences, limiting the ability to utilize cross-frame information. Multi-frame-based image processing methods can use optical flow or deformable convolution to align adjacent frames with the target frame to utilize the inherent temporal correlations within consecutive low-quality frames to promote frame recovery. However, this approach also has some drawbacks: optical flow-based methods inevitably increase computational overhead, and inaccurate flow estimation and alignment may reduce the final quality; deformable convolution methods rely on a U-Net offset prediction network with simple standard convolutions. The limited receptive field hinders the network's ability to focus on key areas and fully extract spatiotemporal features, resulting in offset bias and poor feature fusion.
[0005] Therefore, there is currently a lack of a compressed video quality enhancement method that can overcome the limitations of existing methods. Summary of the Invention
[0006] The present invention provides a method for enhancing the quality of compressed video based on deep learning to address the deficiencies in the above-mentioned related technologies. The technical solution is as follows: In a first aspect, an embodiment of the present application provides a method for enhancing compressed video quality based on deep learning, the method comprising: Extracting each video frame in the compressed video frame by frame as a target video frame, and obtaining a compressed video frame sequence based on each target video frame; Inputting each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence; Adding the residual to a target video frame in the compressed video frame sequence to obtain a corresponding enhanced video frame; Combine the enhanced video frames corresponding to each of the compressed video frame sequences to output an enhanced video corresponding to the compressed video.
[0007] In an optional solution of the first aspect, the processing based on each target video frame to obtain a compressed video frame sequence includes: Selecting the first three frames and the last three frames of each target video frame according to the order of each video frame in the compressed video, merging the target video frame, the first three frames, and the last three frames to generate a compressed video frame sequence corresponding to the target video frame; If the number of video frames before or after the target video frame is less than 3, the video frame closest to the position of the missing video frame is copied until the number of video frames before and after the target video frame is equal to 3.
[0008] In an optional solution of the first aspect, the compressed video quality enhancement model includes an offset modulation prediction module, a feature fusion module, and a residual prediction module connected in sequence; Inputting each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence includes: Inputting the compressed video frame sequence into the offset modulation prediction module of the compressed video quality enhancement model, so as to perform feature extraction on the compressed video frame sequence through the offset modulation prediction module to obtain an offset and a modulation coefficient corresponding to the compressed video frame sequence; Inputting the offset and the modulation coefficient into the feature fusion module of the compressed video quality enhancement model, so as to obtain a fusion feature by performing weighted fusion based on the offset and the modulation coefficient through the feature fusion module; The fused features are input into the residual prediction module of the compressed video quality enhancement model, so that the residual prediction module extracts high-frequency features and low-frequency features based on the fused features, and fuses the high-frequency features and the low-frequency features to obtain the residual corresponding to the compressed video frame sequence.
[0009] In an optional solution of the first aspect, the offset modulation prediction module includes a multi-scale attention network and a standard convolutional network, wherein the multi-scale attention network includes a multi-scale convolution module and a multi-category parallel attention module, and the multi-scale convolution module includes multiple convolution kernels with the same dilation rate and different convolution kernel sizes; The extracting features of the compressed video frame sequence by the offset modulation prediction module to obtain the offset and modulation coefficient corresponding to the compressed video frame sequence includes: Extract features based on the compressed video frame sequence using multiple convolution kernels with the same expansion rate and different sizes in the multi-scale convolution module, and output local features, global features, and spatiotemporal features; Inputting the spatiotemporal features into a multi-category parallel attention module, and determining the weight of a channel corresponding to each video frame in the compressed video frame sequence based on an attention mechanism by the multi-category parallel attention module; The local features, global features, spatiotemporal features and the weights corresponding to each video frame are combined and input into a standard convolutional network to predict the offset and modulation coefficient of the compressed video frame sequence.
[0010] In an optional solution of the first aspect, performing weighted fusion based on the offset and the modulation coefficient by the feature fusion module to obtain a fused feature includes: Performing feature extraction on each video frame in the compressed video frame sequence using the deformable convolution module of the feature fusion module; Inputting the offset and the modulation coefficient into the deformable convolution module of the feature fusion module, and performing feature extraction on each video frame in the compressed video frame sequence through each convolution kernel in the deformable convolution module; Determine the initial sampling position of each convolution kernel on the video frame, determine the offset corresponding to the pixel at the initial sampling position, adjust the initial sampling position according to the offset corresponding to the pixel to obtain the actual sampling position on the video frame, and extract the sampling value at each actual sampling position; Determine the modulation coefficient and sampling value corresponding to each pixel, multiply the modulation coefficient and sampling value to generate an output feature map; Based on the convolution weight parameters of each convolution kernel in the deformable convolution module, a weighted average is performed on the output feature map corresponding to each video frame to output the fusion feature of the compressed video frame sequence.
[0011] In an optional solution of the first aspect, adjusting the initial sampling position by the offset corresponding to the pixel to obtain the actual sampling position on the video frame includes: Adding the offset corresponding to the pixel to the coordinates of the initial sampling position to obtain new sampling position coordinates; If the new sampling position coordinate is not an integer, adjusting the new sampling position coordinate to an integer by bilinear interpolation; The output is the sampling position coordinate adjusted to an integer to obtain the actual sampling position on the video frame.
[0012] In an optional solution of the first aspect, the compressed video quality enhancement model is trained by the following steps, including: Obtain a sample set; wherein each sample uses a compressed video frame sequence constructed based on a sample target frame as a sample input, and uses a high-quality target frame before compression of the sample target frame as a sample label; Inputting the video frame sample set into the compressed video quality enhancement model to be trained, obtaining the residual output by the compressed video quality enhancement model, and adding the residual to the sample target frame to obtain a predicted enhanced frame output by the model; A loss function is constructed based on the comparison results of each predicted enhanced frame and the corresponding high-quality target frame, and the convergence of the compressed video quality enhancement model is determined by the loss function; Determine the model parameters of the compressed video quality enhancement model after convergence, and output the trained compressed video quality enhancement model.
[0013] In a second aspect, an embodiment of the present application further provides a compressed video quality enhancement device based on deep learning, comprising: A data acquisition module is used to extract each video frame in the compressed video frame by frame as a target video frame, and obtain a compressed video frame sequence based on each target video frame; a calculation module, configured to input each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence; a video enhancement module, configured to add the residual to a target video frame in the compressed video frame sequence to obtain a corresponding enhanced video frame; The video enhancement module is further configured to combine the enhanced video frames corresponding to each of the compressed video frame sequences and output an enhanced video corresponding to the compressed video.
[0014] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided in the first aspect of the embodiment of the present application or any one of the implementations of the first aspect is implemented.
[0015] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the method provided by the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect.
[0016] The beneficial effects of the technical solutions provided by some embodiments of the present application include at least: The embodiments of this application provide a method for enhancing the quality of compressed videos based on deep learning. By introducing multi-scale large convolution kernels into a multi-scale attention network, the receptive field is expanded by performing dilated convolutions with the same dilation rate but different kernel sizes. This method effectively extracts local features and global contextual information, achieving more accurate feature extraction. Furthermore, by introducing multi-category parallel attention into the multi-scale attention network, the features of important channels can be highlighted and key spatial regions can be focused on, further enhancing feature extraction capabilities.
[0017] Furthermore, by introducing multi-path fusion attention to predict the residual, using dilated convolution and channel attention mechanism with different dilation rates and the same convolution kernel size, the network focuses on the boundary area of the moving object while maintaining the performance of other areas. This can effectively remove the compression artifacts at the boundary of the moving object while maintaining the artifact removal performance in other areas, which can significantly improve the quality of the compressed video and avoid the defects caused by the deterioration of image quality caused by compressed video. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 This is a flowchart of a method for enhancing compressed video quality based on deep learning provided by an embodiment of the present application; Figure 2 This is a schematic diagram of the structure of a compressed video quality enhancement device based on deep learning provided in an embodiment of the present application; Figure 3 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.
[0021] The terms "including" and "having," and any variations thereof, in the specification and claims of this application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to the process, method, product, or apparatus.
[0022] It should be noted that the terms "first" and "second" used in this application are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the terms "first" and "second" may interchangeably represent a specific order or precedence, where permitted. It should be understood that the objects distinguished by "first" and "second" may interchangeably represent a specific order or precedence, where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that described or illustrated herein.
[0023] It should be noted that, unlike traditional image quality enhancement methods, compressed video quality enhancement tasks typically require simultaneous consideration of temporal consistency, cross-frame redundant information, and the spatial distribution of compression distortion. The compressed video quality enhancement method provided in this application can fully utilize multi-frame information in a time series to perform compression artifact detection, feature alignment, and fusion, thereby generating clearer and more structurally complete video frames.
[0024] The present application is described in detail below with reference to specific embodiments.
[0025] Next, combine Figure 1 , introduces the compressed video quality enhancement method provided by the embodiment of this application. For details, please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart showing a method for enhancing the quality of compressed video provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps: S101 , extracting each video frame in a compressed video frame by frame as a target video frame, and obtaining a compressed video frame sequence based on each target video frame.
[0026] S102: Input each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence.
[0027] S103: Add the residual to the target video frame in the compressed video frame sequence to obtain a corresponding enhanced video frame.
[0028] S104: Combine the enhanced video frames corresponding to each of the compressed video frame sequences and output the enhanced video corresponding to the compressed video.
[0029] In some embodiments, any compression method can be selected to compress any format of video. For example, the HEVC Test Model 16.20 compression algorithm can be used to compress the YUV format video, and the quantization coefficients during compression are 22, 27, 32, 37 and 42 respectively, thereby obtaining a compressed video. The embodiments of the present application do not limit the compression method or video format.
[0030] Specifically, in S101, a compressed video frame sequence may be obtained based on the compressed video processing, specifically including: Selecting the first three frames and the last three frames of each target video frame according to the order of each video frame in the compressed video, merging the target video frame, the first three frames, and the last three frames to generate a compressed video frame sequence corresponding to the target video frame; Specifically, each extracted video frame is regarded as a single-channel vector, and the first three frames, last three frames of each target video frame and the target video frame itself are taken as input. The channels of the first three frames, last three frames and the target video frame itself are merged to generate a 7-channel vector, that is, a compressed video frame sequence is obtained.
[0031] If the number of video frames before or after the target video frame is less than 3, the video frame closest to the position of the missing video frame is copied until the number of video frames before and after the target video frame is equal to 3.
[0032] For example, for the first frame in the compressed video, taking the first frame as the target video frame, it is easy to extract three consecutive video frames after the first frame, but there is no video frame before the first frame. At this time, the nearest video frame to the position where the video frame is missing is the first frame, and the first three frames are copied based on the first frame, so that the number of video frames before and after the first frame is equal to 3.
[0033] In some embodiments, the compressed video quality enhancement model in S102 includes an offset modulation prediction module, a feature fusion module, and a residual prediction module connected in sequence; Specifically, the step S102 includes: S1021, inputting the compressed video frame sequence into the offset modulation prediction module of the compressed video quality enhancement model, so as to perform feature extraction on the compressed video frame sequence through the offset modulation prediction module to obtain an offset and a modulation coefficient corresponding to the compressed video frame sequence; S1022: Input the offset and the modulation coefficient into the feature fusion module of the compressed video quality enhancement model, so that the feature fusion module performs weighted fusion based on the offset and the modulation coefficient to obtain a fusion feature; S1023, inputting the fused features into the residual prediction module of the compressed video quality enhancement model, so that the residual prediction module extracts high-frequency features and low-frequency features based on the fused features, and fuses the high-frequency features and the low-frequency features to obtain the residual corresponding to the compressed video frame sequence.
[0034] In some embodiments, in S1021, the offset modulation prediction module includes a multi-scale attention network and a standard convolutional network, wherein the multi-scale attention network includes a multi-scale convolution module and a multi-category parallel attention module, and the multi-scale convolution module includes multiple convolution kernels with the same dilation rate and different convolution kernel sizes; The process of obtaining the offset and modulation coefficient at step S1021 specifically includes: Extract features based on the compressed video frame sequence using multiple convolution kernels with the same expansion rate and different sizes in the multi-scale convolution module, and output local features, global features, and spatiotemporal features; Inputting the spatiotemporal features into a multi-category parallel attention module, and determining the weight of a channel corresponding to each video frame in the compressed video frame sequence based on an attention mechanism by the multi-category parallel attention module; The local features, global features, spatiotemporal features and the weights corresponding to each video frame are combined and input into a standard convolutional network to predict the offset and modulation coefficient of the compressed video frame sequence.
[0035] Specifically, the multi-scale attention network combines multi-scale large convolution kernels with multi-category parallel attention. The multi-scale large convolution kernels can include three dilated convolution kernels with the same dilation rate and different convolution kernel sizes. The smaller convolution kernels can facilitate the acquisition of local features of each input video frame, while the largest convolution kernel can capture a wider range of contextual information, thereby obtaining global features for each input video frame. This allows the simultaneous extraction of local details and global contextual information, effectively expanding the receptive field. Expanding the receptive field can be understood as expanding the pixel range of the convolution kernel to extract features to enhance global perception. Furthermore, based on the global and local features, spatial feature maps at different scales can be determined to obtain spatial features. Based on the global and local features of consecutive frames, the dynamic changes in the features of the video frames over time can be obtained, and temporal features can be extracted, thereby obtaining spatiotemporal features.
[0036] The multi-category parallel attention module can determine the more critical channels through channel attention, that is, determine the weight of the channel corresponding to each video frame in the compressed video frame sequence, so that it can focus on the more critical areas in the spatiotemporal features and further enhance the accuracy of the spatiotemporal features.
[0037] Specifically, the offset and modulation coefficient of the compressed video frame sequence can be obtained by respectively processing through two standard convolutional networks.
[0038] It should be noted that the extracted offset of the compressed video frame sequence can be understood as an offset matrix consisting of the horizontal and vertical offsets corresponding to each pixel in each video frame; and the extracted modulation coefficient of the compressed video frame sequence can be understood as an offset matrix consisting of the modulation coefficients corresponding to each pixel in each video frame. In other words, the horizontal offset, vertical offset, and modulation coefficient corresponding to each pixel in each video frame are extracted.
[0039] It should be noted that the standard convolutional network can be understood as a common convolution layer with a 3x3 convolution kernel.
[0040] In some embodiments, S1022 includes: Performing feature extraction on each video frame in the compressed video frame sequence using the deformable convolution module of the feature fusion module; Inputting the offset and the modulation coefficient into the deformable convolution module of the feature fusion module, and performing feature extraction on each video frame in the compressed video frame sequence through each convolution kernel in the deformable convolution module; Determine the initial sampling position of each convolution kernel on the video frame, determine the offset corresponding to the pixel at the initial sampling position, adjust the initial sampling position according to the offset corresponding to the pixel to obtain the actual sampling position on the video frame, and extract the sampling value at each actual sampling position; Determine the modulation coefficient and sampling value corresponding to each pixel, multiply the modulation coefficient and sampling value to generate an output feature map; Based on the convolution weight parameters of each convolution kernel in the deformable convolution module, a weighted average is performed on the output feature map corresponding to each video frame to output the fusion feature of the compressed video frame sequence.
[0041] It can be understood that the convolution weight parameters of each convolution kernel are parameters determined during the training process of the compressed video quality enhancement model.
[0042] In this way, the originally fixed sampling positions are adjusted according to the input offset, allowing the convolution kernel to adaptively sample irregular positions in the image, thereby enhancing the ability to model geometric deformations. The modulation coefficient acts as an additional weighting factor to weight the response of each sampling position, thereby controlling the contribution of each sampling position to the output features. This modulation mechanism further enhances the network's expressive power, enabling it to more flexibly select information in space.
[0043] In some embodiments, adjusting the initial sampling position by the offset corresponding to the pixel to obtain the actual sampling position on the video frame includes: Adding the offset corresponding to the pixel to the coordinates of the initial sampling position to obtain new sampling position coordinates; If the new sampling position coordinate is not an integer, adjusting the new sampling position coordinate to an integer by bilinear interpolation; The output is the sampling position coordinate adjusted to an integer to obtain the actual sampling position on the video frame.
[0044] In some embodiments, S1023 includes: The residual prediction module includes multiple multi-path fusion attention networks and two standard convolutional networks. The multi-path fusion attention network first uses three dilated convolutions with 3x3 kernels and different dilation rates to expand the receptive field and enhance global perception, thereby obtaining low-frequency features more comprehensively based on fusion feature extraction.
[0045] The multi-path fusion attention network can further obtain more valuable high-frequency features through the channel attention mechanism.
[0046] Furthermore, by integrating low-frequency and high-frequency features, attention can be focused on the boundaries of moving objects, which greatly improves the accuracy of compression artifact removal.
[0047] In order to optimize the fusion of low-frequency and high-frequency features, a parameter can be added to the multi-path fusion attention network , the parameters can be set through grid search Set it to 0.2 and use the following formula to calculate the residual: High frequency × β + low frequency = residual; It can be understood that the residual is specifically a residual matrix composed of residual values corresponding to each pixel point on the target video frame.
[0048] Specifically, in S103 , the residual may be added to the target video frame in the compressed video frame sequence, specifically by adding the residual value in the residual matrix to the eigenvalue of the corresponding pixel, thereby generating an enhanced video frame of the target video frame.
[0049] Finally, in S104 , the enhanced video frames of each target video frame are spliced together according to the order of the target video frames in the original compressed video to obtain the enhanced video.
[0050] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.
[0051] See next Figure 2 , which is a schematic diagram of the structure of a deep learning-based compressed video quality enhancement device provided in an exemplary embodiment of the present application. The device can be implemented as all or part of a terminal through software, hardware, or a combination of both, or can be integrated into a server as an independent module. The deep learning-based compressed video quality enhancement device in the embodiment of the present application can be applied to a terminal or the cloud. The device 20 includes a data acquisition module 201, a calculation module 202, and a video enhancement module 203, wherein: The data acquisition module 201 is used to extract each video frame in the compressed video frame by frame as a target video frame, and obtain a compressed video frame sequence based on each target video frame; The calculation module 202 is used to input each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence; The video enhancement module 203 is configured to add the residual to the target video frame in the compressed video frame sequence to obtain a corresponding enhanced video frame; The video enhancement module 203 is further configured to combine the enhanced video frames corresponding to each of the compressed video frame sequences and output an enhanced video corresponding to the compressed video.
[0052] It should be noted that the device 20 provided in the above embodiment, when executing the compressed video quality enhancement method based on deep learning, only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the embodiment of the compressed video quality enhancement method based on deep learning are based on the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.
[0053] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method of any of the above embodiments are implemented.
[0054] See Figure 3 , is a structural block diagram of an electronic device provided in an embodiment of the present application.
[0055] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302 .
[0056] In the embodiment of the present application, the processor 301 is the control center of the computer system and can be the processor of a physical machine or the processor of a virtual machine. The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 can be implemented in the form of at least one hardware of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array).
[0057] The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.
[0058] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the method in the embodiment of the present application.
[0059] In some embodiments, the electronic device 300 further includes a peripheral device interface 303 and at least one peripheral device 304. The processor 301, memory 302, and peripheral device interface 303 may be connected via a bus or signal lines. Each peripheral device 304 may be connected to the peripheral device interface 303 via a bus, signal lines, or circuit boards. Specifically, the peripheral devices 304 include a display screen, a camera, and an audio circuit. The peripheral device interface 303 may be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 301 and memory 302.
[0060] In some embodiments of the present application, the processor 301, the memory 302, and the peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 301, the memory 302, and the peripheral device interface 303 may be implemented on separate chips or circuit boards. This embodiment of the present application is not specifically limited to this.
[0061] The electronic device structure block diagram shown in the embodiment of the present application does not constitute a limitation on the electronic device 300. The electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.
[0062] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the aforementioned embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic or optical card, a nanosystem (including a molecular memory IC), or any other type of medium or device suitable for storing instructions and / or data.
[0063] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.
[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for enhancing compressed video quality based on deep learning, characterized in that: include: Extracting each video frame in the compressed video frame by frame as a target video frame, and obtaining a compressed video frame sequence based on each target video frame; Inputting each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence; Adding the residual to a target video frame in the compressed video frame sequence to obtain a corresponding enhanced video frame; Combine the enhanced video frames corresponding to each of the compressed video frame sequences and output an enhanced video corresponding to the compressed video.
2. The method for enhancing compressed video quality based on deep learning according to claim 1, wherein: The step of obtaining a compressed video frame sequence based on each target video frame includes: Selecting the first three frames and the last three frames of each target video frame according to the order of each video frame in the compressed video, merging the target video frame, the first three frames, and the last three frames to generate a compressed video frame sequence corresponding to the target video frame; If the number of video frames before or after the target video frame is less than 3, the video frame closest to the position of the missing video frame is copied until the number of video frames before and after the target video frame is equal to 3.
3. The method for enhancing compressed video quality based on deep learning according to claim 1, wherein: The compressed video quality enhancement model includes an offset modulation prediction module, a feature fusion module and a residual prediction module connected in sequence; Inputting each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence includes: Inputting the compressed video frame sequence into the offset modulation prediction module of the compressed video quality enhancement model, so as to perform feature extraction on the compressed video frame sequence through the offset modulation prediction module to obtain an offset and a modulation coefficient corresponding to the compressed video frame sequence; Inputting the offset and the modulation coefficient into the feature fusion module of the compressed video quality enhancement model, so as to obtain a fusion feature by performing weighted fusion based on the offset and the modulation coefficient through the feature fusion module; The fused features are input into the residual prediction module of the compressed video quality enhancement model, so that the residual prediction module extracts high-frequency features and low-frequency features based on the fused features, and fuses the high-frequency features and the low-frequency features to obtain the residual corresponding to the compressed video frame sequence.
4. The method for enhancing compressed video quality based on deep learning according to claim 3, wherein: The offset modulation prediction module includes a multi-scale attention network and a standard convolutional network, wherein the multi-scale attention network includes a multi-scale convolution module and a multi-category parallel attention module, and the multi-scale convolution module includes multiple convolution kernels with the same expansion rate and different convolution kernel sizes; The extracting features of the compressed video frame sequence by the offset modulation prediction module to obtain the offset and modulation coefficient corresponding to the compressed video frame sequence includes: Extract features based on the compressed video frame sequence using multiple convolution kernels with the same expansion rate and different sizes in the multi-scale convolution module, and output local features, global features, and spatiotemporal features; Inputting the spatiotemporal features into a multi-category parallel attention module, and determining the weight of a channel corresponding to each video frame in the compressed video frame sequence based on an attention mechanism by the multi-category parallel attention module; The local features, global features, spatiotemporal features and the weights corresponding to each video frame are combined and input into a standard convolutional network to predict the offset and modulation coefficient of the compressed video frame sequence.
5. The method for enhancing compressed video quality based on deep learning according to claim 3 or 4, characterized in that: The step of performing weighted fusion based on the offset and the modulation coefficient by the feature fusion module to obtain a fusion feature includes: Performing feature extraction on each video frame in the compressed video frame sequence using the deformable convolution module of the feature fusion module; Inputting the offset and the modulation coefficient into the deformable convolution module of the feature fusion module, and performing feature extraction on each video frame in the compressed video frame sequence through each convolution kernel in the deformable convolution module; Determine the initial sampling position of each convolution kernel on the video frame, determine the offset corresponding to the pixel at the initial sampling position, adjust the initial sampling position according to the offset corresponding to the pixel to obtain the actual sampling position on the video frame, and extract the sampling value at each actual sampling position; Determine the modulation coefficient and sampling value corresponding to each pixel, multiply the modulation coefficient and sampling value to generate an output feature map; Based on the convolution weight parameters of each convolution kernel in the deformable convolution module, a weighted average is performed on the output feature map corresponding to each video frame to output the fusion feature of the compressed video frame sequence.
6. The method for enhancing compressed video quality based on deep learning according to claim 5, characterized in that: The adjusting the initial sampling position by the offset corresponding to the pixel to obtain the actual sampling position on the video frame includes: Adding the offset corresponding to the pixel to the coordinates of the initial sampling position to obtain new sampling position coordinates; If the new sampling position coordinate is not an integer, adjusting the new sampling position coordinate to an integer by bilinear interpolation; The output is the sampling position coordinate adjusted to an integer to obtain the actual sampling position on the video frame.
7. The method for enhancing compressed video quality based on deep learning according to claim 2, wherein: The compressed video quality enhancement model is trained by the following steps, including: Obtain a sample set; wherein each sample uses a compressed video frame sequence constructed based on a sample target frame as a sample input, and uses a high-quality target frame before compression of the sample target frame as a sample label; Inputting the video frame sample set into the compressed video quality enhancement model to be trained, obtaining the residual output by the compressed video quality enhancement model, and adding the residual to the sample target frame to obtain a predicted enhanced frame output by the model; A loss function is constructed based on the comparison results of each predicted enhanced frame and the corresponding high-quality target frame, and the convergence of the compressed video quality enhancement model is determined by the loss function; Determine the model parameters of the compressed video quality enhancement model after convergence, and output the trained compressed video quality enhancement model.
8. A device for enhancing compressed video quality based on deep learning, characterized in that: include: A data acquisition module is used to extract each video frame in the compressed video frame by frame as a target video frame, and obtain a compressed video frame sequence based on each target video frame; a calculation module, configured to input each of the compressed video frame sequences into a trained compressed video quality enhancement model to obtain a residual corresponding to the compressed video frame sequence; a video enhancement module, configured to add the residual to a target video frame in the compressed video frame sequence to obtain a corresponding enhanced video frame; The video enhancement module is further configured to combine the enhanced video frames corresponding to each of the compressed video frame sequences and output an enhanced video corresponding to the compressed video.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.