Video super-resolution method and system, electronic equipment and storage medium

Abstract

The invention discloses a video super-resolution method and system, electronic equipment and a storage medium, and the method comprises the steps: building a video super-resolution model in a complex motion scene, and training and testing the model, the model comprises a feature extraction sub-module, a cross-frame alignment sub-module, a feature enhancement sub-module and an up-sampling reconstruction sub-module, the feature extraction sub-module is used for extracting shallow features of input continuous video frames; the cross-frame alignment sub-module performs feature level alignment on different frames by using deformable convolution to realize cross-frame spatio-temporal information fusion and motion compensation; the feature enhancement sub-module is used for enhancing the features; and the up-sampling reconstruction sub-module amplifies the resolution of the enhanced features output by the feature enhancement sub-module, and reconstructs the enhanced features after resolution amplification into video frames. According to the invention, the super-resolution effect of the video in the complex motion scene is improved, and the quality and visual effect of the video are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of image technology, and in particular relates to a video super-resolution method, system, electronic equipment and storage medium. Background Art

[0002] With the rise of audio-visual entertainment software, such as the popularity of short video platforms, people often want to obtain and watch high-resolution videos. High-resolution video means that the pixel density of the video frame is high, which can provide more details, and these details are indispensable in many practical applications. For example, high-resolution medical images can help doctors make more accurate diagnoses; high-resolution surveillance videos can provide more effective security; using high-resolution satellite images, it is easy to distinguish similar objects. However, the price of high-resolution digital cameras, camcorders or smart devices such as mobile phones is usually expensive. This is because the acquisition of high-resolution videos requires higher-precision optical and image sensors. These high-precision optical and image sensors are expensive and have certain technical barriers. Due to this, there are a large number of low-resolution and low-quality videos in our real world, but people's demand for high-definition videos continues to expand.

[0003] Video super-resolution technology is a new technology that increases the spatial resolution of videos, thereby overcoming the limitations of sensors and optical manufacturing technology. Video super-resolution (abbreviated as video super-resolution) is all about restoring low-resolution, noisy, low-quality videos into high-resolution, high-quality videos. Video super-resolution technology is closely related to us and is playing a vital role in various industries such as video surveillance, virtual reality image quality enhancement, and medical image enhancement. In addition, it can also restore old low-resolution film and television works with AI (enlarge them into high-definition and ultra-clear videos), which is a strong proof of the application of video super-resolution technology in commercial applications.

[0004] In traditional video super-resolution methods, most of them use optical flow to estimate the motion of adjacent frames and reference frames to achieve motion compensation and alignment; or use deformable convolution-based methods to align adjacent frames with reference frames. However, the alignment accuracy of the optical flow method is not high, and the deformable convolution-based method generally uses one or several ordinary convolution layers to uniformly learn the motion offsets between different adjacent frames and reference frames. The learning ability of the offset is insufficient and the receptive field is small. Therefore, the robustness is low when facing complex scenes, especially large motion scenes. At the same time, our low-resolution videos usually contain complex and diverse motion scenes. Most of the existing super-resolution technologies cannot reconstruct high-quality video data and cannot meet the visual needs of users. Summary of the invention

[0005] In view of the above problems, the present invention provides a video super-resolution method, system, electronic device and storage medium, aiming to improve the super-resolution effect of videos in complex motion scenes and enhance the quality and visual effect of the videos.

[0006] According to a first aspect of an embodiment of the present disclosure, a video super-resolution method is provided, the method comprising the following steps: Obtaining a video data set containing a variety of different scenes and preprocessing it, and dividing the preprocessed video data set into a training data set and a test data set; A video super-resolution model is established, and the video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule and an upsampling reconstruction submodule, wherein the feature extraction submodule is used to extract shallow features of input continuous video frames; the cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames for the shallow features output by the feature extraction submodule, so as to realize cross-frame spatiotemporal information fusion and motion compensation; the feature enhancement submodule enhances the features after spatiotemporal information fusion output by the cross-frame alignment submodule; the upsampling reconstruction submodule performs resolution amplification on the enhanced features output by the feature enhancement submodule, and reconstructs the enhanced features after resolution amplification into video frames; Using the training data set and the test data set to train and test the video super-resolution model respectively; Video super-resolution is achieved using the trained and tested video super-resolution model.

[0007] In some embodiments, the cross-frame alignment submodule implements motion compensation according to different frame rate groups, including: based on different groups containing adjacent frames of different time series, aligning the features of adjacent frames and reference frames in a decoupled manner, and fusing the aligned adjacent frame features with the reference frame features through a convolutional layer, and inputting the fused features into the feature enhancement submodule.

[0008] In some embodiments, the cross-frame alignment submodule is specifically implemented by the following steps: Determine a reference frame and adjacent frames based on a feature sequence of an input video frame; The feature sequences are grouped according to different frame rates. This is because the distance of adjacent frames from the reference frame in the time series (i.e., the frame rate) is proportional to the motion offset of the adjacent frames relative to the reference frame, and inversely proportional to the similarity between the adjacent frames and the reference frame. Therefore, more efficient grouping and alignment of adjacent frames with different frame rates can be achieved through decoupling. In each group, a set of cascaded dilated convolutions and a layer of normal convolution are used to learn the offsets of adjacent frames and reference frames at the feature layer; Input the features of each group offset and adjacent frames into a deformable convolutional neural network to obtain aligned features; The Concatenation operation is used to concatenate the reference frame features and the aligned features on the channel, and then a simple convolution layer is used to effectively aggregate the concatenated features to obtain the output features of the cross-frame alignment submodule.

[0009] In some embodiments, the video super-resolution model is trained and tested using the training data set and the test data set, respectively, specifically including: Inputting the training data sets under a plurality of different scenarios into the video super-resolution model in sequence; During iterative training, calculating the loss value between the video frame generated by the video super-resolution model and the true value; The video super-resolution model is trained with the reduction of the loss value as the optimization target, and the network weights are updated and optimized simultaneously until convergence; The test dataset is used to evaluate the reconstruction performance of the video super-resolution model.

[0010] In some embodiments, the loss value is calculated based on a Charbonnier loss function.

[0011] In some embodiments, the feature enhancement submodule includes no less than 3 feature enhancement blocks, each of which is composed of a convolution kernel with a size of 3 3 regular convolutional layers and a convolution kernel size of 3 The structure is composed of 3 group convolutional layers.

[0012] In some embodiments, the cascaded dilated convolution includes three dilated convolutions with a kernel size of 3.

[0013] According to a second aspect of an embodiment of the present disclosure, a video super-resolution system is provided, the system comprising: A data set acquisition module is used to acquire a video data set containing a variety of different scenes and perform preprocessing, and divide the preprocessed video data set into a training data set and a test data set; A model building module is used to build a video super-resolution model, and the video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule and an upsampling reconstruction submodule, wherein the feature extraction submodule is used to extract shallow features of input continuous video frames; the cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames for the shallow features output by the feature extraction submodule, so as to realize cross-frame spatiotemporal information fusion and motion compensation; the feature enhancement submodule enhances the features after spatiotemporal information fusion output by the cross-frame alignment submodule; the upsampling reconstruction submodule performs resolution amplification on the enhanced features output by the feature enhancement submodule, and reconstructs the enhanced features after resolution amplification into video frames; A model training and testing module, used to train and test the video super-resolution model using the training data set and the test data set respectively; The video super-resolution implementation module is used to implement video super-resolution using the video super-resolution model that has been trained and tested.

[0014] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned video super-resolution method when executing the program.

[0015] According to a fourth aspect of an embodiment of the present disclosure, there is provided a non-temporary computer-readable storage medium having computer instructions stored thereon, which implement the steps of the above-mentioned video super-resolution method when executed by a processor.

[0016] The embodiments of the present disclosure provide a video super-resolution method, system, electronic device and storage medium. The method includes establishing a video super-resolution model, wherein a feature extraction submodule fully extracts the features of input continuous video frames through one or several cascaded convolution layers, laying a foundation for further realizing efficient motion compensation at the feature level; a cross-frame alignment submodule performs fine-grained alignment and fusion of adjacent frames of different frame rates through a group alignment mechanism, decouples adjacent frames from the time series relationship, and adopts different alignment submodules for different groups, so that the alignment has greater flexibility. For example, the offset learned by the high frame rate group with large motion is often Based on this, the present invention can realize efficient inter-frame alignment and motion compensation based on the spatiotemporal relationship between consecutive frames, and output the aligned deep features. In addition, the cross-frame submodule based on the hole convolution group can quickly learn the high-precision offset; the alignment submodule method avoids the inaccuracy of the single optical flow method alignment, solves the limitations of the traditional deformable convolution method (insufficient alignment ability under large motion and complex scenes), and avoids the computational cost of 3D convolution, that is, it realizes efficient and accurate cross-frame motion compensation with a lower computational cost; the feature enhancement submodule consists of 6 efficient feature enhancement blocks, each of which consists of a convolution kernel size of 3 3 regular convolutional layers and a convolution kernel size of 3 3, the group convolution layer adopts the idea of ​​group convolution, which has lower computational complexity than ordinary convolution. Conventional convolution is used to solve the problem that channels of different groups are difficult to interact with each other caused by group convolution. The efficient feature enhancement block in the present invention integrates the advantages of conventional convolution and group convolution, while effectively reducing the computational complexity of the model, which helps to achieve an effective balance between model performance and computational complexity.

[0017] In summary, the video super-resolution method, system, electronic device and storage medium in complex motion scenes proposed in the present invention, through a group alignment mechanism, intelligently divides the input continuous video frames in time sequence, and improves the compatibility with complex motion scenes (especially large motion scenes); secondly, by quickly processing the features of different video frames, efficient spatiotemporal information fusion is achieved, and the information extraction capability is enhanced; furthermore, through a novel feature enhancement mechanism, the details and texture of the video frames can be fully maintained, which is beneficial to the final reconstruction result and makes the visual effect better. Thanks to this, the video super-resolution method, system, electronic device and storage medium described in the present invention can achieve higher super-resolution performance with a lower parameter amount, and can be well promoted and applied in actual scenes in the industry.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 is a flow chart of a video super-resolution method according to an embodiment of the present invention; Figure 2 is a block diagram of a video super-resolution model in a complex motion scene according to an embodiment of the present invention; Figure 3 is a flowchart of a cross-frame alignment submodule in an embodiment of the present invention; Figure 4 is a schematic diagram of the structure of a video super-resolution system in an embodiment of the present invention; Figure 5 It is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0022] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0023] Video super-resolution is different from image super-resolution. There is a significant correlation between consecutive frames of a video in time and space. The shooting environment and content of the same clip or scene are similar, but there will be object movement and offset between different frames. Therefore, when performing video super-resolution, it is necessary to focus on how to achieve alignment between frames to better extract and fuse the spatiotemporal information of different frames.

[0024] Based on these constraints, efficient frame alignment of continuous video frames of different input scenes is a key problem that needs to be solved urgently in video super-resolution methods.

[0025] Inter-frame alignment is also called motion compensation. Continuous video frames can be divided into adjacent frames and reference frames to be processed. Inter-frame alignment means calculating the offset of pixel motion between adjacent frames and reference frames, and then aligning adjacent frames to the reference frame based on the offset. Currently, the main methods for inter-frame alignment are optical flow and deformable convolution.

[0026] The purpose of the two major methods, motion compensation and deformable convolution, is to extract inter-frame motion information. Among them, the optical flow method is used to calculate the correlation and motion changes between adjacent frames and reference frames in the spatiotemporal domain, so as to implement a distortion operation on adjacent frames to align them with the reference frame. Deformable convolution adds an offset to the standard convolution kernel to obtain a shape-variable convolution kernel, which is then convolved with the input features. Therefore, the input video frame can be transformed through deformable convolution to achieve inter-frame information fusion. The optical flow method is concise and clear, but the alignment accuracy is not high. The deformable convolution method generally uses one or more convolutional neural network layers to learn the offset of motion, providing a variety of offsets, strong robustness, and relatively good alignment effect. However, the learning cost of the offset is large, and the receptive field of several stacked convolutions is limited. For large motion scenes, its alignment accuracy needs to be further improved, especially the learned offset is usually small, resulting in weak motion compensation capabilities in large motion scenes.

[0027] In addition, 3D convolution and recurrent convolutional neural network (RCNN) can also be used to implicitly fuse the spatiotemporal information of adjacent frames and reference frames. That is, without deliberately performing inter-frame alignment of the video, the excellent spatiotemporal modeling performance of 3D convolution or recurrent neural network is used to implicitly extract and fuse motion information between frames of different time sequences while extracting or enhancing features, thereby removing the motion compensation module and reducing the computational cost, but the robustness is insufficient for complex scenes.

[0028] Based on this, the embodiments of the present invention provide the following embodiments for a video super-resolution method, system, electronic device, and storage medium: One of the embodiments is used to illustrate the video super-resolution method. Figure 1 As shown, the method comprises the following steps: S1, obtaining a video data set containing a variety of different scenes and preprocessing it, and dividing the preprocessed video data set into a training data set and a test data set; During the specific implementation process, a video dataset containing at least 50 complex motion scenes is obtained, and after preprocessing, the dataset is further divided into a training dataset and a test dataset; the dataset consists of continuous video frames containing adjacent frames and reference frames.

[0029] Specifically, video data sets for training and testing can be collected through a variety of ways, such as searching for keywords of scenes through search engines, downloading various free video data sets; obtaining video data stored in public database servers; and using mobile phones, cameras and other devices to record video data. After data preprocessing (such as removing abnormal video frames, unifying the resolution size of different frames, etc.), the data set is randomly divided into a training data set and a test data set. Among them, the training data set accounts for 85% and is used to train the video super-resolution model, and the test set accounts for 15% and is used to test the super-resolution performance of the generated model. By inputting a low-resolution, low-quality video, a high-resolution, high-quality video generated by an embodiment of the present invention is obtained.

[0030] S2. Establish a video super-resolution model, which includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule and an upsampling reconstruction submodule, wherein the feature extraction submodule is used to extract shallow features of input continuous video frames; the cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames for the shallow features output by the feature extraction submodule, so as to achieve cross-frame spatiotemporal information fusion and motion compensation; the feature enhancement submodule enhances the features after spatiotemporal information fusion output by the cross-frame alignment submodule; the upsampling reconstruction submodule performs resolution amplification on the enhanced features output by the feature enhancement submodule, and reconstructs the enhanced features after resolution amplification into video frames; S3, using the training data set and the test data set to train and test the video super-resolution model respectively; S4. Implementing video super-resolution using the trained and tested video super-resolution model.

[0031] In the specific implementation process, Figure 2 As shown in Figure 1, the video super-resolution model in complex motion scenes consists of a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, an upsampling reconstruction submodule, etc. The feature extraction submodule is responsible for extracting shallow features from continuous video frames (including adjacent frames and reference frames) in complex input scenes, laying the foundation for subsequent cross-frame alignment and fusion. Specifically, the feature extraction submodule fully extracts the features of continuous input video frames through one or several cascaded convolutional layers, laying the foundation for further efficient motion compensation at the feature level.

[0032] The cross-frame alignment submodule implements motion compensation according to different frame rate groups, including: aligning the features of adjacent frames with the reference frame in a decoupled manner based on different groups containing adjacent frames of different time series, fusing the aligned adjacent frames with the reference frame through a convolutional layer, and inputting the fused features into the feature enhancement submodule.

[0033] In the specific implementation process, Figure 3 As shown in the figure, the cross-frame alignment submodule uses deformable convolution to quickly align different frames at the feature level, thereby realizing cross-frame spatiotemporal information fusion and motion compensation. The cross-frame alignment submodule can fully and adaptively learn the spatiotemporal relationship between consecutive frames, complete motion compensation and align features, realize efficient fusion of cross-frame information, and improve compatibility with complex motion scenes, especially large motion scenes. The specific implementation of the cross-frame alignment submodule includes the following steps: determining the reference frame and adjacent frames based on the feature sequence of the input video frame; grouping the feature sequence according to different frame rates, taking into account the frame rate size, that is, the distance of the adjacent frames from the reference frame in the time series, which is proportional to the motion offset of the adjacent frames relative to the reference frame and inversely proportional to the similarity between the adjacent frames and the reference frame, assigning adjacent frames of the same frame rate to the same group and then aligning them, so as to achieve efficient motion compensation in a decoupled manner; adopting a set of cascaded hole convolutions and a layer of ordinary convolution in each group, and learning the offsets of adjacent frames and reference frames at the feature layer; inputting the offsets of each group and the features of the adjacent frames into the deformable convolutional neural network to obtain the aligned features; using the Concatenation operation to splice the reference frame features and the aligned features on the channel, and then using a convolution layer to fuse the spliced ​​features to obtain the output features of the cross-frame alignment submodule. Specifically, the cross-frame alignment submodule groups the features of continuous input frames according to different frame rates, and uses a set of cascaded dilated convolutions and a layer of ordinary convolution to learn the motion offsets between adjacent frames and reference frames in groups, thereby increasing the receptive field of convolution without increasing the amount of computation and improving compatibility with complex motion scenes (especially large motion scenes); based on the learned offsets of each group, deformable convolution is used to align adjacent frames with the reference frames, and cross-frame alignment and motion compensation are efficiently and adaptively achieved in a decoupled manner; finally, a convolution layer is used to effectively aggregate the aligned features of each group, deeply exploring the spatiotemporal correlation between adjacent frames and reference frames, and fully extracting and integrating the information in adjacent frames that is beneficial to the super-resolution reconstruction of the reference frame.

[0034] In one embodiment, the cross-frame alignment submodule can fully explore the spatiotemporal relationship between adjacent frames and reference frames, thereby achieving efficient motion compensation and obtaining better video super-resolution effects. Specifically, assuming that the feature sequence of the input video frame is {tn, ..., t, ..., t+n}, there are 2n+1 video frame features. The features of the tth frame are the features of the reference frame, and the features of the remaining 2n frames are the features of the adjacent frames. The frame alignment submodule first adopts a grouping strategy, that is, it is divided into n groups according to the temporal relationship (frame rate). For example, the features of the tnth, tth, and t+nth frames are grouped together, the features of the t-n+1th, tth, and t+n-1th frames are grouped together, and finally the features of the t-1th, tth, and t+1th frames are grouped together. The frame rates of different groups are different. For example, the frame rate in the feature group of the tnth, tth, and t+nth frames is n, that is, the distance between the features of the tnth and t+nth frames and the reference frame in the time series is n. The larger the frame rate, the farther the adjacent frames are from the reference frame in the time series, and the larger the motion offset of the adjacent frames relative to the reference frame, that is, the lower the similarity with the reference frame. In each group, a set of cascaded dilated convolutions and a layer of ordinary convolution are taken to learn the offsets of adjacent frames and reference frames at the feature level. The dilated convolution group contains 3 dilated convolutions with a convolution kernel size of 3, and the dilated rates are 1, 2 and 3 respectively. The larger the dilated rate, the larger the receptive field. Compared with ordinary convolution, dilated convolution can expand the receptive field, enhance the representation ability when learning the offset, and avoid the possible phenomenon of small offset due to small receptive field. At the same time, dilated convolution does not increase the amount of calculation, which can keep the model efficient. Based on this, the cross-frame submodule based on the dilated convolution group can quickly learn high-precision offsets. Then, in each group, the features of the adjacent frames and the reference frame are aligned through the learned offsets and the deformable convolution network. Note that the above layer of ordinary convolution is included in the deformable convolution layer to map the intermediate features learned by the dilated convolution group for generating offsets into the final pixel offsets. After all groups are aligned, all group features are connected through the Concatenation operation and then fused with the reference frame through a simple convolution layer.

[0035] In some preferred embodiments, a downsampling method may be used to scale the video frames before alignment, thereby reducing the amount of motion in disguise, improving alignment accuracy, and improving the effect of video super-resolution for scenes with large motion.

[0036] Different from the standard deformable convolution method, which only uses one or several layers of ordinary convolution to learn the offsets of all adjacent frames at one time and then align them, the cross-frame alignment submodule in the present invention uses a group alignment mechanism to perform fine-grained alignment and fusion for adjacent frames of different frame rates, decouples adjacent frames from the time series relationship, and uses different alignment submodules for different groups, making the alignment more flexible. For example, the offset learned by the high-frame rate group with large motion is often large. Based on this, the present invention can realize efficient inter-frame alignment and motion compensation based on the spatiotemporal relationship between consecutive frames, and output the aligned deep features.

[0037] The alignment method mentioned in the present invention not only avoids the inaccuracy of single optical flow alignment, solves the limitations of traditional deformable convolution methods (insufficient alignment capabilities under large motions and complex scenes), but also avoids the computational cost of 3D convolution, that is, it achieves efficient and accurate cross-frame motion compensation with a lower computational cost.

[0038] In the embodiment, a cross-frame alignment submodule is designed to achieve alignment and motion between frames. First, a grouping strategy is used for the features of continuous video frames input by the feature extraction submodule to group and efficiently extract the motion offset between adjacent frames and the reference frame, then motion compensation is performed, and finally the aligned features are fused to achieve a decoupled inter-frame alignment.

[0039] Specifically, the input of the cross-frame alignment submodule can be continuous video frames of any scene (assuming that the features of 2n+1 frames are input in total, including the features of 2n adjacent frames and the features of 1 reference frame). In a specific embodiment, an efficient temporal grouping strategy is adopted to group the features of the input 2n+1 frames, wherein the features of the tth (t = n+1) frame are the features of the reference frame, and the features of the remaining 2n frames are the features of the adjacent frames. The frame alignment submodule is first performed according to the time series (i.e., frame rate) and is divided into n groups in total. For example, the features of the tnth, tth, and t+nth frames are one group, the features of the t-n+1th, tth, and t+n-1th frames are one group, and they are divided in sequence, and finally the features of the t-1th, tth, and t+1th frames are one group. Assume that the feature set of the input frame is: , represents the features of the i-th video frame, the grouping process can be expressed as follows:

[0040] in Represents a grouping operation. Note that t=n+1, that is, there are 2n+1 frames of features. This is the feature of the first frame, and so on. The frame rates in different groups are different, for example The frame rate in this group is n. The larger the frame rate, the farther the adjacent frames are from the reference frame. It also means that the motion offset of the adjacent frames in the group is larger, and the similarity with the reference frame decreases. In each group, a set of cascaded dilated convolutions and a layer of normal convolution are independently adopted to learn the motion offsets of adjacent frames and reference frames at the feature level. The dilated convolution group contains 3 dilated convolutions with a kernel size of 3, and the dilation rates are set to 1, 2, and 3 respectively. Compared with ordinary convolution, dilated convolution can expand the receptive field and enhance the representation ability without increasing the amount of calculation. It can effectively solve the problem that the learned offset may be small due to the small receptive field. The above-mentioned cascaded dilated convolution group is used to obtain the intermediate features for generating the final offset, and the single-layer ordinary convolution is used to map the intermediate features into the final offset. It should be noted that Figure 3 In the present invention, the single convolution layer is integrated into the deformable convolution alignment and is not explicitly drawn. Furthermore, the present invention inputs the offset and the features of the adjacent frames into the deformable convolution neural network to obtain the final aligned features.

[0041] In this embodiment, Figure 3 As shown, assuming that after group alignment, the features of the aligned 2n adjacent frames are output. Assuming that the features of the aligned 2n adjacent frames are ,in To align the features of the tnth frame (i.e. the first frame), and so on. Then connect the aligned features with the features of the tth frame (n+1th frame). Specifically, use the Concatenation operation to concatenate the above features on the channel. Next, use a convolution kernel with a size of 1 The convolutional network layer of 1 performs a deep fusion operation on the concatenated features, which can be explained by the following formula:

[0042] in, Representation 1 1 convolution function, Represents the Concatenation operation of features in the channel dimension. It is the result of fusing the reference frame features with the aligned adjacent frame features, that is, the output of the entire cross-frame alignment submodule.

[0043] In this embodiment, the cross-frame alignment submodule integrates the spatiotemporal grouping strategy, hole convolution, deformable convolution and other technologies, and realizes fine-grained alignment and motion compensation in a motion complexity decoupling manner, avoiding the problem of small learning offset caused by the small receptive field of conventional deformable convolution. The larger receptive field ensures the accuracy of motion offset learning. Thanks to this, the present invention can adapt to complex motion scenes, such as large motion scenes. At the same time, since the hole convolution does not increase the computational cost relative to ordinary convolution, the cross-frame alignment submodule can effectively balance the amount of calculation. This submodule ultimately realizes a novel spatiotemporal alignment, efficiently models the motion relationship between the same pixel in the reference frame and the adjacent frame, and effectively extracts useful information from the adjacent frames, thereby promoting the enhancement of video reconstruction performance.

[0044] The feature enhancement submodule continuously enhances the fused features to improve the model's expressiveness. Specifically, the feature enhancement submodule is used to extract and enhance useful information in the fused features, such as textures, details, and other high-frequency information that are conducive to super-resolution reconstruction, thereby improving the model's expressiveness; the upsampling reconstruction submodule amplifies the resolution of the enhanced features and reconstructs them into higher-resolution, higher-quality video frames. Specifically, the feature enhancement submodule includes no less than 3 feature enhancement blocks, each of which consists of a convolution kernel of size 3 3 regular convolutional layers and a convolution kernel size of 3 The cascaded dilated convolution comprises three cascaded dilated convolution layers with a convolution kernel size of 3.

[0045] During the specific implementation process, the feature enhancement submodule is used to refine the high-frequency information of the aligned and fused features to achieve further depth enhancement, fully retain information such as object texture and details, and use it to amplify the spatial resolution in the subsequent upsampling and reconstruction submodule.

[0046] In a preferred embodiment, the feature enhancement submodule consists of 6 efficient feature enhancement blocks, each of which consists of a convolution kernel with a size of 3 3 regular convolutional layers and a convolution kernel size of 3 3. The group convolution layer adopts the idea of ​​group convolution, which has lower computational complexity than ordinary convolution. Conventional convolution is used to solve the problem that channels of different groups are difficult to interact with each other caused by group convolution. Therefore, the efficient feature enhancement block in the present invention integrates the advantages of conventional convolution and group convolution, while effectively reducing the computational complexity of the model, which helps to achieve an effective balance between model performance and computational complexity.

[0047] With the help of a deep network architecture, the feature enhancement submodule can further refine and enhance the fused features, and extract information that is conducive to super-resolution reconstruction (such as high-frequency features such as details and textures). It is worth noting that the inventor used residual technology in the feature enhancement submodule, that is, the input features of the feature enhancement submodule and the outputs of the six efficient feature enhancement blocks were element-wise added as the final output of the feature enhancement submodule. In addition, similar residual technology is also used inside the efficient feature enhancement block, which will not be repeated here. The application of residual technology can effectively improve training efficiency, avoid problems such as gradient explosion, and enhance the representation ability of the model. It should be pointed out that the number of efficient feature enhancement blocks can be adjusted according to the actual use of the network model, for example, it can also be set to any one of 4, 8, and 10.

[0048] The upsampling reconstruction submodule uses a CNN-based upsampling method to amplify the resolution of the enhanced features and restore them to high-definition reference frames for output, which can restore low-quality, low-resolution videos of different scenes into high-resolution, high-definition videos.

[0049] In the specific implementation process, the upsampling reconstruction submodule, facing the feature map output by the feature enhancement submodule, first uses the CNN-based upsampling method to amplify the resolution, that is, the Pixel Shuffle technology (sub-pixel convolution operation) is used to first amplify the resolution of the feature map to achieve the upsampling function; then a layer of convolution is used to reconstruct the upsampled features into high-definition, noise-free, high-resolution video frames and output them, achieving video super-resolution in complex motion scenes, ensuring the processing effect of the super-resolution method on video frames of different scenes. Finally, the present invention successfully constructed a video super-resolution network model in complex motion scenes.

[0050] It should be noted that, for the sake of clarity and simplicity, the present invention does not provide detailed diagrams and formulaic descriptions of the feature extraction submodule, feature enhancement submodule, and upsampling reconstruction submodule. However, the present invention has clearly described the technologies used in these three submodules, and therefore, a person skilled in the art should be able to recognize the functions and effects of the submodules.

[0051] In S3, the video super-resolution model is trained and tested using a training data set and a test data set, respectively, specifically including: inputting the training data sets in a variety of different scenarios into the video super-resolution model in turn; during iterative training, calculating the loss value between the video frame generated by the video super-resolution model and the true value; training the video super-resolution model with the reduction of the loss value as the optimization goal, and synchronously updating and optimizing the network weights until convergence; and using the test data set to evaluate the reconstruction performance of the video super-resolution model. The loss value is calculated based on the Charbonnier loss function. This implementation improves the super-resolution effect of videos in complex motion scenes and enhances the quality and visual effects of the video.

[0052] During the specific implementation process, a pre-built neural network-based video super-resolution model for complex motion scenes is initialized, and the hyperparameters of the neural network are set, including the number of neural network layers, the number of feature channels, the learning rate, and the number of iterations.

[0053] Continuous video frames of different scenes are selected from the preprocessed training data set, and are sequentially input into the constructed deep learning-based video super-resolution network model for training. The continuous video frames to be input (i.e., training data set) should contain more than 50 scenes, and at least 5 consecutive video frames (including reference frames and adjacent frames) should be input each time.

[0054] Continuous video frames of different scenes are selected from the training set in turn, and training is iterated continuously. In each iterative training, a certain batch of continuous video frames will be selected. The selected video frames contain low-resolution, low-quality inputs and corresponding high-resolution video frames as true values ​​for supervised training. The final loss value is calculated based on the true value, the output of the model and the loss function, and the loss value is continuously optimized, and the network parameters of the video super-resolution model are continuously updated. During the iteration process, the changes in learning rate and the decreasing law of loss value can be observed to determine whether re-adjustment and training are needed. After the training is completed, the loss value tends to converge.

[0055] During each iterative training of the video super-resolution network, multiple batches of continuous video frames are selected from the training data set and input into the video super-resolution network model based on deep learning. The loss value of the video super-resolution network is calculated based on the loss function pre-set for the video super-resolution network.

[0056] The video super-resolution model in complex motion scenes uses the robust Charbonnier function as the loss function. Compared with L1 loss and L2 loss, the Charbonnier loss function is less likely to cause the image to be too smooth, making the reconstructed video more realistic. Moreover, compared with the commonly used L2 loss function, the proposed loss function has better compatibility with outliers and can improve the performance of reconstruction.

[0057] In order to perform global optimization for the super-resolution task, the present invention needs to input the continuous video frames to be super-resolution enhanced into the super-resolution model based on deep learning in complex motion scenes. After the model outputs the predicted video frames, the predicted video frames (i.e., the super-resolution reconstruction results of the reference frames) and the corresponding true value video frames (i.e., the high-resolution true values ​​of the reference frames) are input into the proposed Charbonnier loss function to obtain the final global loss value.

[0058] Calculate the global loss value of the video super-resolution network. In an optional implementation of this embodiment, calculate the loss value of the video super-resolution model in a complex motion scene, that is, the final global loss value. Specifically, assume , are the kth video frame after super-resolution enhancement of the model (i.e., the result of super-resolution reconstruction) and its corresponding true value, K is the number of samples of video frames that need super-resolution enhancement, is the final global loss function. Then the calculation formula of the Charbonnier loss function mentioned in the present invention is as follows: in, The parameter is a fixed value, usually a small constant. The present invention sets it to 1e-6, which can improve the processing capability of abnormal data and make the training process of the network model more stable, thereby enhancing the performance of the model.

[0059] According to the preset learning rate, the present invention sets the learning rate to 1e-4, and then uses the gradient descent algorithm to continuously iterate the training, and updates the weight parameters of each submodule of the network model, while making the loss value continuously decrease and converge. When the loss value converges, the selected network parameters can be used as the final optimized model parameters to generate a video super-resolution model in a complex motion scene.

[0060] According to the pre-divided data set, continuous video frames in the test data set are selected to test the video super-resolution model in complex motion scenes generated when the loss value converges to verify the actual super-resolution reconstruction performance of the model. After the test and verification, it can be used as the final video super-resolution model in complex motion scenes.

[0061] Another embodiment is used to illustrate the video super-resolution system, see Figure 4 , the system 400 comprises: The data set acquisition module 410 is used to acquire a video data set containing a variety of different scenes and perform preprocessing, and divide the preprocessed video data set into a training data set and a test data set; The model building module 420 is used to build a video super-resolution model, which includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule and an upsampling reconstruction submodule, wherein the feature extraction submodule is used to extract shallow features of input continuous video frames; the cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames for the shallow features output by the feature extraction submodule, so as to achieve cross-frame spatiotemporal information fusion and motion compensation; the feature enhancement submodule enhances the features after spatiotemporal information fusion output by the cross-frame alignment submodule; the upsampling reconstruction submodule performs resolution amplification on the enhanced features output by the feature enhancement submodule, and reconstructs the enhanced features after resolution amplification into video frames; A model training and testing module 430, configured to train and test the video super-resolution model using the training data set and the test data set, respectively; The video super-resolution implementation module 440 is used to implement video super-resolution using the trained and tested video super-resolution model.

[0062] In addition to the above modules, the system 400 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0063] The other specific working processes of the video super-resolution system 400 refer to the description of the above-mentioned video super-resolution method embodiment, which will not be repeated here.

[0064] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 5 The architecture of the computing device shown is implemented. Figure 5 The architecture of the computing device is shown. Figure 5 As shown, a computer system 510, a system bus 530, one or more CPUs 540, an input / output 520, a memory 550, etc. The memory 550 can store various data or files used for computer processing and / or communication and program instructions including the video super-resolution method of the embodiment executed by the CPU. Figure 5 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 5One or more components in the memory 550. The memory 550, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the video super-resolution method in the embodiment of the present invention (for example, the data set acquisition module 410, the model building module 420, the model training and testing module 430 and the video super-resolution implementation module 440 in the video super-resolution system 400). One or more CPUs 540 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 550, that is, to implement the above-mentioned video super-resolution method, and the method includes the following steps: Obtaining a video data set containing a variety of different scenes and preprocessing it, and dividing the preprocessed video data set into a training data set and a test data set; A video super-resolution model is established, and the video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule and an upsampling reconstruction submodule, wherein the feature extraction submodule is used to extract shallow features of input continuous video frames; the cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames for the shallow features output by the feature extraction submodule, so as to realize cross-frame spatiotemporal information fusion and motion compensation; the feature enhancement submodule enhances the features after spatiotemporal information fusion output by the cross-frame alignment submodule; the upsampling reconstruction submodule performs resolution amplification on the enhanced features output by the feature enhancement submodule, and reconstructs the enhanced features after resolution amplification into video frames; Using the training data set and the test data set to train and test the video super-resolution model respectively; Video super-resolution is achieved using the trained and tested video super-resolution model.

[0065] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute related operations in the video super-resolution method provided in any embodiment of the present invention.

[0066] The memory 550 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 550 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 550 may further include a memory remotely arranged relative to one or more CPUs 540, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0067] The input / output 520 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 520 may also include a display device such as a display screen.

[0068] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the video super-resolution method described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0069] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0070] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0071] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the above-mentioned video super-resolution method embodiment and will not be repeated here.

[0072] In summary, the technical solutions provided by the above embodiments are a video super-resolution method, system, electronic device and storage medium. The method includes establishing a video super-resolution model, wherein the feature extraction submodule fully extracts the features of the input continuous video frames through one or several cascaded convolution layers, laying the foundation for further realizing efficient motion compensation at the feature level; the cross-frame alignment submodule performs fine-grained alignment and fusion of adjacent frames of different frame rates through a group alignment mechanism, decouples adjacent frames from the time series relationship, and adopts different alignment submodules for different groups, so that the alignment has greater flexibility. For example, the biased image learned by the high frame rate group with large motion is The displacement is often too large. Based on this, the present invention can realize efficient inter-frame alignment and motion compensation based on the spatiotemporal relationship between consecutive frames, and output the aligned deep features. In addition, the cross-frame submodule based on the hole convolution group can quickly learn high-precision offsets; the alignment submodule method avoids the inaccuracy of single optical flow alignment, solves the limitations of traditional deformable convolution methods (insufficient alignment capabilities under large motion and complex scenes), and avoids the computational cost of 3D convolution, that is, it realizes efficient and accurate cross-frame motion compensation with a lower computational cost; the feature enhancement submodule consists of 6 efficient feature enhancement blocks, each of which consists of a convolution kernel size of 3 3 regular convolutional layers and a convolution kernel size of 3 3, the group convolution layer adopts the idea of ​​group convolution, which has lower computational complexity than ordinary convolution. Conventional convolution is used to solve the problem that channels of different groups are difficult to interact with each other caused by group convolution. The efficient feature enhancement block in the present invention integrates the advantages of conventional convolution and group convolution, while effectively reducing the computational complexity of the model, which helps to achieve an effective balance between model performance and computational complexity.

[0073] In summary, the video super-resolution method, system, electronic device and storage medium in complex motion scenes proposed in the present invention, through a group alignment mechanism, intelligently divides the input continuous video frames in time sequence, and improves the compatibility with complex motion scenes (especially large motion scenes); secondly, by quickly processing the features of different video frames, efficient spatiotemporal information fusion is achieved, and the information extraction capability is enhanced; furthermore, through a novel feature enhancement mechanism, the details and texture of the video frames can be fully maintained, which is beneficial to the final reconstruction result and makes the visual effect better. Thanks to this, the video super-resolution method, system, electronic device and storage medium described in the present invention can achieve higher super-resolution performance with a lower parameter amount, and can be well promoted and applied in actual scenes in the industry.

[0074] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0075] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A video super-resolution method, characterized in that: The method comprises the following steps: Obtaining a video data set containing a variety of different scenes and preprocessing it, and dividing the preprocessed video data set into a training data set and a test data set; A video super-resolution model is established, and the video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule and an upsampling reconstruction submodule, wherein the feature extraction submodule is used to extract shallow features of input continuous video frames; the cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames for the shallow features output by the feature extraction submodule, so as to realize cross-frame spatiotemporal information fusion and motion compensation; the feature enhancement submodule enhances the features after spatiotemporal information fusion output by the cross-frame alignment submodule; the upsampling reconstruction submodule performs resolution amplification on the enhanced features output by the feature enhancement submodule, and reconstructs the enhanced features after resolution amplification into video frames; Using the training data set and the test data set to train and test the video super-resolution model respectively; Video super-resolution is achieved using the trained and tested video super-resolution model.

2. The video super-resolution method according to claim 1, characterized in that: The cross-frame alignment submodule implements motion compensation according to different frame rate groups, including: based on different groups containing adjacent frames of different time series, aligning the features of adjacent frames and reference frames in a decoupled manner, and fusing the aligned adjacent frames with the reference frame through a convolutional layer, and inputting the fused features into the feature enhancement submodule.

3. The video super-resolution method according to claim 2, characterized in that: The cross-frame alignment submodule is specifically implemented by the following steps: Determine a reference frame and adjacent frames based on a feature sequence of an input video frame; Based on the fact that the frame rate is proportional to the motion offset of the adjacent frame relative to the reference frame and inversely proportional to the similarity between the adjacent frame and the reference frame, the feature sequences are grouped according to different frame rates, and the adjacent frames with different frame rates are grouped and aligned in a decoupled manner; A set of cascaded dilated convolutions and a layer of normal convolution are used in each group to effectively learn the motion offset between adjacent frames and the reference frame at the feature level; Input the features of each group offset and adjacent frames into a deformable convolutional neural network to obtain aligned features; The Concatenation operation is used to concatenate the reference frame features and the aligned features on the channel, and then a convolution layer is used to perform feature fusion to obtain the output features of the cross-frame alignment submodule.

4. The video super-resolution method according to claim 1, characterized in that: The video super-resolution model is trained and tested using the training data set and the test data set, respectively, specifically including: Inputting the training data sets under a plurality of different scenarios into the video super-resolution model in sequence; During iterative training, calculating the loss value between the video frame generated by the video super-resolution model and the true value; Taking reducing the loss value as an optimization goal, training the video super-resolution model, and simultaneously updating and optimizing the network model weights until convergence; The test dataset is used to evaluate the reconstruction performance of the video super-resolution model.

5. The video super-resolution method according to claim 4, characterized in that: The loss value is calculated based on the Charbonnier loss function.

6. The video super-resolution method according to claim 1, characterized in that: The feature enhancement submodule includes no less than 3 feature enhancement blocks, each of which consists of a convolution kernel with a size of 3 3 regular convolutional layers and a convolution kernel size of 3 The structure is composed of 3 group convolutional layers.

7. The video super-resolution method according to claim 3, characterized in that: The set of cascaded dilated convolutions includes 3 cascaded dilated convolutions with a convolution kernel size of 3.

8. A video super-resolution system, characterized in that: The system comprises: A data set acquisition module is used to acquire a video data set containing a variety of different scenes and perform preprocessing, and divide the preprocessed video data set into a training data set and a test data set; A model building module is used to build a video super-resolution model, and the video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule and an upsampling reconstruction submodule, wherein the feature extraction submodule is used to extract shallow features of input continuous video frames; the cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames for the shallow features output by the feature extraction submodule, so as to realize cross-frame spatiotemporal information fusion and motion compensation; the feature enhancement submodule enhances the features after spatiotemporal information fusion output by the cross-frame alignment submodule; the upsampling reconstruction submodule performs resolution amplification on the enhanced features output by the feature enhancement submodule, and reconstructs the enhanced features after resolution amplification into video frames; A model training and testing module, used to train and test the video super-resolution model using the training data set and the test data set respectively; The video super-resolution implementation module is used to implement video super-resolution using the video super-resolution model that has been trained and tested.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the video super-resolution method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the video super-resolution method according to any one of claims 1 to 7 are implemented.

Images