Video super-resolution method based on local feature enhancement

Abstract

The present invention discloses a video super-resolution method based on local features, and belongs to the technical field of video super-resolution. The present invention mainly includes the following steps: extracting features after preprocessing each frame image in a low-resolution video to obtain shallow features; inputting shallow features of each frame image and adjacent frames into a convolution layer to obtain inter-frame motion compensation information; using a multi-scale network structure, according to the inter-frame motion compensation information, bending and deforming the shallow features of adjacent frames, and aligning them to a target frame; fusing the aligned features of adjacent frames with the shallow features of the target frame to obtain the fused features of the target frame; reconstructing the fused features of the target frame in combination with its shallow features to obtain a high-resolution image; traversing each frame image to obtain a high-resolution video. The video super-resolution method proposed by the present invention can obtain high-quality output with rich details and textures.

Description

Technical Field

[0001] The present invention belongs to the technical field of video super-resolution, and in particular relates to a video super-resolution method based on local feature enhancement. Background Art

[0002] Multimedia devices such as images and videos are affected by uncontrollable factors during the acquisition, transmission, and storage process, which ultimately destroys the best presentation effect; at the same time, directly transmitting high-quality multimedia consumes a lot of resource bandwidth, so in reality, low-quality media is often transmitted during the transmission process. In addition, with the substantial increase in the computing power of hardware devices in recent years, users have higher and higher requirements for image quality. How to efficiently restore low-quality carriers to high-quality carriers has become a key research topic. Super-resolution refers to the process of converting low-resolution input into high-resolution output, which can effectively solve the above contradictions and has a large number of practical scenarios, such as medical image reconstruction, satellite image remote sensing, digital high-definition, microscopic imaging, game live broadcast, movie video, etc.

[0003] Traditional video super-resolution methods usually use affine transformation or probability statistics. Although these methods save computational costs, it is difficult to effectively estimate the complex motion process of objects in the video. As deep learning has achieved great success in various fields, video super-resolution algorithms based on deep learning have been widely studied. The network structure based on convolutional neural network (CNN) is currently one of the main super-resolution algorithms based on deep learning. This structure uses a sliding window method to extract features between adjacent frames and participate in the calculation.

[0004] Video is a spatiotemporal continuous sequence with a large degree of redundancy between video frames. Adjacent frames close to the target frame can often provide more useful information, and convolutional neural networks use feature extraction methods such as local sliding windows to extract and fuse features, making more efficient use of inter-frame information. Existing methods based on convolutional networks have low efficiency in using reference frames and generally have problems with inaccurate alignment between frames, which has caused certain bottlenecks in the performance of convolutional network-based video super-resolution methods. Summary of the invention

[0005] In order to solve the above technical problems, the present invention discloses a video super-resolution method based on local feature enhancement.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] A video super-resolution method based on long-range-short-range combination, comprising:

[0008] S1: Obtain the video to be processed and perform image enhancement frame by frame to obtain enhanced images of each frame;

[0009] S2: Extract shallow features of each frame enhanced image;

[0010] S3: Calculating inter-frame motion compensation information of each frame according to the enhanced image of each frame and the enhanced images of adjacent frames;

[0011] S4: The current frame image is regarded as the target frame, and the two frames before and after the target frame are adjacent frames. A multi-scale alignment method is adopted to use the inter-frame motion compensation information of the target frame to bend and align the shallow features of the adjacent frames to the shallow features of the target frame to obtain the final aligned features;

[0012] S5: Fusing the final aligned features of the adjacent frames with the shallow features of the target frame to obtain the fused features of the target frame;

[0013] S6: reconstructing the fusion features and shallow features of the target frame to obtain reconstructed features, upsampling interpolation and channel dimension transformation of the reconstructed features to obtain a high-resolution image of the target frame;

[0014] S7: Repeat S4 to S6 for the next frame image until all frames are traversed, and the high-resolution images of all frames obtained constitute a high-resolution video.

[0015] On the other hand, the present invention also proposes a video super-resolution system based on local feature enhancement, which is used to implement the above method.

[0016] Beneficial effects of the present invention:

[0017] The present invention designs a complete network structure of video super-resolution based on multi-scale local feature enhancement. The network includes a data preprocessing module, a feature extraction module, a motion compensation calculation module, a feature alignment module, a feature fusion module, and a super-resolution reconstruction module, which belongs to a multi-level network architecture. The network structure first uses the feature extraction module to calculate the shallow features of each frame of low-resolution image, and uses the motion compensation calculation module to calculate the relative motion between frames. The motion information and the shallow features of the target frame and the reference frame are input into the multi-scale alignment module for alignment to obtain the aligned features, and the aligned features of the target true nuclear reference frame are further fused to obtain the fused features of the target frame. Finally, the shallow features and fused features of the target frame are reconstructed and calculated to obtain high-resolution output. The multi-scale alignment method is used to enhance the alignment effect between frames, so that the super-resolution video has better detail texture. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0019] Figure 2 4 is a structural block diagram of a video super-resolution method based on local feature enhancement used in an embodiment of the present invention;

[0020] Figure 3 Flow chart of step S3 in the video super-resolution method in an embodiment of the present invention;

[0021] Figure 4 is a flowchart of step S4 in the video super-resolution method in an embodiment of the present invention;

[0022] Figure 5 Flow chart of step S6 in the video super-resolution method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] Super-resolution refers to the process of converting a low-resolution input image into a high-resolution output image, and has a large number of practical scenarios. For video super-resolution methods, the present invention makes full use of reference frames to improve the accuracy of inter-frame alignment and proposes a video super-resolution method based on local feature enhancement.

[0025] The flowchart of the video super-resolution method based on local feature enhancement of the present invention is as follows Figure 1 As shown, specifically including:

[0026] Step S1. Obtain a low-resolution sequence and perform mirror symmetry, horizontal 90° flip, and vertical 90° flip on each frame in each sequence to achieve image enhancement. The low-resolution video sequence after image enhancement is recorded as , where T represents the length of the video sequence, Indicates Frame enhanced image.

[0027] Step S2. Extract shallow features of each enhanced image frame. Frame Enhanced Image The convolutional layer is input to transform the channel dimension of the image from 3 to 64, and then feature extraction is performed through stacked convolutional layers to obtain the shallow features of each frame of the image.

[0028] The above shallow feature extraction process is expressed as:

[0029]

[0030] in, Indicates Frame low resolution image The shallow characteristics of Represents the stacked convolutional layer structure used to extract features, represents the activation function, Represents a convolutional layer. The spatial resolution of the feature is kept constant during feature extraction, and only the channel dimension is changed.

[0031] In the steps S1-S2, the present embodiment first uses rotation, symmetry and other means to perform image enhancement on the input low-resolution video sequence, and then extracts the spatial features of each frame image in the video sequence. The feature extraction module is usually composed of stacked convolutional layers, and because the convolutional layer has the characteristics of local connection and weight sharing, it is good at extracting the spatial features of the image. Therefore, the shallow features extracted from the original image can more completely retain the information of the original image. These simple shallow features can be used as guides to participate in the calculation in the subsequent alignment and fusion process, helping the network to converge quickly while improving the utilization efficiency of the features, ensuring the super-resolution reconstruction effect.

[0032] A core design point of the present invention is that the network uses a multi-scale local feature alignment module to align the shallow features of the reference frame to the shallow features of the target frame. The target frame is each frame, the reference frame is the two adjacent frames of the target frame, the next frame of the target frame is called the forward reference frame, and the previous frame is called the backward reference frame.

[0033] Step S3. Calculate the inter-frame motion compensation information between the target frame and the forward reference frame and the backward reference frame. The inter-frame motion compensation will be used as guidance information in the subsequent steps to guide the target frame features to align with the reference frame features. Frame Enhanced Image , adjacent frame enhanced image and Input the optical flow calculation network for calculation, such as Figure 3 As shown, specifically:

[0034] Step (3.1). The low-resolution video obtained in step S2 is Frame Enhanced Image and its backward reference frame enhanced image Input the pre-trained optical flow calculation network for feature calculation to obtain the backward inter-frame compensation information. The expression is as follows:

[0035]

[0036] in, Indicates Frame Enhanced Image To the Frame Enhanced Image Relative to the Frame Image Backward inter-frame compensation; represents the pre-trained optical flow computation network, Represents an activation function. The optical flow computing network (FlowNet) is a common neural network used to compute the motion of pixels at each point in a video image over time, which is not described in detail in this embodiment.

[0037] Step (3.2). The low-resolution video obtained in step S2 is Frame Enhanced Image and its forward reference frame enhanced image Input the pre-trained optical flow calculation network for feature calculation to obtain the forward inter-frame compensation information. The expression is as follows:

[0038]

[0039] in, Indicates Frame Enhanced Image To the Frame Enhanced Image Relative to the Frame Enhanced Image Forward inter-frame compensation; represents the pre-trained optical flow computation network, Represents the activation function. The optical flow calculation network used in steps (3.1) and (3.2) is the same network.

[0040] In particular, when calculating the inter-frame compensation of the first frame of a low-resolution video, that is, when the target frame is the first frame, let , when calculating the inter-frame compensation of the last frame of a low-resolution video, that is, when the target frame is the last frame, let .

[0041] In the step S3, motion compensation calculation is performed on each enhanced image frame in the low-resolution sequence, and the forward inter-frame compensation and backward inter-frame compensation of each frame image relative to the forward reference frame and the backward reference frame are calculated. During the calculation of this step, an implicit inter-frame alignment needs to be completed inside the alignment module. The inter-frame motion compensation information can be obtained by splicing the target frame and the adjacent frame images in the channel dimension and then passing through a series of convolution modules. Since the present embodiment uses an optical flow calculation network to calculate the inter-frame motion compensation, the forward inter-frame compensation and backward inter-frame compensation Also known as the Forward and backward optical flow of frame images.

[0042] In the present invention, implicit alignment between frames is selected, and the variable convolution guided by multi-scale optical flow is selected to perform frame bending and frame alignment. The backward inter-frame compensation information and the forward inter-frame compensation information of each enhanced image obtained in step S3 are used to align the backward reference frame and the forward reference frame to the target frame, and the method in the following step S4 is executed.

[0043] Step S4: The enhanced low-resolution video image obtained in step S2 is Shallow features of adjacent frames The inter-frame compensation obtained in step S3 , Perform implicit motion compensation, such as Figure 4 As shown, specifically:

[0044] Step (4.1). Frame Enhanced Image Shallow features Pass Frame Enhanced Image Shallow features For downsampling, the expression is as follows:

[0045]

[0046]

[0047]

[0048] in, Indicates Frame Enhanced Image Shallow downsampling features, Indicates Frame Enhanced Image Shallow downsampling features, Indicates Frame Enhanced Image Shallow downsampling features; represents the activation function, represents the convolutional layer, Represents a max pooling operation.

[0049] Step (4.2). Frame Image The corresponding backward inter-frame compensation and Frame Image The corresponding forward inter-frame compensation For downsampling, the expression is as follows:

[0050]

[0051]

[0052] in, Indicates Frame Enhanced Image The backward downsampled optical flow, Indicates Frame Enhanced Image The forward downsampled optical flow. The network structure used to implement downsampling in steps (4.1) and (4.2) is the same, and the downsampling multiple is the same.

[0053] Step (4.3). Frame Enhanced Image Shallow downsampling features and Frame Enhanced Image Shallow downsampling features Guided by the downsampled optical flow information in step (4.2), shallow downsample features to the low-resolution target frame To align, the expression is as follows:

[0054]

[0055]

[0056] in, Indicates Frame Enhanced Image After shallow downsampling alignment, Indicates Frame Enhanced Image After shallow downsampling alignment, Represents a bend alignment operation.

[0057] Step (4.4). Frame Enhanced Image Shallow downsampling of aligned features and Frame Enhanced Image Shallow downsampling of aligned features Respectively with Frame Enhanced Image Shallow downsampling features To perform fusion, the expression is as follows:

[0058]

[0059]

[0060] in, Indicates Frame Enhanced Image Backward downsampled optical flow The residual of express The mask, Indicates Frame Image Forward downsampled optical flow The residual of express The mask, represents the stacked convolutional layers, Represents a channel concatenation operation.

[0061] Step (4.5). Frame Enhanced Image The residual of the downsampled optical flow And the mask of downsampled optical flow Pass Frame Enhanced Image The residual of the downsampled optical flow And the mask of downsampled optical flow The input deformable convolution is used for secondary correction alignment, and the expression is as follows:

[0062]

[0063]

[0064] in, Indicates Frame Enhanced Image The downsampled aligned features, Indicates Frame Enhanced Image The downsampled aligned features, Represents deformable convolution.

[0065] Step (4.6). Frame Enhanced Image The downsampled aligned features Pass Frame Enhanced Image The downsampled aligned features Upsample to the original size and combine with S2 to get the Frame Enhanced Image Shallow features For feature fusion, the expression is as follows:

[0066]

[0067]

[0068] in, Indicates Frame Image After alignment, the fusion features are Indicates Frame Image After alignment, the fusion features are represents the stacked convolutional layers, Represents channel dimension concatenation.

[0069] Step (4.7). Frame Enhanced Image After alignment, the fusion features and Frame Enhanced Image After alignment, the fusion features To the Frame shallow features To align, the expression is as follows:

[0070]

[0071]

[0072] in, Indicates Frame Enhanced Image The shallow aligned features of Indicates Frame Enhanced Image The shallow aligned features of Represents a bend alignment operation.

[0073] Step (4.8). Frame Enhanced Image Shallow alignment features and Frame Enhanced Image Shallow alignment features Respectively with Frame Enhanced Image Shallow downsampling features After fusion, the fusion result is separated in the channel dimension to obtain the optical flow residual and mask, which are expressed as follows:

[0074]

[0075]

[0076] in, Indicates Frame Enhanced Image Backward optical flow The residual of express The mask, Indicates Frame Enhanced Image The forward optical flow The residual of express The mask, represents the stacked convolutional layers, Represents a channel concatenation operation.

[0077] Step (4.9). Residual of the backward optical flow of the frame , Optical Flow Mask Pass Residual of frame forward optical flow And optical flow mask The input deformable convolution is used for secondary correction alignment, and the expression is as follows:

[0078]

[0079]

[0080] in, Indicates Frame Enhanced Image The final aligned features of Indicates Frame Enhanced Image The final aligned features of Represents deformable convolution.

[0081] In the step S4, the shallow features of the reference frame obtained in step S2, the shallow features of the target frame and the optical flow information obtained in step S3 are aligned in multi-scales, and the reference frame image is bent and deformed toward the target frame image, and the reference frame is aligned with the target frame from coarse to fine. The optical flow information is used as the guiding information of the variable convolution, and the shallow features of the adjacent frames and the shallow features of the target frame are used as the residual values ​​and masks for calculating the optical flow, and finally the final aligned features of the reference frame and the target frame are obtained.

[0082] After obtaining the final aligned features of each enhanced image frame of the low-resolution video, the method in the following step S5 is performed on the final aligned features of the target frame and the reference frame.

[0083] Step S5. The final aligned features of adjacent frames of the low-resolution video sequence obtained in step S4 and shallow features of the target frame Perform feature fusion, specifically:

[0084] The low-resolution video obtained in step S2 is Frame Enhanced Image Shallow features , the first Frame Enhanced Image The final aligned features Pass Frame Enhanced Image The final aligned features For feature fusion, the expression is as follows:

[0085]

[0086] in, Indicates Frame Enhanced Image The fusion characteristics of represents the stacked convolutional layers used for fusion reconstruction; represents the activation function; Represents a channel concatenation operation.

[0087] Step S6. Combine the shallow features of the target frame obtained in step (2) , the fusion features of the target frame obtained in step (5) Perform feature reconstruction, such as Figure 5 As shown, specifically:

[0088] Step (6.1). Use the low-resolution video obtained in step (2) Frame Enhanced Image Shallow features And the fusion feature obtained in step (5) Perform feature reconstruction, the expression is as follows:

[0089]

[0090] in, Indicates Frame Enhanced Image The reconstruction features of represents the stacked convolutional layers used for fusion reconstruction; Represents a channel concatenation operation.

[0091] Step (6.2). Use the low-resolution video obtained in step (6.1) Frame Enhanced Image Reconstruction features Perform upsampling and feature reconstruction:

[0092]

[0093] in, Indicates Frame Enhanced Image High-resolution image after super-resolution; represents sub-pixel convolution; Represents the convolution operation, which is used to convert the Frame Enhanced Image Reconstruction features The channel dimension of is changed from 64 to 3.

[0094] The final super-resolution result That is, the high-resolution video is outputted by the super-resolution method of the present invention from the low-resolution video.

[0095] In this embodiment, a video super-resolution system based on local feature enhancement is also provided, which is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0096] The video super-resolution system based on local feature enhancement in this embodiment is designed with six modules: data preprocessing module, feature extraction module, motion compensation calculation module, feature alignment module, feature fusion module, and super-resolution reconstruction module. Among them, the data preprocessing module is used to obtain the video to be processed and perform image enhancement frame by frame, that is, to execute the method in step S1;

[0097] A feature extraction module is used to extract shallow features of each enhanced image frame, that is, to execute the method in step S2;

[0098] A motion compensation calculation module, used to calculate inter-frame motion compensation information of each frame, that is, to execute the method in step S3;

[0099] A feature alignment module is used to bend and align the shallow features of the adjacent frames to the shallow features of the target frame, that is, to execute the method in step S4;

[0100] A feature fusion module is used to fuse the final aligned features of adjacent frames with the shallow features of the target frame, that is, to execute the method in step S5;

[0101] The super-resolution reconstruction module is used to perform feature reconstruction, upsampling interpolation and channel dimension conversion on the fusion features of the target frame combined with the shallow features of the target frame to obtain a high-resolution video, that is, to execute the method in step S6.

[0102] The above network structure takes low-resolution video as input, first obtains shallow features through the feature extraction module, inputs the shallow features into the motion compensation calculation module to obtain motion compensation information, and inputs the motion compensation information into the feature alignment module for multiple frame alignment to obtain the aligned features of each frame image, and then passes the aligned features through the feature fusion module and the super-resolution reconstruction module for subsequent feature fusion and super-resolution reconstruction to obtain high-resolution video. Since this method designs a feature alignment module, the module ensures the accuracy of alignment through multiple alignments from coarse to fine, improves the super-resolution effect, and makes the super-resolution image have richer detail texture.

[0103] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0104] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0105] The above description is only a specific embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the concept of the present application. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A video super-resolution method based on local feature enhancement, characterized in that: include: S1: Obtain the video to be processed and perform image enhancement frame by frame to obtain enhanced images of each frame; S2: Extract shallow features of each frame enhanced image; S3: Calculating inter-frame motion compensation information of each frame according to the enhanced image of each frame and the enhanced images of adjacent frames; S4: The current frame image is regarded as the target frame, and the two frames before and after the target frame are adjacent frames. A multi-scale alignment method is adopted to use the inter-frame motion compensation information of the target frame to bend and align the shallow features of the adjacent frames to the shallow features of the target frame to obtain the final aligned features; S5: Fusing the final aligned features of the adjacent frames with the shallow features of the target frame to obtain the fused features of the target frame; S6: reconstructing the fusion features and shallow features of the target frame to obtain reconstructed features, upsampling interpolation and channel dimension transformation of the reconstructed features to obtain a high-resolution image of the target frame; S7: Repeat S4 to S6 for the next frame of image until all frames are traversed, and the high-resolution images of all frames obtained form a high-resolution video; The multi-scale alignment method described in S4 is specifically as follows: firstly, the shallow features of the target frame and the adjacent frames and the inter-frame motion compensation information of the target frame are downsampled, and the shallow features of the adjacent frames are bent and deformed toward the shallow features of the target frame using the inter-frame motion compensation information at the downsampled scale, and the features are aligned to obtain the downsampled aligned features; The downsampled and aligned features are then upsampled to the original scale, and the inter-frame motion compensation information is used to bend and align the shallow features of adjacent frames to the shallow features of the target frame at the original scale to obtain the final aligned features.

2. The video super-resolution method based on local feature enhancement according to claim 1, characterized in that: In the S1, the image enhancement includes mirror symmetry, horizontal 90° flipping and vertical 90° flipping.

3. The video super-resolution method based on local feature enhancement according to claim 1, characterized in that: The S2 includes: extracting features after expanding the channel dimension of each enhanced image frame obtained in step S1 through a convolution layer, keeping the spatial resolution unchanged, and obtaining shallow features of each enhanced image frame.

4. The video super-resolution method based on local feature enhancement according to claim 1, characterized in that: The S3 includes: for the t-th frame enhanced image, respectively calculating the backward inter-frame motion compensation of the t-th frame enhanced image according to the two previous and next enhanced frames; and forward inter-frame motion compensation 5. The video super-resolution method based on local feature enhancement according to claim 4, characterized in that: In S3, the backward inter-frame motion compensation and the forward inter-frame motion compensation The calculation method is: in, and represents the enhanced images of the t-1, t and t+1 frames, spy(.) represents the pre-trained optical flow calculation network, and sigmoid(.) represents the activation function.

6. The video super-resolution method based on local feature enhancement according to claim 4, characterized in that: In the above S4, the bending deformation and feature alignment of the shallow features of the adjacent frames toward the shallow features of the target frame specifically includes: S4.1: Shallow features of the target frame Shallow features of the frame before the target frame And the shallow features of the frame after the target frame Downsample separately to obtain shallow downsampling features and S4.2: Backward inter-frame motion compensation of the target frame and forward inter-frame motion compensation Downsample them respectively to get the backward downsampled optical flow and forward downsampled optical flow S4.3: Guided by the backward downsampled optical flow and the forward downsampled optical flow obtained in step S4.2, the shallow downsampled features of the adjacent frames obtained in step S4.1 are and Shallow downsampling features to the target frame Perform the bending alignment operation respectively to obtain the shallow downsampling alignment features and S4.4: Align the shallow downsampling features obtained in step S4.3 and Respectively with the shallow downsampling features of the target frame To perform the fusion: in, Represents the backward downsampled optical flow The residual, mask t-1→t ↓ express The mask, represents the forward downsampled optical flow The residual, mask t+1→t ↓ express The mask of , R(.) represents the stacked convolutional layer, and c(.) represents the channel concatenation operation; S4.5: Based on the residual and mask of the downsampled optical flow obtained in step S4.4, the shallow downsampled features of the adjacent frames are processed by deformable convolution and Perform a secondary correction alignment: in, represents the downsampled and aligned features of the previous frame of the target frame, represents the downsampled and aligned features of the frame after the target frame, and Dconv(.) represents deformable convolution; S4.6: Align the downsampled features obtained in step S4.5 and Upsample to the size of the original shallow features and compare with the shallow features of the target frame Perform feature fusion, which includes splicing according to channel dimensions to obtain aligned fusion features and S4.7: Align and fuse features and Perform bending alignment operations on the target frame images respectively to obtain shallow aligned features and S4.8: Align shallow layers to later features and Respectively with the shallow downsampling features of the target frame To perform the fusion: in, Represents backward optical flow The residual, mask t-1→t express The mask, Represents the forward optical flow The residual, mask t+1→t express Mask; S4.9: Based on the residual and mask of the backward optical flow and the residual and mask of the forward optical flow obtained in step S4.8, the shallow features of adjacent frames are secondarily corrected and aligned through deformable convolution: in, represents the final aligned features of the previous frame of the target frame, Represents the final aligned features of the frame after the target frame.

7. The video super-resolution method based on local feature enhancement according to claim 1, characterized in that: In the above S5, the feature fusion of the final aligned features of the adjacent frames and the shallow features of the target frame specifically includes: in, represents the fused features of the target frame, R(.) represents the stacked convolutional layer, conv(.) represents the convolutional layer, Relu(.) represents the activation function, and c(.,.) represents the channel splicing operation.

8. The video super-resolution method based on local feature enhancement according to claim 1, characterized in that: The S6 specifically includes: combining the fusion features of the target frame with the shallow features of the target frame through a channel splicing operation, and reconstructing the combined features to obtain a reconstructed feature. Reconstruction features Interpolate and compress its channel dimension to 3 to obtain a high-resolution image of the target frame after super-resolution 9. A video super-resolution system based on local feature enhancement, used to implement the method of claim 1, characterized in that: The system comprises: A data preprocessing module is used to obtain the video to be processed and perform image enhancement frame by frame; A feature extraction module is used to extract shallow features of each frame of enhanced image; A motion compensation calculation module, used to calculate inter-frame motion compensation information of each frame; A feature alignment module is used to bend and align the shallow features of adjacent frames to the shallow features of the target frame using a multi-scale alignment method; The feature fusion module is used to fuse the final aligned features of adjacent frames with the shallow features of the target frame; The super-resolution reconstruction module is used to reconstruct, upsample, interpolate and transform the channel dimension of the fused features of the target frame combined with the shallow features of the target frame to obtain a high-resolution video.

Images