Super-resolution video reconstruction method and device, equipment and medium

Abstract

The embodiment of the invention relates to the technical field of video reconstruction, in particular to a super-resolution video reconstruction method and device, equipment and a medium. The super-resolution video reconstruction method comprises the steps that a video sequence set generated according to a target video is acquired, the video sequence set comprises a plurality of video trainings, and each video sequence is composed of a plurality of continuous video frames; extracting mixed attention features and inter-frame optical flow features of each video frame in multiple scales from each video sequence; determining a video alignment feature of each video sequence according to the mixed attention feature and the inter-frame optical flow feature of each video frame at multiple scales; and reconstructing the video alignment features of the video sequences into a super-resolution video of the target video. The accuracy of inter-frame alignment in a complex scene is realized, and the details of the reconstructed video are enriched.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of video reconstruction, and in particular to a super-resolution video reconstruction method, device, equipment and medium. Background Art

[0002] Video Super-Resolution (VSR) reconstruction aims to reconstruct high-resolution video from low-resolution video. This technology has important application significance in video surveillance, medical imaging and digital entertainment. Compared with single image super-resolution reconstruction (SISR) technology, VSR technology needs to use the information of neighboring images to improve the reconstruction effect. Although the most basic approach is to regard VSR as an extension of SISR, since the position of the same object between adjacent frames in a video sequence may be misaligned, applying SISR to each frame separately may reduce the smoothness of the video and thus reduce the clarity of the overall picture. Therefore, the focus of VSR is to study how to better utilize the information between multiple frames and align the relationship between frames.

[0003] The two main approaches to multi-frame alignment are optical flow and deformable convolutional networks. Optical flow computes optical flow or pixel motion vectors between frames to warp adjacent frames to ensure accurate alignment with the central frame. In contrast, deformable convolutions achieve implicit alignment by adaptively computing motion offsets between frames. Both methods rely on the accuracy of motion calculations between different frames. However, these methods usually operate at a single scale and show limited adaptability and robustness in scenes involving large-scale and structurally complex displacements. Existing multi-scale feature alignment often adopts a pyramid structure. The three-layer pyramid structure allows alignment at different scales, which helps capture motion and deformation of different sizes, thereby improving the accuracy of alignment. However, alignment at each level of the pyramid may lead to the accumulation of errors, especially alignment errors at lower levels may affect the alignment results at higher levels. In addition, the alignment process may be limited by the edge regions of the image, resulting in boundary artifacts, which in turn affect the quality of the final super-resolution output. In addition, existing methods often ignore the potential relationship between frames in the temporal dimension. They usually splice aligned frames or features and fuse them through convolutional layers, but the limited receptive field of convolution limits the expressive power of features. Summary of the invention

[0004] The purpose of the present invention is to at least provide a super-resolution video reconstruction method, device, equipment and medium, which can at least solve the problem that inaccurate optical flow estimation and motion offset in the current VSR reconstruction model in complex scenes make it difficult to accurately align multiple frames, achieve the accuracy of inter-frame alignment in complex scenes, and enrich the details of the reconstructed video.

[0005] To solve the above technical problems, this application proposes four aspects.

[0006] In a first aspect, the present application provides a method for reconstructing a super-resolution video, comprising: obtaining a video sequence set generated according to a target video, wherein the video sequence set includes multiple video trainings, and each video sequence is composed of multiple continuous video frames; extracting hybrid attention features and inter-frame optical flow features of each video frame at multiple scales from each of the video sequences; determining video alignment features of each video sequence based on the hybrid attention features and the inter-frame optical flow features of each video frame at multiple scales; and reconstructing the video alignment features of each of the video sequences into a super-resolution video of the target video.

[0007] In some embodiments, the extracting of mixed attention features and inter-frame optical flow features of each video frame at multiple scales from each of the video sequences includes: sending the video sequence into a network architecture based on convolution and pooling layers to extract inter-frame optical flow features of each video frame at multiple scales; determining channel attention features of each video frame at multiple scales according to the video sequence; determining motion attention features of each video frame at multiple scales according to the video sequence; determining the mixed attention features of each video frame at multiple scales according to the motion attention features and the channel attention features of each video frame at multiple scales.

[0008] In some embodiments, the channel attention features of each video frame at each scale are determined according to the video sequence, including: using a global average pooling layer to extract the global spatial features of each video frame from the video sequence; reshaping the global spatial features of each video frame to obtain a first reshaping result; convolving the first reshaping result of each video frame to obtain a time domain enhancement feature; reshaping the time domain enhancement feature of each video frame to obtain a second reshaping result; channel decompressing the second reshaping result of each video frame, and sending it to an activation function after decompression to obtain an attention map of each video frame; determining the channel attention features of each video frame at a single scale according to the attention map and each video frame of the video sequence.

[0009] In some embodiments, motion attention features of each video frame at various scales are determined according to the video sequence, including: determining a first intermediate feature sequence according to the original features of the central frame and the original features of each left frame in the global spatial features, wherein the left frame is a video frame that is temporally located before the central frame in the video sequence; determining a second intermediate feature sequence according to the original features of each right frame and the original features of the central frame in the global spatial features, wherein the right frame is a video frame that is temporally located after the central frame in the video sequence; determining motion features of each video frame according to the first intermediate feature sequence, the second intermediate feature sequence and the original features of the central frame; and determining motion attention features of each video frame at a single scale according to the motion features and the video sequence.

[0010] In some embodiments, the video alignment feature of each video sequence is determined according to the hybrid attention features and the inter-frame optical flow features of each video frame at multiple scales, including: determining a current alignment strategy according to whether there is a scale alignment feature of the minimum scale; when there is no scale alignment feature of the minimum scale, the current alignment strategy is to align step by step from large to small scales, and generate a first alignment feature step by step according to the hybrid attention features and the inter-frame optical flow features of the corresponding scales; when there is a scale alignment feature of the minimum scale, the current alignment strategy is to align step by step from small to large scales, and generate a second alignment feature step by step according to the hybrid attention features and the inter-frame optical flow features of the corresponding scales, and output the second alignment feature of the maximum scale as the video alignment feature of the video sequence.

[0011] In some embodiments, the mixed attention feature includes: a central mixed feature of a central frame and a neighboring frame mixed feature of other frames; the first alignment feature is generated step by step according to the mixed attention feature of the corresponding scale and the inter-frame optical flow feature, including: determining whether the first target scale is the maximum scale; when the first target scale is not the maximum scale, obtaining the first alignment feature of the previous scale of the first target scale; determining the first central mixed feature of the first target scale according to the first alignment feature; determining the first alignment feature of the first target scale according to the first central mixed feature and each neighboring frame mixed feature corresponding to the first target scale and the inter-frame optical flow feature; when the first target scale is the maximum scale, determining the first alignment feature of the first target scale according to the central mixed feature of the corresponding scale, each neighboring frame mixed feature and the inter-frame optical flow feature.

[0012] In some embodiments, the step-by-step generation of the second alignment feature based on the mixed attention feature and the inter-frame optical flow feature of the corresponding scale includes: obtaining a second alignment feature of a scale below the second target scale; obtaining a first central mixed feature of the first alignment feature corresponding to the second target scale; determining a second central mixed feature of the second target scale based on the second alignment feature and the first central mixed feature; determining a second alignment feature of the second target scale based on the second central mixed feature and each adjacent frame mixed feature corresponding to the second target scale and the inter-frame optical flow feature.

[0013] In a third aspect, the present application provides a super-resolution video reconstruction device, comprising: a first acquisition module, used to acquire a video sequence set generated according to a target video, wherein the video sequence set includes multiple video trainings, and each video sequence is composed of multiple continuous video frames; a first execution module, used to extract hybrid attention features and inter-frame optical flow features of each video frame at multiple scales from each of the video sequences; a first determination module, used to determine the video alignment features of each video sequence based on the hybrid attention features and the inter-frame optical flow features of each video frame at multiple scales; a second execution module, used to reconstruct the video alignment features of each of the video sequences into a super-resolution video of the target video.

[0014] In a third aspect, the present application proposes a computer electronic production device, characterized in that it includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0015] In a fourth aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0016] The hybrid attention features extracted by this application provide useful feature information for subsequent inter-frame alignment. Secondly, this application uses a U-shaped alignment strategy, which starts from large-size features, then downsamples and aligns to the minimum size step by step, and then reversely aligns back to the original size step by step through upsampling and jump connections, ensuring the accuracy of inter-frame alignment in complex scenes. In addition, after each level of alignment, the reconstruction details are further enriched. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0018] Figure 1 A main flow chart of a super-resolution video reconstruction method provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a structure for determining mixed attention features provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a feature alignment process provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a process for aggregating features into alignment features provided in an embodiment of the present application;

[0022] Figure 5 This is a main structural block diagram of a super-resolution video reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0024] Video Super-Resolution (VSR) reconstruction aims to reconstruct high-resolution video from low-resolution video. This technology has important application significance in video surveillance, medical imaging and digital entertainment. Compared with single image super-resolution reconstruction (SISR) technology, VSR technology needs to use the information of neighboring images to improve the reconstruction effect. Although the most basic approach is to regard VSR as an extension of SISR, since the position of the same object between adjacent frames in a video sequence may be misaligned, applying SISR to each frame separately may reduce the smoothness of the video and thus reduce the clarity of the overall picture. Therefore, the focus of VSR is to study how to better utilize the information between multiple frames and align the relationship between frames.

[0025] The two main approaches to multi-frame alignment are optical flow and deformable convolutional networks. Optical flow computes optical flow or pixel motion vectors between frames to warp adjacent frames to ensure accurate alignment with the central frame. In contrast, deformable convolutions achieve implicit alignment by adaptively computing motion offsets between frames. Both methods rely on the accuracy of motion calculations between different frames. However, these methods usually operate at a single scale and show limited adaptability and robustness in scenes involving large-scale and structurally complex displacements. Existing multi-scale feature alignment often adopts a pyramid structure. The three-layer pyramid structure allows alignment at different scales, which helps capture motion and deformation of different sizes, thereby improving the accuracy of alignment. However, alignment at each level of the pyramid may lead to the accumulation of errors, especially alignment errors at lower levels may affect the alignment results at higher levels. In addition, the alignment process may be limited by the edge regions of the image, resulting in boundary artifacts, which in turn affect the quality of the final super-resolution output. In addition, existing methods often ignore the potential relationship between frames in the temporal dimension. They usually splice aligned frames or features and fuse them through convolutional layers, but the limited receptive field of convolution limits the expressive power of features.

[0026] In order to solve the above technical problems, the present invention proposes a super-resolution video reconstruction method. The implementation details of the method for determining the bandwidth of this embodiment are specifically described below. The following content is only the implementation details provided for easy understanding and is not necessary for implementing this solution.

[0027] Embodiment 1:

[0028] The present application provides a method for reconstructing a super-resolution video, and the method is applied to electronic production equipment, and the electronic production equipment can be a server, a mobile terminal, a computer, a cloud platform, etc. The function implemented by the production equipment data processing provided in the embodiment of the present application can be implemented by calling a program code by a processor of the electronic production equipment, wherein the program code can be stored in a computer storage medium. The method for reconstructing a super-resolution video includes:

[0029] Step S1: obtaining a video sequence set generated according to a target video, wherein the video sequence set includes a plurality of video trainings, and each video sequence is composed of a plurality of continuous video frames.

[0030] To reconstruct a low-resolution video into a super-resolution video, it is necessary to first improve the resolution of each frame in the video. However, the resolution of a single frame cannot be improved. Therefore, in this application, the target video is split into multiple video sequences, each of which contains multiple continuous video frames. Generally, a video sequence contains 7 video frames, and there is only one frame difference between two adjacent video sequences. The resolution of the central frame of each video sequence is improved through the multiple video frames in the video sequence.

[0031] Step S2: extracting hybrid attention features and inter-frame optical flow features at multiple scales for each video frame from each of the video sequences.

[0032] The limited receptive field of convolution limits the effect of video sequence feature extraction, and the self-attention structure focuses on feature extraction in the spatial dimension, but pays insufficient attention to the temporal correlation between features. Therefore, the present invention fully combines the characteristics of convolution and self-attention structures, and integrates convolution into the self-attention structure to obtain richer visual representation.

[0033] In some embodiments, Figure 2 As shown, step S2 "extracting mixed attention features and inter-frame optical flow features at multiple scales for each video frame from each of the video sequences" includes:

[0034] Step S21: Send the video sequence into a network architecture based on convolution and pooling layers to extract inter-frame optical flow features of each video frame at multiple scales.

[0035] Step S22: determining channel attention features of each video frame at multiple scales according to the video sequence.

[0036] Step S23: determining motion attention features of each video frame at multiple scales according to the video sequence.

[0037] Step S24: Determine the mixed attention features of each video frame at multiple scales based on the motion attention features and the channel attention features of each video frame at multiple scales.

[0038] This application combines the channel attention module and the motion attention module to form a hybrid attention module, and extracts the hybrid attention features of the video sequence through the hybrid attention module. Of course, the extracted hybrid attention is the hybrid attention feature of each frame in the video sequence.

[0039] In some embodiments, step S22 “determining channel attention features of each video frame at multiple scales according to the video sequence” includes:

[0040] Step S221: extracting global spatial features of each video frame from the video sequence using a global average pooling layer.

[0041] Step S222: reshape the global spatial features of each video frame to obtain a first reshape result.

[0042] Step S223: performing convolution on the first reshaping result of each video frame to obtain a time domain enhancement feature.

[0043] Step S224: reshape the time domain enhancement feature of each video frame to obtain a second reshape result.

[0044] Step S225: perform channel decompression on the second reshaping result of each video frame, and send it into the activation function after decompression to obtain the attention map of each video frame.

[0045] Step S226: Determine a channel attention feature of each video frame at a single scale according to the attention map and each video frame of the video sequence.

[0046] Taking a hybrid attention block as an example, immediately after the first layer is normalized, the channel motion attention module we proposed is paralleled with the window-based multi-head self-attention module, and a small constant α is multiplied on the output of the channel motion attention module to effectively handle the possible conflicts between the hybrid attention in terms of optimization and visual representation. For a given input sequence feature X, the process of the hybrid attention block can be expressed as follows:

[0047]

[0048] Where X N and X M represents the intermediate feature, Y represents the output of a hybrid attention block, LN represents the layer normalization operation, (S)W-MSA represents (shifted) window multi-head self-attention, CMA represents channel motion attention, and MLP represents multi-layer perceptron. The hybrid attention module contains three hybrid attention groups to extract features of different scales. Each hybrid attention group integrates 8 hybrid attention blocks. Each hybrid attention group is connected by a convolution with a stride of 2 to downsample the features.

[0049] The channel-motion attention module is a key component of the hybrid attention module, which contains two attention mechanisms: channel attention and motion attention. The integration of these two mechanisms enables the model to effectively identify and emphasize relevant features across spatial and temporal dimensions. In order to pay attention to the importance of different frames, our channel attention module adds 1D temporal convolution between the compression and decompression stages to enhance the interdependence of channels in the temporal domain. Figure 3 As shown, specifically, given an X∈RB ×T×C×H×W , we first extract the global spatial features of the input features through spatial global average pooling, and then perform channel compression (r = 16) on the global spatial features to reduce the feature channels to improve efficiency, which can be expressed as

[0050] X r =SqConv(Apool(X))

[0051] Where X r is the global space feature, Apool represents the spatial global average pooling layer, and SqConv represents the 2D convolution with a convolution kernel of 1. Afterwards, we transform the global spatial feature X r Reshape as the first reshape result And the time domain features are enhanced by 1D convolution with a convolution kernel of 3, which can be expressed as

[0052]

[0053] Where X temp1 X temp It is the time domain enhancement feature. Then the time domain enhanced feature X temp1 Reshape to the second reshape result After that, it is decompressed by a 2D convolution with a convolution kernel of 1, and then fed into the activation function to obtain the channel attention map, which can be expressed as

[0054]

[0055] Finally, the feature X of the input is multiplied by the obtained attention map to stimulate the sensitive channel. However, stimulating the sensitive channel alone will suppress the background information. Therefore, a residual connection is used to add the result of the dot product to X to stimulate the sensitive channel while retaining the scene information. It can be expressed as

[0056]

[0057] in Represents the dot product operation, and finally obtains the channel attention feature X c .

[0058] In some embodiments, step S23 “determining motion attention features of each video frame at multiple scales according to the video sequence” includes:

[0059] Step S231: determining a first intermediate feature sequence according to the original features of the central frame and the original features of each left frame in the global spatial features, wherein the left frame is a video frame that is located before the central frame in time sequence in the video sequence.

[0060] Step S232: determining a second intermediate feature sequence according to the original features of each right frame in the global spatial features and the original features of the center frame, wherein the right frame is a video frame that is located after the center frame in the video sequence.

[0061] Step S233: determining the motion features of each video frame according to the first intermediate feature sequence, the second intermediate feature sequence and the original features of the central frame.

[0062] Step S234: Determine the motion attention features of each video frame at a single scale according to the motion features and the video sequence.

[0063] In the design of the motion attention module, we extract motion information by analyzing the feature differences between the center frame and adjacent frames. To adapt to the ordered nature of the time series, we divide the frame features into two groups according to their temporal position relative to the center frame: for the left sequence, the center frame features are first convolved and then the left frame features are subtracted from the result; for the right sequence, the right frame features are first convolved and then the center frame features are subtracted. In addition, 0 padding is used when processing the center frame features to enhance the model's sensitivity to motion changes. Next, the motion features obtained above are used to generate modulation weights, which adjust the contribution of adjacent frame features to emphasize motion-sensitive information. This method encourages the model to identify and enhance key differential information, thereby optimizing the model's ability to capture inter-frame motion information, which is beneficial to motion compensation between subsequent frames.

[0064] Specifically, given the feature global space features compressed by The motion feature is obtained by subtracting the features of adjacent frames from the central frame, which can be expressed as

[0065]

[0066] Where Conv represents a 2D convolution with a convolution kernel of 3. represents the characteristics of the center frame, Represents the features of the frame sequence on the left and right sides of the central frame, X temp2 For sports characteristics, Then use spatial average pooling and decompression modules and residual connections to process X temp2 Thus outputting motion attention feature X m ,X m ∈R B×T×C×H×W .

[0067] Finally, channel attention and motion attention are integrated into hybrid attention features, where the output of the CMA module ensures that the network focuses on the most relevant features and can comprehensively model the temporal information of multiple frames through motion perception.

[0068] Therefore, the mixed attention features of each video frame obtained through step S2 include: the central mixed features of the central frame and the mixed features of the adjacent frames of other frames, and the inter-frame optical flow features obtained are the inter-frame optical flow features between each other frame and the central frame. After steps S2 to S234, the mixed features and inter-frame optical flow features of each video frame are successfully extracted from each video sequence. Next, these features need to be aligned at different scales to form a super-resolution video frame corresponding to the central frame in each video sequence.

[0069] Step S3: Determine the video alignment features of each video sequence according to the mixed attention features and the inter-frame optical flow features of each video frame at multiple scales.

[0070] In some embodiments, step S3 “determining the video alignment feature of each video sequence according to the hybrid attention feature and the inter-frame optical flow feature of each video frame at multiple scales” includes:

[0071] Step S31: determining the current alignment strategy according to whether there is a scale alignment feature with a minimum scale.

[0072] Step S32: When there is no scale alignment feature of the minimum scale, the current alignment strategy is to align step by step from large to small scales, and generate the first alignment feature step by step according to the mixed attention feature and the inter-frame optical flow feature of the corresponding scale.

[0073] Step S33: When there is a scale alignment feature with a minimum scale, the current alignment strategy is to align the scales step by step from small to large, and generate a second alignment feature step by step according to the mixed attention feature and the inter-frame optical flow feature of the corresponding scale, and output the second alignment feature with the maximum scale as the video alignment feature of the video sequence.

[0074] In the case of large motion, aligning adjacent frames with the reference frame at different levels is the key to handling large motion scenes. Generally speaking, low-level alignment mainly deals with larger motion and deformation. This is because the image resolution of the low-level image is low, so it focuses on global motion and rough feature matching. Low-level alignment mainly provides a constraint for high-level alignment. Since the image resolution of the high-level image is high, the alignment at the high-level image focuses on more detailed features and small-scale motion. However, since the existing three-level pyramid alignment method aligns from low level to high level step by step, once an alignment error occurs at a low level, it will affect the alignment of the next level, resulting in error accumulation. Therefore, we propose a U-shaped multi-scale feature alignment module, which starts from large-scale features, aligns step by step to the minimum size, and then reversely aligns step by step back to the original size through skip connections. The advantages of this alignment are (1) first fine feature matching is performed at a high level, and then further optimization and precise global motion are performed at a low level, thereby finding a better balance. (2) Aligning at a high level first can detect possible alignment problems, such as occlusion or fast motion, earlier, and then these problems can be dealt with in a targeted manner at the lower level, thereby enhancing the robustness of the algorithm. (3) Skip connections ensure the full utilization of high- and low-frequency information in the image. In order to distinguish the process in the U-shaped alignment process, the features aligned from large to small scales are called the first alignment features, and the features aligned from small to large scales are called the second alignment features.

[0075] In some embodiments, the alignment process is as follows Figure 3 As shown, step S32 "generates first alignment features step by step according to the mixed attention features and the inter-frame optical flow features of the corresponding scale", includes:

[0076] Step S321: Determine whether the first target scale is the maximum scale.

[0077] Step S322: when the first target scale is not the maximum scale, obtaining a first alignment feature of a scale above the first target scale.

[0078] Step S323: determining a first central mixed feature of the first target scale according to the first alignment feature.

[0079] Step S324: determining a first alignment feature of the first target scale according to the first central mixed feature and each adjacent frame mixed feature corresponding to the first target scale and the inter-frame optical flow feature.

[0080] Step S325: when the first target scale is the maximum scale, determining a first alignment feature of the first target scale according to a central mixed feature of a corresponding scale, a mixed feature of each adjacent frame, and an inter-frame optical flow feature.

[0081] In some embodiments, step S33 “generating second alignment features step by step according to the mixed attention features and the inter-frame optical flow features of the corresponding scale” includes:

[0082] Step S331: Acquire a second alignment feature at a scale next to the second target scale.

[0083] Step S332: Acquire a first central mixed feature of a first alignment feature corresponding to the second target scale.

[0084] Step S333: determining a second central mixed feature of the second target scale according to the second alignment feature and the first central mixed feature.

[0085] Step S334: determining a second alignment feature of the second target scale according to the second central mixed feature and each adjacent frame mixed feature corresponding to the second target scale and the inter-frame optical flow feature.

[0086] This application takes the alignment of three scales as an example. When forming the first alignment feature of the second or third scale, it is necessary to obtain the central frame mixed attention feature of the second scale alignment, that is, the first central mixed attention feature, from the alignment feature of the first scale, that is, the first alignment feature, by downsampling.

[0087] After obtaining the central mixed features of the second scale, the first alignment features of the second scale are obtained by combining the neighboring frame mixed features of other frames and the inter-frame optical flow features in the mixed attention features corresponding to the second scale.

[0088] In the process of obtaining the first alignment feature of the second scale through the first center mixed feature of the second scale, the adjacent frame mixed feature and the inter-frame optical flow feature, the adjacent frame mixed feature of other video frames is first distorted using the inter-frame optical flow feature to obtain a distorted feature. The distorted feature is concatenated with the first center mixed feature and the inter-frame optical flow feature, and then convolution is used to learn the offset between frames. Finally, the adjacent frame mixed feature and the offset are sent to the deformable volume block to obtain the final alignment feature.

[0089] Subsequently, the aligned features are aggregated with the features of the center frame through the non-local feature aggregation module. The non-local feature aggregation module uses non-local operations to capture long-range dependencies, thereby enhancing the representation ability of features globally. In particular, at the second and third scale layers, the features of the center frame are obtained by upsampling and downsampling from the output of the previous layer. Among them, downsampling is achieved by strided convolution and upsampling is achieved by sub-pixel convolution. This means that at each level, the features are processed based on the information of the previous level, allowing the model to capture and utilize motion and feature information at different scales. The operation at each scale can be expressed as follows:

[0090]

[0091] Where T ref2 ,T neig2 , f2 represent the first center mixed feature of the first scale, the mixed feature of adjacent frames and the optical flow feature between frames, respectively. FG-DCN represents the flow-conductance deformable convolution module, L NLFA represents the non-local feature aggregation module, T Aligned2 represents the compensation characteristic, F Aligned2 The final alignment feature of the second scale. Finally, the feature F Aligned2 The final alignment features are obtained after upsampling and downsampling fusion.

[0092] like Figure 4 As shown in the figure, the compensation features are input into three 1x1 convolutional layers (respectively denoted as g, u, and v) to extract different feature maps. The similarity score of the outputs of u and v is calculated (using the softmax function) and this score is multiplied by the output of g. The output of this step is processed by another 1x1 convolution z and added to the first alignment feature to obtain the enhanced alignment feature. This process can be expressed as:

[0093]

[0094] Next, the compensation feature is subjected to non-local operation with the first central mixed feature to further enhance the feature expression capability. This process can be expressed as

[0095]

[0096] F Aligned2 =Contact(F a ,F b )

[0097] Among them, by integrating F a 、F b The two parts of the features get the second scale alignment, which is very difficult to pass F Aligned2 ,Through such operations, we not only improve the accuracy of feature ,alignment and enhance the detail expression of the aligned features, but also avoid ,alignment error accumulation and boundary artifacts.

[0098] However, for the alignment of the first scale, since the first scale cannot obtain the alignment features of the previous scale through downsampling, the first central mixed feature of the first scale is the central mixed feature of the corresponding scale in the mixed attention feature.

[0099] The above process is only the alignment of the first half of the U-shaped alignment model. After completing the alignment features of the smallest scale, it enters the second half of the U-shaped alignment module. In the second half of the alignment, the alignment is based on the scale from small to large, so

[0100] The above process is applicable to all processes except the first scale to generate the first alignment feature. Of course, when generating the second scale alignment feature, the next scale needs to be obtained, that is, the upsampling process, and the central mixed feature of the corresponding scale, that is, the first central mixed feature, needs to be combined. Taking the second scale as an example again, after completing the feature alignment of the minimum scale, upsampling is performed through the alignment feature of the minimum scale, that is, the second alignment feature of the next scale is upsampled, and the first central mixed feature of the scale when the first feature is aligned is obtained at the same time. The first central mixed feature and the upsampling result are superimposed to obtain the second central mixed feature, that is, the central mixed feature of the second scale when determining the second alignment feature, and then the adjacent frame mixed features of other frames in the obtained mixed attention feature and the inter-frame optical flow feature are combined to form the second alignment feature of the second scale. After the second alignment feature of the maximum scale is obtained, the second alignment feature of the maximum scale is output as the video alignment feature.

[0101] Step S4: reconstructing the video alignment features of each of the video sequences into a super-resolution video of the target video.

[0102] In this application, for a given 2N+1 continuous low-resolution frames, we use the adjacent 2N frames to output the super-resolution result of the intermediate frame. The model consists of three modules, including our hybrid attention module, multi-scale feature alignment aggregation module, and feature reconstruction module. First, the low-resolution frame sequence is sent to the hybrid attention module to extract features of three scales, and the low-resolution sequence is sent to the network architecture based on convolution and pooling layers to extract optical flows of three scales for subsequent alignment; then, the features of each adjacent frame and the central frame and the inter-frame optical flow are sent to the multi-scale feature alignment aggregation module for alignment to obtain the aligned features; then, each obtained aligned feature is spliced ​​with the features of the central frame and sent to the feature reconstruction module composed of multiple residual blocks. Finally, the feature reconstruction is performed through the sub-pixel convolution layer, and the high-resolution frame is obtained by adding the image directly upsampled from the central frame, and supervised training is performed with the original high-resolution frame. Therefore, compared with other video super-resolution methods based on deep learning, this application performs coarse-to-fine alignment between frames based on a U-shaped multi-scale structure. First, this model innovatively combines channel motion attention with the self-attention module, providing useful feature information for subsequent inter-frame alignment. Secondly, this model uses a U-shaped alignment strategy, which starts from large-scale features, then downsamples and aligns them to the minimum size step by step, and then reversely aligns them back to the original size step by step through upsampling and skip connections, ensuring the accuracy of inter-frame alignment in complex scenes. In addition, after alignment at each level, the non-local feature aggregation module is used to further enrich the reconstruction details.

[0103] Embodiment 2:

[0104] Based on the foregoing embodiments, the embodiments of the present application provide a super-resolution video reconstruction device, and the modules included in the device, as well as the units included in the modules, can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), a digital signal processor (DSP, Digital Signal Processing) or a field programmable gate array (FPGA, Field Programmable Gate Array), etc.

[0105] like Figure 5 As shown, a super-resolution video reconstruction device includes: a first acquisition module 1, a first execution module 2, a first determination module 3 and a second execution module 4.

[0106] The first acquisition module 1 is used to acquire a video sequence set generated according to a target video, wherein the video sequence set includes multiple video trainings, and each video sequence is composed of multiple continuous video frames. The first execution module 2 is used to extract the mixed attention features and inter-frame optical flow features of each video frame at multiple scales from each of the video sequences. The first determination module 3 is used to determine the video alignment features of each video sequence based on the mixed attention features and inter-frame optical flow features of each video frame at multiple scales. The second execution module 4 is used to reconstruct the video alignment features of each of the video sequences into a super-resolution video of the target video.

[0107] Each module in the above-mentioned super-resolution video reconstruction device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the device in the form of hardware, or can be stored in the memory in the processing device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0108] Embodiment 3:

[0109] In a third aspect, the present application provides a computer electronic production device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the methods described in the first aspect.

[0110] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0111] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0112] Embodiment 4:

[0113] In a fourth aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0114] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0115] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A super-resolution video reconstruction method, characterized in that: include: Acquire a video sequence set generated according to a target video, wherein the video sequence set includes a plurality of video trainings, and each video sequence is composed of a plurality of continuous video frames; Extracting hybrid attention features and inter-frame optical flow features at multiple scales for each video frame from each of the video sequences; Determine a video alignment feature for each video sequence based on the hybrid attention feature and the inter-frame optical flow feature at multiple scales of each video frame; The video alignment features of each of the video sequences are reconstructed into a super-resolution video of the target video.

2. The method according to claim 1, characterized in that Extracting mixed attention features and inter-frame optical flow features at multiple scales for each video frame from each video sequence includes: The video sequence is fed into a network architecture based on convolution and pooling layers to extract inter-frame optical flow features of each video frame at multiple scales; Determining channel attention features at multiple scales for each video frame according to the video sequence; Determine motion attention features at multiple scales for each video frame according to the video sequence; The mixed attention features of each video frame at multiple scales are determined according to the motion attention features and the channel attention features of each video frame at multiple scales.

3. The method according to claim 2, characterized in that Determine channel attention features of each video frame at each scale according to the video sequence, including: Extracting global spatial features of each video frame from the video sequence using a global average pooling layer; Reshaping the global spatial features of each video frame to obtain a first reshaping result; Convolving the first reshaping result of each video frame to obtain a time domain enhancement feature; Reshaping the time domain enhancement feature of each video frame to obtain a second reshaping result; Decompress the second reshaped result of each video frame through a channel, and feed the decompressed result into an activation function to obtain an attention map of each video frame; A channel attention feature of each video frame at a single scale is determined according to the attention map and each video frame of the video sequence.

4. The method according to claim 3, characterized in that Determine the motion attention features of each video frame at various scales according to the video sequence, including: Determining a first intermediate feature sequence according to the original features of the central frame and the original features of each left frame in the global spatial features, wherein the left frame is a video frame that is located before the central frame in time sequence in the video sequence; Determining a second intermediate feature sequence according to the original features of each right frame in the global spatial features and the original features of the center frame, wherein the right frame is a video frame that is located after the center frame in time sequence in the video sequence; Determine the motion feature of each video frame according to the first intermediate feature sequence, the second intermediate feature sequence and the original feature of the central frame; A motion attention feature at a single scale is determined for each video frame according to the motion feature and the video sequence.

5. The method according to claim 1, characterized in that The step of determining the video alignment feature of each video sequence according to the mixed attention feature and the inter-frame optical flow feature of each video frame at multiple scales comprises: Determine the current alignment strategy according to whether there is a scale alignment feature of the minimum scale; When there is no scale alignment feature of the minimum scale, the current alignment strategy is to align the scales step by step from large to small, and generate the first alignment feature step by step according to the mixed attention feature and the inter-frame optical flow feature of the corresponding scale; When there is a scale alignment feature of the smallest scale, the current alignment strategy is to align the scales step by step from small to large, and generate second alignment features step by step according to the mixed attention features and the inter-frame optical flow features of the corresponding scales, and output the second alignment features of the largest scale as the video alignment features of the video sequence.

6. The method according to claim 5, characterized in that The mixed attention feature includes: a central mixed feature of a central frame and a mixed feature of neighboring frames of other frames; the first alignment feature is generated step by step according to the mixed attention feature of the corresponding scale and the inter-frame optical flow feature, including: determining whether the first target scale is a maximum scale; When the first target scale is not the maximum scale, obtaining a first alignment feature of a scale above the first target scale; determining a first central mixed feature of the first target scale according to the first alignment feature; Determine a first alignment feature of the first target scale according to the first central mixed feature and each adjacent frame mixed feature corresponding to the first target scale and an inter-frame optical flow feature; When the first target scale is the maximum scale, a first alignment feature of the first target scale is determined according to a central mixed feature of a corresponding scale, a mixed feature of each adjacent frame, and an inter-frame optical flow feature.

7. The method according to claim 6, characterized in that The step of generating the second alignment feature according to the mixed attention feature and the inter-frame optical flow feature of the corresponding scale comprises: Obtain a second alignment feature at a next scale of the second target scale; Acquire a first central mixed feature of a first alignment feature corresponding to the second target scale; Determine a second central mixed feature of the second target scale according to the second alignment feature and the first central mixed feature; A second alignment feature of the second target scale is determined according to the second central mixed feature, each adjacent frame mixed feature corresponding to the second target scale, and an inter-frame optical flow feature.

8. A super-resolution video reconstruction device, characterized in that: include: A first acquisition module is used to acquire a video sequence set generated according to a target video, wherein the video sequence set includes a plurality of video trainings, and each video sequence is composed of a plurality of continuous video frames; A first execution module is used to extract hybrid attention features and inter-frame optical flow features at multiple scales for each video frame from each of the video sequences; A first determination module is used to determine the video alignment feature of each video sequence according to the mixed attention feature and the inter-frame optical flow feature of each video frame at multiple scales; The second execution module is used to reconstruct the video alignment features of each of the video sequences into a super-resolution video of the target video.

9. A computer electronic production equipment, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images