Video super-resolution reconstruction method and device, storage medium and electronic equipment

Abstract

Embodiments of the present application provide a video super-resolution reconstruction method and device, a storage medium and an electronic device. In the embodiments of the present application, a key frame requiring super-resolution reconstruction is obtained from a video frame sequence, and a neighboring frame of the key frame is obtained; feature extraction is performed on the key frame and the neighboring frame to obtain key frame features and neighboring frame features; the neighboring frame features are filtered based on the key frame features using an attention mechanism to obtain a filtering result of the neighboring frame features; convolution operation is performed on the key frame features and the filtering result to obtain a convolution operation result; and super-resolution reconstruction is performed on the convolution operation result and the key frame features to obtain a reconstructed frame. In the present application, the neighboring frame features are filtered using the attention mechanism to filter out redundant feature information in the neighboring frame with respect to the key frame, thereby avoiding interference of the redundant feature information on the reconstruction of the key frame and effectively improving the processing effect of the super-resolution reconstruction.

Description

Technical Field

[0001] This application relates to the field of video processing technology, specifically to a video super-resolution reconstruction method, apparatus, storage medium, and electronic device. Background Technology

[0002] Super-resolution reconstruction is a technique that reconstructs a high-resolution image from a low-resolution image. Super-resolution reconstruction is widely used in various fields, such as video processing.

[0003] However, current video super-resolution reconstruction technologies applied in the field of video processing generally suffer from poor processing results. Summary of the Invention

[0004] This application provides a video super-resolution reconstruction method, apparatus, storage medium, and electronic device, which can effectively improve the processing effect of video super-resolution reconstruction.

[0005] In a first aspect, embodiments of this application provide a video super-resolution reconstruction method, including:

[0006] Obtain the key frames that need to be reconstructed in super-resolution from the video frame sequence, and obtain the adjacent frames of the key frames;

[0007] Feature extraction processing is performed on the key frame and the adjacent frame to obtain key frame features and adjacent frame features;

[0008] Based on the keyframe features, an attention mechanism is used to filter the features of adjacent frames to obtain the filtering results of the features of adjacent frames.

[0009] The keyframe features and the filtering results are subjected to convolution operations to obtain the convolution operation result;

[0010] The convolution operation result and the keyframe features are subjected to super-resolution reconstruction processing to obtain the reconstructed frame.

[0011] Secondly, embodiments of this application provide a video super-resolution reconstruction apparatus, comprising:

[0012] The acquisition module is used to acquire key frames that need to be reconstructed by super-resolution from the video frame sequence, and to acquire adjacent frames of the key frames.

[0013] The feature module is used to perform feature extraction processing on the key frame and the adjacent frame to obtain key frame features and adjacent frame features;

[0014] A filtering module is used to filter the features of adjacent frames based on the keyframe features using an attention mechanism, so as to obtain the filtering result of the features of adjacent frames.

[0015] The convolution operation module is used to perform convolution operation processing on the keyframe features and the filtering results to obtain the convolution operation result;

[0016] The reconstruction module is used to perform super-resolution reconstruction processing on the convolution operation results and the keyframe features to obtain reconstructed frames.

[0017] Thirdly, the storage medium provided in the embodiments of this application stores a computer program thereon, which, when run on a computer, causes the computer to execute the video super-resolution reconstruction method provided in any embodiment of this application.

[0018] Fourthly, the electronic device provided in the embodiments of this application includes a processor and a memory, the memory having a computer program, and the processor executing the video super-resolution reconstruction method provided in any embodiment of this application by calling the computer program.

[0019] This application embodiment obtains keyframes that need to be reconstructed for super-resolution from a video frame sequence, as well as adjacent frames of the keyframes; performs feature extraction processing on the keyframes and adjacent frames to obtain keyframe features and adjacent frame features; uses an attention mechanism to filter the adjacent frame features based on the keyframe features to obtain the filtered adjacent frame features; performs convolution operation processing on the keyframe features and the filtered results to obtain the convolution operation result; and performs super-resolution reconstruction processing on the convolution operation result and the keyframe features to obtain the reconstructed frame.

[0020] In this application, an attention mechanism is used to filter the features of adjacent frames, thereby filtering out redundant feature information in adjacent frames for key frames, avoiding interference of redundant feature information with key frame reconstruction, and effectively improving the processing effect of super-resolution reconstruction. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the first process of the video super-resolution reconstruction method provided in the embodiments of this application.

[0023] Figure 2This is a schematic diagram of the second process of the video super-resolution reconstruction method provided in the embodiments of this application.

[0024] Figure 3 This is a schematic diagram of the first structure of the video super-resolution reconstruction network provided in the embodiments of this application.

[0025] Figure 4 This is a schematic diagram of the second structure of the video super-resolution reconstruction network provided in the embodiments of this application.

[0026] Figure 5 This is a schematic diagram of the first partial structure of the attention module provided in the embodiments of this application.

[0027] Figure 6 This is a schematic diagram of a second partial structure of the attention module provided in an embodiment of this application.

[0028] Figure 7 This is a schematic diagram of the first overall structure of the attention module provided in the embodiments of this application.

[0029] Figure 8 This is a schematic diagram of the second overall structure of the attention module provided in the embodiments of this application.

[0030] Figure 9 This is a schematic diagram of a convolution operation processing module provided in an embodiment of this application.

[0031] Figure 10 This is a schematic diagram of a convolution calculation unit provided in an embodiment of this application.

[0032] Figure 11 This is a schematic diagram of the first structure of the convolutional subunit provided in the embodiments of this application.

[0033] Figure 12 This is a schematic diagram of the second structure of the convolutional subunit provided in the embodiments of this application.

[0034] Figure 13 This is a schematic diagram of a reconstruction module provided in an embodiment of this application.

[0035] Figure 14 This is a schematic diagram of the video super-resolution reconstruction apparatus provided in the embodiments of this application.

[0036] Figure 15 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0037] Figure 16 This is another structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0038] Please refer to the diagrams, where the same component symbols represent the same components. The principles of this application are illustrated by way of example implementation in a suitable computing environment. The following description is based on the specific embodiments of this application exemplified, and should not be construed as limiting other specific embodiments not detailed herein.

[0039] This application provides a video super-resolution reconstruction method. The execution subject of this video super-resolution reconstruction method can be the video super-resolution reconstruction device provided in this application, or an electronic device integrating the video super-resolution reconstruction device. The video super-resolution reconstruction device can be implemented in hardware or software. The electronic device can be a smartphone, tablet computer, PDA (Personal Digital Assistant), etc. A detailed analysis follows.

[0040] For example, in a scenario where an electronic device is used, the device can acquire a sequence of video frames that need to be processed. It then identifies the keyframes from the video frame sequence that require super-resolution reconstruction and acquires adjacent frames. Feature extraction is then performed on both the keyframes and adjacent frames to obtain their respective keyframe features and adjacent frame features. Based on the keyframe features, an attention mechanism is used to filter the adjacent frame features, resulting in a filtered version. A convolution operation is then performed on the keyframe features and the filtered version to obtain the convolution result. Finally, super-resolution reconstruction is performed on the convolution result and the keyframe features to obtain a reconstructed frame. The resolution of the reconstructed frame is greater than that of the keyframe. A detailed analysis follows.

[0041] This application provides a video super-resolution reconstruction method, such as... Figure 1 As shown, Figure 1 This is a schematic diagram of a first flowchart of a video super-resolution reconstruction method provided in an embodiment of this application. The video super-resolution reconstruction method may include:

[0042] 101. Obtain the key frames that need to be reconstructed in super-resolution from the video frame sequence, and obtain the adjacent frames of the key frames.

[0043] There are several ways to obtain video frame sequences. For example, you can acquire the real-time video stream during a live broadcast and extract the necessary video frame sequence. Another example is when a user is watching a video online, the video frame sequence can be obtained by pre-buffering the content. Yet another example is when a user is watching a local video, the video frame sequence can be obtained by parsing the local video file. The specific acquisition method can be determined based on actual needs.

[0044] In this context, adjacent frames refer to video frames that are adjacent to the keyframe. For example, if the keyframe is a video frame at time T, then adjacent frames could refer to the video frames at times T-1 and T+1. Alternatively, the time span can be further; when the keyframe is a video frame at time T, adjacent frames could refer to the video frames at times T-2, T-1, T+1, and T+2.

[0045] It is understood that the technical solutions provided in this application are not limited to a three-frame scheme where the keyframe is the current frame and adjacent frames are the previous and next frames. More adjacent frames can also be provided to obtain more information from the previous and next frames, thereby improving the accuracy of the reconstructed frames. For example, a multi-frame scheme can be used where the keyframe is the current frame and adjacent frames are multiple frames before and after. The specific configuration can be tailored to actual needs.

[0046] In some embodiments, the input keyframes and adjacent frames can be transformed first, and then the transformed keyframes and adjacent frames can be processed by feature extraction and other processes.

[0047] For example, when the keyframe is the current video frame at time T, and the adjacent frames are video frames at times T-1 and T+1, color space conversion processing can be performed on all three video frames at times T, T-1, and T+1 to convert them to the YUV color space. During the conversion to the YUV color space, the Y channel of each of the three video frames is processed.

[0048] For example, the current video frame at time T can be set as a keyframe, and the video frames at times T-3, T-2, T-1, T+1, T+2, and T+3 can be set as adjacent frames. The acquired keyframe and adjacent frames can be processed to convert the color space to YUV color space, and then the Y channel of the seven video frames can be processed further.

[0049] It is understood that the technical solution of this application is not limited to the three-frame scheme of adjacent frames being the previous and next frames and the keyframe being the current frame; it can also be achieved by inputting more preceding and following frames. By using an attention mechanism to extract features from more frame images and utilizing more frame information, the accuracy of the reconstructed frames can be improved.

[0050] 102. Perform feature extraction processing on keyframes and adjacent frames to obtain keyframe features and adjacent frame features.

[0051] There are several methods for feature extraction processing of keyframes and adjacent frames. For example, when the keyframe is the video frame at time T and the adjacent frames are the video frames at times T-1 and T+1, the Y channels of the three video frames can be input into a convolutional block, and the frame features corresponding to the three video frames, namely the keyframe features and the adjacent frame features, can be extracted through the convolutional block.

[0052] Convolutional blocks can have various structural representations. For example, a convolutional block can include multiple convolutional layers containing different convolutional kernels.

[0053] For example, by using multiple convolutional layers with different kernels to simultaneously extract features from keyframes and adjacent frames, keyframe features and adjacent frame features can be obtained.

[0054] In feature extraction using convolutional layers, the number of convolutional layers can be adjusted according to actual computational needs. For example, when high computational accuracy is required, a larger number of convolutional layers can be used for feature extraction; when low computational accuracy is required, a smaller number of convolutional layers can be used.

[0055] 103. Based on keyframe features, use an attention mechanism to filter the features of adjacent frames to obtain the filtering results of adjacent frame features.

[0056] After extracting the keyframe features and adjacent frame features, since there is redundant feature information in the adjacent frame features that is not related to the keyframe, it is necessary to use an attention mechanism to determine which features in the adjacent frame features are needed in the reconstruction process. Specifically, using the attention mechanism to filter adjacent frames involves using the attention mechanism to extract feature masks of adjacent frame features based on the keyframe features and adjacent frame features, and then using the masks to filter the adjacent frame features.

[0057] In some embodiments, the method of "filtering adjacent frame features based on keyframe features using an attention mechanism to obtain the filtering result of adjacent frame features" includes:

[0058] An attention mechanism is used to perform mask extraction on keyframe features and features of adjacent frames to obtain feature masks of adjacent frames.

[0059] The features of adjacent frames are processed based on feature masks to obtain the filtering results of adjacent frame features.

[0060] In some embodiments, the method of "using an attention mechanism to perform mask extraction processing on keyframe features and features of adjacent frames to obtain feature masks of adjacent frames" includes:

[0061] The keyframe features and adjacent frame features are concatenated to obtain the feature concatenation result.

[0062] The concatenated feature result is then subjected to convolution to obtain the feature convolution result;

[0063] The feature convolution results are extracted to obtain feature masks for adjacent frames.

[0064] There are several ways to concatenate keyframe features and features from adjacent frames. For example, the `concat` function can be used to concatenate keyframe features and features from adjacent frames.

[0065] There are several ways to perform convolution processing on the feature concatenation results. For example, the feature concatenation results can be sequentially input into multiple convolutional layers, and the feature convolution result is obtained after multi-layer convolution calculation. Alternatively, the feature concatenation results can be input into a single convolutional layer, and the feature convolution result is obtained after single-layer convolution calculation. The number of convolutional layers can be changed according to the actual computational needs. For example, when higher computational accuracy is required, multiple convolutional layers can be used; when lower computational accuracy is required, a single convolutional layer can be used.

[0066] There are several ways to extract and process the feature convolution results to obtain a feature mask. For example, an activation function can be used to extract and process the feature convolution results to obtain a feature mask. There are various activation functions, such as the ReLU function and the Sigmoid function, and the specific type of activation function can be selected according to actual needs.

[0067] For example, when the keyframe feature is the video frame feature at time T, and the adjacent frame features are the video frame features at time T-1, the feature mask of the video frame feature at time T-1 is obtained. The video frame features at time T and time T-1 are concatenated using a concatenation function to obtain the feature concatenation result. When the feature concatenation result is processed through multiple convolutional layers to obtain the feature convolution result, this result is input into the Sigmoid function to output the feature mask of the video frame feature at time T-1. It can be understood that when the keyframe feature is the video frame feature at time T, and the adjacent frame features are video frame features from multiple preceding and following times, a similar method can be used to obtain the feature masks of multiple adjacent frame features.

[0068] In this embodiment, an attention mechanism is actually used to obtain a feature mask. The mask obtained through the attention mechanism is used to select features from consecutive video frames, reducing the impact of artifacts caused by object motion on the reconstructed frames. Furthermore, the attention mechanism can achieve implicit alignment between consecutive video frames, which is relatively easy to implement.

[0069] Compared to commonly used video super-resolution reconstruction schemes based on optical flow networks, which require global information during reconstruction and thus place a large demand on device-side caching, the technical solution in this application does not require global information. When calculating a pixel in a video frame, only feature information from a small receptive field is needed, resulting in less cache requirements on the device. Furthermore, by eliminating the need for global information, the computational load in the video super-resolution reconstruction process is reduced, improving reconstruction efficiency.

[0070] There are several ways to process features of adjacent frames based on feature masks. For example, the filtering result of adjacent frame features can be obtained by multiplying the feature mask and the features of adjacent frames.

[0071] For example, when the feature mask of the video frame features at time T-1 is obtained, the video frame features at time T-1 and the feature mask are multiplied to filter out the unnecessary feature information in the video frame features at time T-1, and the filtering result is obtained.

[0072] 104. Perform convolution operations on the keyframe features and filtering results to obtain the convolution operation results.

[0073] In some embodiments, the method of "convolutional processing of keyframe features and filtering results" may include:

[0074] The keyframe features and filtering results are then concatenated to obtain the final concatenated result.

[0075] The splicing result is downsampled to obtain the downsampled result.

[0076] The downsampling results and the latent features of adjacent frames are convolved to obtain multi-frame convolution results;

[0077] Upsampling is performed on the convolution results of multiple frames to obtain the convolution operation result.

[0078] When the filtering results are obtained, the keyframe features and the filtering results need to be concatenated. There are several ways to concatenate them. For example, a concatenation function can be used to concatenate the keyframe features and the filtering results to achieve merging. Alternatively, multiplication or addition operations can be performed on the keyframe features and the filtering results to achieve merging.

[0079] When the stitched result is obtained, it can be downsampled to achieve reconstruction at a lower resolution. There are several ways to downsample the stitched result. For example, a downsampling function can be used. For instance, space2depth (a downsampling method without precision loss) can be used to downsample the stitched result.

[0080] By downsampling the stitched results, reconstruction can be achieved at a low resolution, reducing the overall computational load of the reconstruction process and improving the efficiency of super-resolution reconstruction. Furthermore, the reconstruction method provided in this application performs calculations at a smaller image scale, resulting in lower computational load and lower requirements for device power consumption and computing power, making it more suitable for deployment on existing devices.

[0081] In some embodiments, when multiple frames of convolution results are obtained, upsampling can be performed on these results to enlarge them to the video frame size of the keyframe. There are several ways to perform upsampling. For example, pixel rearrangement (depth2space) can be used to enlarge the multi-frame convolution results, after multiple calculations, to the original video frame size.

[0082] Since space2depth and depth2space are sampling methods without precision loss, using these two methods for sampling processing can effectively improve the accuracy of video frame super-resolution reconstruction and avoid the impact of sampling loss on reconstruction accuracy.

[0083] In the process of convolution processing the downsampling results, latent features output by adjacent frames during the reconstruction process can be added, and the latent features and the downsampling results can be convolved simultaneously.

[0084] In some embodiments, before performing convolution operations on the keyframe features and filtering results, the method further includes:

[0085] During the super-resolution reconstruction of adjacent frames, latent features are extracted from the features of adjacent frames to obtain the latent features of adjacent frames.

[0086] When performing super-resolution reconstruction on keyframes, adjacent frames can also be reconstructed. For example, when performing super-resolution reconstruction on adjacent frames, if the features of the adjacent frames are those of the video frames at time T-1, the video frame features at time T-2 and time T can be obtained. By performing mask extraction on the video frame features at the three times and processing the features based on the masks, the filtered results of the video frame features at time T-2 and time T can be obtained. The filtered results of the video frame features at the two times are concatenated with the video frame features at time T-1, and the concatenated result is downsampled to obtain the downsampled result at time T-1. The downsampled result is then convolved, and the latent features of the video frame at time T-2 can be added during the convolution process to output the latent features of the video frame at time T-1.

[0087] Since the acquisition of the latent features of the video frame at time T-1 is not only related to the features of the video frame at time T-2, the video frame at time T-1, and the video frame at time T, but also incorporates the latent features of the video frame at time T-2, the latent features of the video frame at time T-1 can convey the video frame information at time T-1, as well as the information of the previous few video frames.

[0088] Incorporating latent features into the reconstruction process allows for the transfer of information from the previous video frame and several preceding frames. This information transfer enables the current frame to utilize information not only from the preceding and following frames but also from several frames prior. Obtaining more inter-frame information helps prevent inter-frame perturbations. Furthermore, the transfer of latent features effectively contributes to inter-frame stabilization, ensuring the reconstructed frame maintains continuity with the preceding and following frames.

[0089] In some embodiments, the method of "convolving the downsampling result and the latent features of adjacent frames to obtain multi-frame convolution results" may include:

[0090] The downsampling result is convolved to obtain the downsampling convolution result;

[0091] The downsampling convolution result and the latent features are convolved to obtain the latent feature convolution result;

[0092] The convolution results of the latent features are processed by convolution to obtain multi-frame convolution results.

[0093] For example, the downsampled results can be convolved, then latent features can be added, and then convolved again to obtain multi-frame convolution results.

[0094] For example, the downsampling result and latent features can be convolved simultaneously to obtain multi-frame convolution results. In some embodiments, the method of "convolving the downsampling result and the latent features of adjacent frames to obtain multi-frame convolution results" may include:

[0095] The downsampling result and the latent features are convolved to obtain the latent feature convolution result;

[0096] The convolution results of the latent features are processed by convolution to obtain multi-frame convolution results.

[0097] In some embodiments, the method of "convolving the downsampling result to obtain the downsampling convolution result" may include:

[0098] The downsampling results are convolved to obtain the downsampling calculation results;

[0099] The downsampling calculation results are activated to obtain the downsampling activation results.

[0100] By fusing the downsampling results and the downsampling activation results, the downsampling convolution result is obtained.

[0101] There are several ways to fuse the downsampling results and the downsampling activation results. For example, the downsampling results and the downsampling activation results can be added or multiplied to achieve fusion of the two results.

[0102] For example, during convolution processing, the downsampling result can be passed sequentially through a convolutional layer and an activation function to obtain the downsampling activation result. Adding the downsampling result and the downsampling activation result yields the downsampling convolution result.

[0103] It is understandable that when performing convolution processing on the downsampling convolution result and latent features, and when performing convolution processing on the latent feature convolution result, the processing method can be referred to as "performing convolution processing on the downsampling result to obtain the downsampling convolution result".

[0104] For example, when performing convolution processing on the downsampled convolution result and the latent features, the downsampled convolution result and the latent features are sequentially passed through a convolutional layer and an activation function for convolution and activation processing to obtain the activation result. Then, the activation result, the downsampled convolution result, and the latent features are fused to obtain the latent feature convolution result.

[0105] For example, when performing convolution processing on the downsampled convolution result and latent features, the downsampled convolution result and latent features are sequentially passed through a convolutional layer and an activation function for convolution and activation processing to obtain the activation result. Then, the activation result, the downsampled convolution result, and the latent features are fused together, and the fused result is then sequentially passed through a convolutional layer and an activation function for convolution and activation processing. This process is repeated multiple times to obtain the latent feature convolution result.

[0106] It is understood that in the video super-resolution reconstruction method provided in this application embodiment, when performing convolution processing, the number of convolution blocks, convolution layers, or any modules that can perform convolution processing can be adjusted according to actual computing needs. For example, when the computing accuracy requirement is high, more convolution layers can be used for convolution processing; when the computing accuracy requirement is low, fewer convolution layers can be used for convolution processing.

[0107] 105. Perform super-resolution reconstruction on the convolution operation results and keyframe features to obtain the reconstructed frame.

[0108] In some embodiments, the method of "performing super-resolution reconstruction of the convolution operation result and keyframe features to obtain a reconstructed frame" may include:

[0109] The results of convolution operations and keyframe features are fused to obtain the frame fusion result;

[0110] The frame fusion result is processed by convolution to obtain the convolution result to be reconstructed;

[0111] The reconstructed convolution result is upsampled to obtain the reconstructed frame.

[0112] After multiple convolution calculations and upsampling using pixel rearrangement (depth2space), the convolution result is obtained. This result is then fused with the features of the keyframe, and multiple convolution operations are performed again to obtain a feature that needs to be amplified (i.e., the convolution result to be reconstructed). Upsampling this reconstructed convolution result yields the reconstructed frame.

[0113] For example, when converting to the YUV color space for reconstruction, the Y channel of the keyframe is used. After a series of processing steps, what is actually obtained is the Y channel of the reconstructed frame. However, the Y channel alone is not enough to obtain the reconstructed frame. At this point, the U and V channels of the keyframe also need to be obtained and bilinearly magnified. The bilinearly magnified U and V channels are then merged with the Y channel of the reconstructed frame, and the merged image is converted back to the original color space to obtain the complete reconstructed frame image.

[0114] This application embodiment obtains keyframes for super-resolution reconstruction from a video frame sequence, as well as adjacent frames of the keyframes; performs feature extraction processing on the keyframes and adjacent frames to obtain keyframe features and adjacent frame features; filters adjacent frame features using an attention mechanism based on the keyframe features to obtain filtered adjacent frame features; performs convolution operations on the keyframe features and the filtered results to obtain convolution operation results; and performs super-resolution reconstruction processing on the convolution operation results and keyframe features to obtain reconstructed frames, wherein the resolution of the reconstructed frames is greater than the resolution of the keyframes.

[0115] This application utilizes an attention mechanism to filter features from adjacent frames, removing redundant feature information from adjacent frames for keyframes. This avoids interference from redundant feature information in keyframe reconstruction and effectively improves the processing performance of super-resolution reconstruction.

[0116] This application also provides a video super-resolution upscaling method, such as... Figure 2 As shown, Figure 2 This is a second flowchart illustrating the video super-resolution upscaling method provided in this application embodiment. This method can be implemented on a neural network, for example, on a pre-trained video super-resolution reconstruction network. Figure 3 As shown, Figure 3 This is a schematic diagram of a first structure of a video super-resolution reconstruction network provided in an embodiment of this application. The method may include:

[0117] 201. Obtain the key frames that need to be reconstructed in super-resolution from the video frame sequence, and obtain the adjacent frames of the key frames.

[0118] There are several ways to obtain video frame sequences. For example, you can acquire the real-time video stream during a live broadcast and extract the necessary video frame sequence. Another example is when a user is watching a video online, the video frame sequence can be obtained by pre-buffering the content. Yet another example is when a user is watching a local video, the video frame sequence can be obtained by parsing the local video file. The specific acquisition method can be determined based on actual needs.

[0119] In this context, adjacent frames refer to video frames that are adjacent to the keyframe. For example, if the keyframe is a video frame at time T, then adjacent frames could refer to the video frames at times T-1 and T+1. Alternatively, the time span can be further; when the keyframe is a video frame at time T, adjacent frames could refer to the video frames at times T-2, T-1, T+1, and T+2.

[0120] In some embodiments, the input keyframes and adjacent frames can be transformed first, and then the transformed keyframes and adjacent frames can be processed by feature extraction and other processes.

[0121] For example, when the keyframe is the current video frame at time T, and the adjacent frames are video frames at times T-1 and T+1, color space conversion processing can be performed on the acquired video frames at times T, T-1, and T+1 to convert the color space to YUV color space. After conversion to YUV color space, the Y channel of the three video frames is then processed further.

[0122] For example, the current video frame at time T can be set as a keyframe, and the video frames at times T-3, T-2, T-1, T+1, T+2, and T+3 can be set as adjacent frames. The acquired keyframe and adjacent frames can be processed to convert the color space to YUV color space, and then the Y channel of the seven video frames can be processed further.

[0123] It is understood that the technical solution of this application is not limited to a three-frame scheme where adjacent frames are the previous and next frames, and the keyframe is the current frame. More preceding and following frames can also be input. By using an attention mechanism to extract features from more frame images and utilizing more frame information, the accuracy of the reconstructed frames can be improved. For example, a five-frame scheme can be used, where the previous two and next two frames are adjacent frames and the current frame is the keyframe. Alternatively, it can be extended to a multi-frame scheme where multiple preceding and following frames are adjacent frames and the current frame is the keyframe. The specific configuration can be tailored to actual needs.

[0124] 202. Obtain a pre-trained video super-resolution reconstruction network, which includes a feature extraction module, an attention module, a convolution operation processing module, and a reconstruction module.

[0125] Before acquiring the video frames to be reconstructed, the video super-resolution reconstruction network can be pre-trained. Before pre-training, it is necessary to acquire the input and supervision images for training the network model, using the input and supervision images as sample data sets.

[0126] In some embodiments, before "obtaining the keyframes for super-resolution reconstruction from the video frame sequence", the method may further include:

[0127] Acquire multiple first video frame sequences, each first video frame sequence including multiple consecutive video frames;

[0128] Each first video frame sequence is downsampled to obtain a corresponding second video frame sequence, wherein the resolution of each video frame in the second video frame sequence is less than the resolution of each video frame in the first video frame sequence.

[0129] Each first video frame sequence and its corresponding second video frame sequence are set as a video frame sequence sample group.

[0130] The video super-resolution reconstruction network is trained based on a set of video frame sequence samples.

[0131] For example, a sequence of real high-resolution video frames can be obtained and used as the supervision image. By downsampling the video frame sequence to reduce the resolution of each video frame, a low-resolution video frame sequence can be obtained and used as the input image. The high-resolution and low-resolution video frame sequences are then set as the sample group, and the video super-resolution reconstruction network is trained using the sample group.

[0132] For another example, an existing high-resolution video frame sequence can be obtained, blurred to varying degrees, and then downsampled to obtain a low-resolution video frame sequence. Noise can then be added to this low-resolution video frame sequence to generate the input image for training the network model, while the original high-resolution video frame sequence serves as the supervision image.

[0133] In some embodiments, the method of "training a video super-resolution reconstruction network based on a set of video frame sequence samples" may include:

[0134] The second video frame sequence is reconstructed using a video super-resolution reconstruction network to obtain the reconstructed video frame sequence corresponding to the second video frame sequence.

[0135] Based on the reconstructed video frame sequence and the first video frame sequence, the reconstruction loss of the video super-resolution reconstruction network is obtained;

[0136] The video super-resolution reconstruction network is trained based on the reconstruction loss.

[0137] For example, a low-resolution video frame sequence can be input into a video super-resolution reconstruction network for reconstruction, resulting in a reconstructed video frame sequence. Based on the high-resolution video frame sequence, the reconstructed video frame sequence, and the loss function, the loss of the reconstruction network can be obtained. The reconstruction network is then trained iteratively using this loss function. When the network parameters converge, a pre-trained video super-resolution reconstruction network is obtained.

[0138] The reconstruction loss can take various forms. For example, it can be a regression loss function, such as the L1 Loss function, which is the Mean Square Error (MSE) function. Another example is the L2 Loss function, which is the Mean Absolute Error (MAE) function. Yet another example is using a neural network as the reconstruction loss, such as VGGLoss, where VGG is a neural network model. Furthermore, multiple loss functions can be combined to obtain a hybrid loss function, and the result of this hybrid loss function can be used as the reconstruction loss. For instance, in some embodiments, the reconstruction loss can take the following forms:

[0139] Loss=α×L1(HR,GT)+β×L2(HR,GT)+γ×VGG(HR,GT)

[0140] Where α, β, and γ are weighting coefficients, which can be set according to the actual training requirements. HR is the reconstructed video frame sequence, and GT is the high-resolution video frame sequence.

[0141] It is understood that the number of feature extraction modules, attention modules, convolution operation processing modules, and reconstruction modules in the video super-resolution reconstruction network provided in this application embodiment can be set according to actual needs. For example, when it is necessary to obtain more information from previous and subsequent frames to improve the accuracy of the reconstructed frame images, the number of attention modules can be increased to utilize the attention mechanism to process more video frame images and obtain more image information. The specific settings can be configured according to actual needs.

[0142] 203. The feature extraction module performs feature extraction processing on the key frame and adjacent frames to obtain the key frame features and adjacent frame features.

[0143] like Figure 4 As shown, Figure 4 This is a schematic diagram of a second structure of the video super-resolution reconstruction network provided in an embodiment of this application. For example, when the keyframe is a video frame at time T, the adjacent frames can be video frames at time T-1 and time T+1. Figure 4 As shown, by inputting the video frames at time T, T-1, and T+1 into each feature extraction module for feature extraction, the features of the video frames at time T, T-1, and T+1 can be obtained.

[0144] For example, the current video frame at time T can be set as a key frame, and the video frames at times T-3, T-2, T-1, T+1, T+2, and T+3 can be set as adjacent frames. The obtained key frames and adjacent frames are then input into seven feature extraction modules for feature extraction, which can yield the features of the video frames at each time.

[0145] The feature extraction module can take various forms. For example, it can consist of multiple convolutional layers with different kernels. Alternatively, it can consist of a single convolutional layer. Understandably, the number of convolutional layers in the feature extraction module can be varied to meet different computational needs. For instance, when the device's computing power is strong, the feature extraction module can contain multiple convolutional layers; when the device's computing power is weak, it can contain a single convolutional layer. The specific configuration depends on the actual requirements.

[0146] 204. The keyframe features and adjacent frame features are extracted using the attention module to obtain the feature masks of adjacent frames.

[0147] In some embodiments, the attention module may include convolutional units and activation units, such as Figure 5 As shown, Figure 5 This is a schematic diagram of a first partial structure of the attention module provided in an embodiment of this application.

[0148] The method of "extracting feature masks from keyframe features and adjacent frame features using an attention module" can include:

[0149] The keyframe features and adjacent frame features are concatenated to obtain the feature concatenation result.

[0150] The convolutional unit is used to perform convolution processing on the feature concatenation result to obtain the feature convolution result;

[0151] The feature convolution results are extracted by activation units to obtain feature masks for adjacent frames.

[0152] For example, such as Figure 5 As shown, the feature masks of adjacent frames can be obtained by passing the keyframe features and adjacent frame features through convolutional units and activation units.

[0153] There are several ways to concatenate keyframe features and features from adjacent frames. For example, the `concat` function can be used to concatenate keyframe features and features from adjacent frames.

[0154] There are various ways to combine convolutional units and activation units. For example, an attention module can consist of one convolutional unit and one activation unit. Alternatively, an attention module can include multiple convolutional units and multiple activation units, arranged in a convolutional unit-activation unit configuration.

[0155] There are various forms of convolutional units. For example, a convolutional unit can include multiple convolutional layers containing different convolutional kernels.

[0156] There are several ways to represent an activation unit. For example, the activation function in an activation unit can be the ReLU function. Or, the activation function can be the Sigmoid function, etc. The specific type of activation function can be set according to actual needs.

[0157] This means that the number of convolutional and activation units in the attention module can be set according to actual computational needs to meet the requirements of different devices. For example, when the device has strong computing power, the number of convolutional units in the attention module can be increased to improve the accuracy of the reconstructed frames.

[0158] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram of a second partial structure of the attention module provided in an embodiment of this application. The attention module may include multiple convolutional units and activation units. When the features of the video frame at time T and the features of the video frame at time T-1 are obtained, the features of the video frames at time T and T-1 are sequentially input into the convolutional units and activation units in the attention module to obtain the feature mask of the video frame at time T-1.

[0159] 205. By fusing the feature mask and features from adjacent frames through the attention module, the filtering results of features from adjacent frames are obtained.

[0160] When processing features of adjacent frames based on feature masks, the feature mask and features of adjacent frames can actually be fused by an attention module to obtain the filtering result.

[0161] There are several ways to fuse feature masks and features from adjacent frames. For example, feature masks and features from adjacent frames can be added or subtracted. Another example is multiplying feature masks and features from adjacent frames.

[0162] For example, Figure 7 This is a schematic diagram of a first overall structure of the attention module provided in an embodiment of this application. Figure 7 As shown, after the keyframe features and adjacent frame features are passed through convolutional units and activation units in sequence to obtain the feature masks of adjacent frames, the adjacent frame features and feature masks are fused to obtain the filtering results of adjacent frame features.

[0163] For example, Figure 8 This is a schematic diagram of a second overall structure of the attention module provided in an embodiment of this application. For example... Figure 8 As shown, by sequentially inputting the features of the video frame at time T and the video frame at time T-1 into a combination of multiple convolutional units and activation units, a feature mask for the video frame at time T-1 can be obtained. When the feature mask for the video frame at time T-1 is obtained, the features of the video frame at time T-1 are multiplied by the feature mask to obtain the filtered feature result for the video frame at time T-1. Similarly, the same operation is performed on the features of the video frame at time T+1 to obtain the filtered feature result for the video frame at time T+1.

[0164] 206. The convolution operation processing module performs convolution operations on the keyframe features and filtering results to obtain the convolution operation results.

[0165] In some embodiments, the convolution operation processing module may include a downsampling unit, a convolution calculation unit, and a first upsampling unit. For example... Figure 9 As shown, Figure 9 This is a schematic diagram of a convolution operation processing module provided in an embodiment of this application.

[0166] The method of "performing convolution operations on keyframe features and filtering results using a convolution operation processing module to obtain the convolution operation result" can include:

[0167] The keyframe features and filtering results are then concatenated to obtain the final concatenated result.

[0168] The splicing result is downsampled by a downsampling unit to obtain the downsampled result;

[0169] The downsampling results and the latent features of adjacent frames are convolved by the convolutional computation unit to obtain multi-frame convolution results;

[0170] The first upsampling unit upsamples the convolution results of multiple frames to obtain the convolution operation result.

[0171] When the filtering results are obtained, it is necessary to merge the keyframe features and the filtering results. There are several ways to merge them. For example, a concatenation function can be used to concatenate the keyframe features and the filtering results to achieve merging. Another example is to multiply or add the keyframe features and the filtering results to achieve merging.

[0172] When the stitched result is obtained, it can be downsampled to achieve reconstruction at a lower resolution. There are several ways to downsample the stitched result. For example, it can be done using downsampling units. These downsampling units can utilize space2depth (a lossless downsampling method) to downsample the stitched result. By downsampling the stitched result, reconstruction can be achieved at a lower resolution, reducing the overall computational load of the reconstruction process and improving the efficiency of super-resolution reconstruction.

[0173] When multiple frames of convolution results are obtained, they can be upsampled to enlarge the results to the size of the keyframe video frame. There are several ways to perform upsampling. For example, a first upsampling unit can be used to upsample the multiple frames of convolution results. This first upsampling unit can enlarge the multi-frame convolution results, after multiple calculations, to the original video frame size using pixel rearrangement (depth2space).

[0174] Since space2depth and depth2space are sampling methods without precision loss, using these two methods for sampling processing can effectively improve the accuracy of video frame super-resolution reconstruction and avoid the impact of sampling loss on reconstruction accuracy.

[0175] In the process of convolution processing the downsampling results, latent features output by adjacent frames during the reconstruction process can be added, and the latent features and the downsampling results can be convolved simultaneously.

[0176] In some embodiments, before "performing convolution operations on keyframe features and filtering results through the convolution operation processing module", the following may be included:

[0177] During the super-resolution reconstruction of adjacent frames, latent features are extracted from the features of adjacent frames to obtain the latent features of adjacent frames.

[0178] For example, when performing super-resolution reconstruction on adjacent frames, if the features of the adjacent frames are those of the video frames at time T-1, then the features of the video frames at time T-2 and time T can be obtained. By performing mask extraction processing on the video frame features at time T-2 and time T-1, and then fusing the extracted mask with the video frame features at time T-2, the filtered result of the video frame features at time T-2 can be obtained. Similarly, by performing the same processing on the video frame features at time T-1 and time T, the filtered result of the video frame features at time T can also be obtained.

[0179] The filtered video frame features from two time points are concatenated with the video frame features from time point T-1. The concatenated result is then downsampled to obtain the downsampled result at time point T-1. The downsampled result is then convolved, and latent features from the video frame at time point T-2 can be incorporated during this process to output the latent features of the video frame at time point T-1.

[0180] By incorporating latent features into the reconstruction process, information from the previous video frame and the preceding frames can be transmitted, allowing the video frame to be reconstructed to obtain more inter-frame information, thus avoiding inter-frame disturbances. Furthermore, through the transmission of information from latent features, the reconstructed frames can maintain consistency.

[0181] In some embodiments, a convolution computation unit includes N convolution sub-units, where N is a positive integer greater than zero, such as... Figure 10 As shown, Figure 10 This is a schematic diagram of a convolution calculation unit provided in an embodiment of this application.

[0182] The phrase "convolution processing the downsampling result and the latent features of adjacent frames using convolutional computation units to obtain multi-frame convolution results" can include:

[0183] The downsampling result is obtained by convolving the first M convolutional subunits, where M is an integer and N > M ≥ 0;

[0184] The downsampling convolution result and the latent feature are convolved by the (M+1)th convolution subunit to obtain the latent feature convolution result.

[0185] The latent feature convolution results are processed by NM-1 convolutional sub-units to obtain multi-frame convolution results.

[0186] For example, the downsampled result can be obtained by convolving the first M convolutional subunits. After adding latent features, the latent features and the downsampled convolution result are convolved again by the (M+1)th convolutional subunit to obtain the latent feature convolution result. The latent feature convolution result is then convolved by the remaining NM-1 convolutional subunits to obtain the multi-frame convolution result.

[0187] In some embodiments, the downsampling result and latent features can be convolved simultaneously using N convolutional subunits to obtain multi-frame convolutional results. Therefore, the method of "convolving the downsampling result and latent features of adjacent frames using convolutional computation units to obtain multi-frame convolutional results" can include:

[0188] The downsampling result and the latent feature are convolved by the first convolutional subunit to obtain the latent feature convolution result.

[0189] The latent feature convolution results are processed by convolutional processing using N-1 convolutional sub-units to obtain multi-frame convolution results.

[0190] In some embodiments, each convolutional subunit includes a convolutional layer and an activation layer, such as Figure 11 As shown, Figure 11 This is a schematic diagram of the first structure of the convolutional subunit provided in the embodiments of this application.

[0191] The phrase "by performing convolution processing on the downsampling result using the first M convolutional subunits to obtain the downsampling convolution result" can include:

[0192] The downsampling result is obtained by performing convolution processing on the convolutional layer of the first convolutional subunit;

[0193] The downsampling calculation result is activated by the activation layer of the first convolutional subunit to obtain the downsampling activation result;

[0194] The downsampling result and the downsampling activation result are fused to obtain the downsampling fusion result;

[0195] This process continues, with the downsampling fusion result input into the next convolutional subunit for processing, until the Mth convolutional subunit outputs the downsampling convolution result.

[0196] Convolutional layers can take many forms. For example, convolutional layers can be implemented by using multiple sub-convolutional layers containing different convolutional kernels.

[0197] The activation layer can take many forms. For example, activation layers can be activated using different activation functions.

[0198] Taking the convolution process of the first convolutional subunit as an example, such as Figure 12 As shown, Figure 12 This is a schematic diagram of a second structure of a convolutional subunit provided in an embodiment of this application. The downsampling result is processed sequentially through a convolutional layer, an activation layer, and another convolutional layer to obtain a downsampling activation result. The downsampling activation result and the downsampling result are then added together to obtain a downsampling fusion result, thus achieving the fusion of the downsampling result and the downsampling activation result. This processing is repeated in each convolutional subunit until the last (i.e., the Mth) convolutional subunit outputs the downsampling convolution result.

[0199] 207. The reconstruction module performs super-resolution reconstruction processing on the convolution operation results and keyframe features to obtain reconstructed frames, where the resolution of the reconstructed frames is greater than that of the keyframes.

[0200] In some embodiments, the reconstruction module may include a convolution calculation unit and a second upsampling unit. For example... Figure 13 As shown, Figure 13 This is a schematic diagram of a reconstruction module provided in an embodiment of this application.

[0201] The method of "obtaining reconstructed frames by performing super-resolution reconstruction processing on the convolution operation results and keyframe features through the reconstruction module" can include:

[0202] The results of convolution operations and keyframe features are fused to obtain the frame fusion result;

[0203] The frame fusion result is processed by convolutional computation unit to obtain the convolutional result to be reconstructed;

[0204] The reconstructed frame is obtained by upsampling the convolution result to be reconstructed through the second upsampling unit.

[0205] For example, the reconstructed structure and keyframe features can be added together to obtain a frame fusion result, thus achieving fusion. Then, the frame fusion result is convolved using a convolutional computation unit to obtain the convolutional result to be reconstructed. This convolutional result is a feature that needs to be magnified by a factor of several squared channels. Finally, the convolutional result is upsampled using pixel rearrangement to obtain the reconstructed frame.

[0206] The convolutional computation unit can take many forms. For example, a convolutional computation unit can include multiple convolutional neural network (CNN) blocks. When performing convolution processing through the convolutional computation unit, the multiple CNN blocks contained within it can be convolved sequentially.

[0207] In some embodiments, when the Y channel of the keyframe is used in the reconstruction process, what is actually obtained in the end is the Y channel of the reconstructed frame. Therefore, after obtaining the Y channel of the reconstructed frame, it is necessary to merge the U channel and V channel of the keyframe and the Y channel of the reconstructed frame to obtain the reconstructed frame.

[0208] After "upsampling the convolution result to be reconstructed through the second upsampling unit", the method may further include:

[0209] When the Y channel of the reconstructed frame is obtained, the U and Y channels of the key frame are magnified.

[0210] The U and Y channels of the magnified keyframe are merged with the Y channel of the reconstructed frame to obtain the reconstructed frame.

[0211] There are several ways to perform magnification. For example, it can be done through bilinear magnification. Another example is through pixel rearrangement.

[0212] It is understood that in the video super-resolution reconstruction method provided in this application embodiment, when performing convolution processing, the number of convolution blocks, convolution layers, or any modules that can perform convolution processing can be adjusted according to actual computing needs. For example, when the computing accuracy requirement is high, more convolution layers can be used for convolution processing; when the computing accuracy requirement is low, fewer convolution layers can be used for convolution processing.

[0213] In this embodiment, keyframes requiring super-resolution reconstruction are obtained from a video frame sequence, along with adjacent frames. A pre-trained video super-resolution reconstruction network is acquired, comprising a feature extraction module, an attention module, a convolution operation processing module, and a reconstruction module. The feature extraction module performs feature extraction processing on the keyframes and adjacent frames to obtain keyframe features and adjacent frame features. The attention module performs mask extraction processing on the keyframe features and adjacent frame features to obtain feature masks for adjacent frames. The attention module fuses the feature masks and adjacent frame features to obtain filtered results for adjacent frame features. The convolution operation processing module performs reconstruction processing on the keyframe features and the filtered results to obtain convolution operation results. The reconstruction module performs super-resolution reconstruction processing on the convolution operation results and keyframe features to obtain reconstructed frames, wherein the resolution of the reconstructed frames is greater than the resolution of the keyframes.

[0214] Because this application uses an attention mechanism to obtain a feature mask, and filters out redundant feature information in adjacent frames for keyframes, it avoids interference from redundant feature information on keyframe reconstruction, effectively improving the processing effect of super-resolution reconstruction.

[0215] To facilitate better implementation of the video super-resolution reconstruction method provided in this application, this application also provides an apparatus based on the above-described video super-resolution reconstruction method. The meanings of the terms used are the same as in the video super-resolution reconstruction method described above, and specific implementation details can be found in the descriptions within the method embodiments.

[0216] Please see Figure 14 , Figure 14 This is a schematic diagram of the video super-resolution reconstruction apparatus 300 provided in an embodiment of this application. Specifically, the video super-resolution reconstruction apparatus 300 includes: an acquisition module 301, a feature module 302, a filtering module 303, a convolution operation module 304, and a reconstruction module 305.

[0217] The acquisition module 301 is used to acquire key frames that need to be super-resolution reconstructed from the video frame sequence, and to acquire adjacent frames of the key frames.

[0218] Feature module 302 is used to perform feature extraction processing on the key frame and the adjacent frame to obtain key frame features and adjacent frame features;

[0219] The filtering module 303 is used to filter the features of adjacent frames based on the key frame features using an attention mechanism to obtain the filtering results of the features of adjacent frames.

[0220] The convolution operation module 304 is used to perform convolution operation on keyframe features and filtering results to obtain the convolution operation result.

[0221] The reconstruction module 305 is used to perform super-resolution reconstruction processing on the convolution operation result and the keyframe features to obtain a reconstructed frame, wherein the resolution of the reconstructed frame is greater than the resolution of the keyframe.

[0222] In some implementations, the filtering module 303 is used for:

[0223] An attention mechanism is used to perform mask extraction on keyframe features and features of adjacent frames to obtain feature masks of adjacent frames.

[0224] The features of adjacent frames are processed based on feature masks to obtain the filtering results of adjacent frame features.

[0225] In some implementations, the filtering module 303 is used for:

[0226] The keyframe features and adjacent frame features are concatenated to obtain the feature concatenation result.

[0227] The concatenated feature result is then subjected to convolution to obtain the feature convolution result;

[0228] The feature convolution results are extracted to obtain feature masks for adjacent frames.

[0229] In some implementations, the convolution operation module 304 is used for:

[0230] During the super-resolution reconstruction of adjacent frames, latent features are extracted from the features of adjacent frames to obtain the latent features of adjacent frames.

[0231] In some implementations, the convolution operation module 304 is used for:

[0232] The keyframe features and filtering results are then concatenated to obtain the final concatenated result.

[0233] The splicing result is downsampled to obtain the downsampled result.

[0234] The downsampling results and the latent features of adjacent frames are convolved to obtain multi-frame convolution results;

[0235] Upsampling is performed on the convolution results of multiple frames to obtain the convolution operation result.

[0236] In some implementations, the convolution operation module 304 is used for:

[0237] The downsampling result is convolved to obtain the downsampling convolution result;

[0238] The downsampling convolution result and the latent features are convolved to obtain the latent feature convolution result;

[0239] The convolution results of the latent features are processed by convolution to obtain multi-frame convolution results.

[0240] In some implementations, the convolution operation module 304 is used for:

[0241] The downsampling results are convolved to obtain the downsampling calculation results;

[0242] The downsampling calculation results are activated to obtain the downsampling activation results.

[0243] By fusing the downsampling results and the downsampling activation results, the downsampling convolution result is obtained.

[0244] In some implementations, the reconstruction module 305 is used for:

[0245] The convolution operation result and the keyframe features are fused to obtain the frame fusion result;

[0246] The frame fusion result is subjected to convolution processing to obtain the convolution result to be reconstructed;

[0247] The convolution result to be reconstructed is upsampled to obtain the reconstructed frame.

[0248] The video super-resolution reconstruction apparatus provided in this application embodiment obtains key frames for super-resolution reconstruction from a video frame sequence through an acquisition module 301, and obtains adjacent frames of the key frames; performs feature extraction processing on the key frames and adjacent frames through a feature module 302 to obtain key frame features and adjacent frame features; performs filtering processing on adjacent frame features based on key frame features using an attention mechanism through a filtering module 303 to obtain filtering results of adjacent frame features; performs convolution operation processing on key frame features and filtering results through a convolution operation module 304 to obtain convolution operation results; and performs super-resolution reconstruction processing on the convolution operation results and key frame features through a reconstruction module 305 to obtain reconstructed frames, wherein the resolution of the reconstructed frames is greater than the resolution of the key frames.

[0249] In this embodiment, by using an attention mechanism to filter features of adjacent frames, redundant feature information in adjacent frames is filtered out for key frames, thus avoiding interference from redundant feature information on key frame reconstruction and effectively improving the processing effect of super-resolution reconstruction.

[0250] This application also provides an electronic device 400. Please refer to... Figure 15 The electronic device 400 includes a processor 401 and a memory. The processor 401 is electrically connected to the memory.

[0251] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device through various interfaces and lines. By running or loading computer programs stored in the memory 402 and calling data stored in the memory 402, it performs various functions of the electronic device 400 and processes data, thereby monitoring the electronic device 400 as a whole.

[0252] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, computer programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0253] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to realize various functions, as follows:

[0254] Extract the key frames that need to be reconstructed from the video frame sequence, as well as the adjacent frames of the key frames;

[0255] Feature extraction is performed on keyframes and adjacent frames to obtain keyframe features and adjacent frame features.

[0256] Based on keyframe features, an attention mechanism is used to filter features of adjacent frames to obtain the filtered results of adjacent frame features.

[0257] The keyframe features and filtering results are processed by convolution to obtain the convolution result;

[0258] The convolution operation result and keyframe features are subjected to super-resolution reconstruction processing to obtain a reconstructed frame, wherein the resolution of the reconstructed frame is greater than the resolution of the keyframe.

[0259] In some embodiments, the processor 401 may specifically perform the following steps:

[0260] An attention mechanism is used to perform mask extraction on keyframe features and features of adjacent frames to obtain feature masks of adjacent frames.

[0261] The features of adjacent frames are processed based on feature masks to obtain the filtering results of adjacent frame features.

[0262] In some embodiments, the processor 401 may specifically perform the following steps:

[0263] The keyframe features and adjacent frame features are concatenated to obtain the feature concatenation result.

[0264] The concatenated feature result is then subjected to convolution to obtain the feature convolution result;

[0265] The feature convolution results are extracted to obtain feature masks for adjacent frames.

[0266] In some embodiments, the processor 401 may specifically perform the following steps:

[0267] During the super-resolution reconstruction of the adjacent frames, latent feature extraction is performed on the features of the adjacent frames to obtain the latent features of the adjacent frames.

[0268] In some embodiments, the processor 401 may specifically perform the following steps:

[0269] The keyframe features and filtering results are then concatenated to obtain the final concatenated result.

[0270] The splicing result is downsampled to obtain the downsampled result.

[0271] The downsampling results and the latent features of adjacent frames are convolved to obtain multi-frame convolution results;

[0272] Upsampling is performed on the convolution results of multiple frames to obtain the convolution operation result.

[0273] In some embodiments, the processor 401 may specifically perform the following steps:

[0274] The downsampling result is convolved to obtain the downsampling convolution result;

[0275] The downsampling convolution result and the latent features are convolved to obtain the latent feature convolution result;

[0276] The convolution results of the latent features are processed by convolution to obtain multi-frame convolution results.

[0277] In some embodiments, the processor 401 may specifically perform the following steps:

[0278] The downsampling results are convolved to obtain the downsampling calculation results;

[0279] The downsampling calculation results are activated to obtain the downsampling activation results.

[0280] By fusing the downsampling results and the downsampling activation results, the downsampling convolution result is obtained.

[0281] In some embodiments, the processor 401 may specifically perform the following steps:

[0282] The convolution operation result and the keyframe features are fused to obtain the frame fusion result;

[0283] The frame fusion result is subjected to convolution processing to obtain the convolution result to be reconstructed;

[0284] The convolution result to be reconstructed is upsampled to obtain the reconstructed frame.

[0285] As can be seen from the above, the electronic device in this embodiment of the application uses an attention mechanism to filter the features of adjacent frames, filtering out redundant feature information in adjacent frames for key frames, avoiding interference of redundant feature information on key frame reconstruction, and effectively improving the processing effect of super-resolution reconstruction.

[0286] Please refer to the following: Figure 16 In some embodiments, the electronic device 400 may further include a display 403, a radio frequency circuit 404, an audio circuit 405, and a power supply 406. The display 403, radio frequency circuit 404, audio circuit 405, and power supply 406 are electrically connected to the processor 401.

[0287] The display 403 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. The display 403 may include a display panel, which in some embodiments may be configured as a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0288] The radio frequency circuit 404 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0289] The audio circuit 405 can be used to provide an audio interface between a user and an electronic device via a speaker or microphone.

[0290] The power supply 406 can be used to power various components of the electronic device 400. In some embodiments, the power supply 406 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0291] Although not shown in the diagram, the electronic device 400 may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0292] The term "module" as used herein can be considered as a software object executing on the computing system. The different components, modules, engines, and services described herein can be considered as implementation objects on the computing system. The apparatus and method described herein are preferably implemented in software, but can also be implemented in hardware, both of which are within the scope of this application.

[0293] This application also provides a storage medium storing a computer program that, when run on a computer, causes the computer to perform the video super-resolution reconstruction method described in any of the above embodiments, such as:

[0294] Extract the key frames that need to be reconstructed from the video frame sequence, as well as the adjacent frames of the key frames;

[0295] Feature extraction is performed on keyframes and adjacent frames to obtain keyframe features and adjacent frame features.

[0296] Based on keyframe features, an attention mechanism is used to filter features of adjacent frames to obtain the filtered results of adjacent frame features.

[0297] The keyframe features and filtering results are processed by convolution to obtain the convolution result;

[0298] The convolution operation result and keyframe features are subjected to super-resolution reconstruction processing to obtain a reconstructed frame, wherein the resolution of the reconstructed frame is greater than the resolution of the keyframe.

[0299] In the embodiments of this application, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0300] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0301] It should be noted that, regarding the video super-resolution reconstruction method of this application embodiment, those skilled in the art will understand that all or part of the process of implementing the video super-resolution reconstruction method of this application embodiment can be accomplished by a computer program controlling related hardware. This computer program can be stored in a storage medium, such as the memory of an electronic device, and executed by at least one processor within the electronic device. During execution, it can include the process of the video super-resolution reconstruction method embodiment. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, etc.

[0302] For the video super-resolution reconstruction apparatus of this application embodiment, its functional modules can be integrated into a single processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a storage medium, such as a read-only memory, a disk, or an optical disk.

[0303] The above provides a detailed description of a video super-resolution reconstruction method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video super-resolution reconstruction method, characterized in that, include: Obtain the key frames that need to be reconstructed in super-resolution from the video frame sequence, and obtain the adjacent frames of the key frames; Color space conversion processing is performed on the keyframe and the adjacent frames to convert the color space to the YUV color space; The keyframe and the adjacent frames are subjected to feature extraction processing to obtain keyframe features and adjacent frame features, including: inputting the Y channel of the transformed keyframe and the adjacent frames into a convolutional block, and extracting keyframe features and adjacent frame features through the convolutional block; Based on the keyframe features, an attention mechanism is used to filter the adjacent frame features to remove redundant feature information in the adjacent frame features that is not related to the keyframe, and to obtain the filtering result of the adjacent frame features. This includes: when the feature mask of the adjacent frame features is obtained, multiplying the adjacent frame features and the feature mask to remove unnecessary feature information in the adjacent frame features, and to obtain the filtering result. The convolution operation is performed on the keyframe features and the filtering results to obtain the convolution operation result, including: extracting latent features from the features of adjacent frames to obtain the latent features of the adjacent frames; concatenating the keyframe features and the filtering results to obtain the concatenated result; downsampling the concatenated result to obtain the downsampled result; convolving the downsampled result and the latent features of the adjacent frames to obtain the multi-frame convolution result; and enlarging the multi-frame convolution result to the keyframe size by pixel rearrangement to obtain the convolution operation result. The super-resolution reconstruction process is performed on the convolution operation result and the key frame features to obtain the reconstructed frame, including: obtaining the U channel and V channel of the key frame and performing bilinear magnification on them, merging the bilinearly magnified U channel and V channel with the Y channel of the reconstructed frame, and converting the merged image back to the original color space to obtain the complete frame image of the reconstructed frame. The process of extracting latent features from adjacent frame features to obtain latent features of adjacent frames includes: when performing super-resolution reconstruction on adjacent frames, if the adjacent frame features are video frame features at time T-1, obtaining video frame features at time T-2 and time T; performing mask extraction on the video frame features at the three times and processing the features based on the mask to obtain the filtered results of the video frame features at time T-2 and time T; concatenating the filtered results of the video frame features at the two times and the video frame features at time T-1, and performing downsampling on the concatenated result to obtain the downsampling result at time T-1; performing convolution on the downsampling result, adding the latent features of the video frame at time T-2 during the convolution process, and outputting the latent features of the video frame at time T-1.

2. The video super-resolution reconstruction method according to claim 1, characterized in that, The step of using an attention mechanism to filter the features of adjacent frames based on the keyframe features to obtain the filtering results of the adjacent frame features includes: An attention mechanism is used to perform mask extraction processing on the keyframe features and the features of adjacent frames to obtain the feature mask of the adjacent frames; The adjacent frame features are processed based on the feature mask to obtain the filtering result of the adjacent frame features.

3. The video super-resolution reconstruction method according to claim 2, characterized in that, The process of using an attention mechanism to extract features from the keyframe features and the features of adjacent frames to obtain the feature masks of the adjacent frames includes: The keyframe features and the adjacent frame features are concatenated to obtain the feature concatenation result. The concatenated feature result is then subjected to convolution processing to obtain the feature convolution result; The feature convolution result is extracted to obtain the feature mask of the adjacent frame.

4. The video super-resolution reconstruction method according to claim 1, characterized in that, The step of convolving the downsampling result and the latent features of the adjacent frames to obtain multi-frame convolution results includes: The downsampling result is then subjected to convolution processing to obtain the downsampling convolution result; The downsampling convolution result and the latent feature are convolved to obtain the latent feature convolution result; The convolution results of the hidden features are subjected to convolution processing to obtain multi-frame convolution results.

5. The video super-resolution reconstruction method according to claim 4, characterized in that, The downsampling result is convolved to obtain the downsampling convolution result, which includes: The downsampling results are convolved to obtain the downsampling calculation results; The downsampling calculation result is activated to obtain the downsampling activation result; The downsampling result and the downsampling activation result are fused to obtain the downsampling convolution result.

6. The video super-resolution reconstruction method according to claim 1, characterized in that, The step of performing super-resolution reconstruction processing on the convolution operation result and the keyframe features to obtain the reconstructed frame includes: The convolution operation result and the keyframe features are fused to obtain the frame fusion result; The frame fusion result is subjected to convolution processing to obtain the convolution result to be reconstructed; The convolution result to be reconstructed is upsampled to obtain the reconstructed frame.

7. A video super-resolution reconstruction device, characterized in that, include: The acquisition module is used to acquire key frames that need to be reconstructed by super-resolution from the video frame sequence, and to acquire adjacent frames of the key frames. The transformation module is used to perform color space transformation processing on the key frame and the adjacent frame, converting the color space to the YUV color space. The feature module is used to perform feature extraction processing on the key frame and the adjacent frame to obtain key frame features and adjacent frame features, including: inputting the Y channel of the transformed key frame and the adjacent frame into a convolutional block, and extracting key frame features and adjacent frame features through the convolutional block; The filtering module is used to filter the adjacent frame features based on the key frame features using an attention mechanism, filtering out redundant feature information in the adjacent frame features that is not related to the key frame, and obtaining the filtering result of the adjacent frame features. The filtering module includes: when the feature mask of the adjacent frame features is obtained, multiplying the adjacent frame features and the feature mask to filter out unnecessary feature information in the adjacent frame features and obtain the filtering result. The convolution operation module is used to perform convolution operations on the keyframe features and the filtering results to obtain convolution operation results, including: performing latent feature extraction processing on the features of adjacent frames to obtain the latent features of the adjacent frames; performing concatenation processing on the keyframe features and the filtering results to obtain a concatenated result; performing downsampling processing on the concatenated result to obtain a downsampled result; performing convolution processing on the downsampled result and the latent features of the adjacent frames to obtain a multi-frame convolution result; and enlarging the multi-frame convolution result to the keyframe size by pixel rearrangement to obtain the convolution operation result; wherein, performing latent feature extraction processing on the features of adjacent frames to obtain the latent features of the adjacent frames includes: performing latent feature extraction processing on the features of adjacent frames to obtain the latent features of the adjacent frames. During super-resolution reconstruction, when the features of adjacent frames are the video frame features at time T-1, the video frame features at time T-2 and time T are obtained. Mask extraction is performed on the video frame features at the three times, and the features are processed based on the masks to obtain the filtered results of the video frame features at time T-2 and time T. The filtered results of the video frame features at the two times and the video frame features at time T-1 are concatenated, and the concatenated result is downsampled to obtain the downsampled result at time T-1. The downsampled result is then convolved, and the latent features of the video frame at time T-2 are added during the convolution process to output the latent features of the video frame at time T-1. The reconstruction module is used to perform super-resolution reconstruction processing on the convolution operation result and the keyframe features to obtain a reconstructed frame. The module includes: acquiring the U and V channels of the keyframe and performing bilinear magnification on them; merging the bilinearly magnified U and V channels with the Y channel of the reconstructed frame; and converting the merged image back to the original color space to obtain a complete frame image of the reconstructed frame.

8. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor loads the computer program to execute the video super-resolution reconstruction method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, It stores a computer program that, when run on a computer, causes the computer to perform the video super-resolution reconstruction method as described in any one of claims 1 to 6.

Images