Video super-resolution method and device, equipment and storage medium

Abstract

The invention relates to the technical field of video processing, in particular to a video super-resolution method, device and equipment and a storage medium, and the method comprises the steps: carrying out the multi-scale coding of a to-be-processed video, and obtaining video features under a plurality of scales; fusing the video frame features corresponding to the same moment in the video features under different scales to obtain fused features; obtaining a high-resolution video frame based on the fusion feature and the up-sampling frame feature corresponding to the fusion feature at the same moment; and obtaining a target video based on the high-resolution video frames corresponding to the moments. According to the invention, the video super-resolution effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video super-resolution method, device, equipment and storage medium. Background Art

[0002] In the development of digital video technology, video super-resolution (VSR) technology has always been an important research field. Video super-resolution technology aims to restore high-resolution (HR) video from low-resolution (LR) video and meet users' high standards for visual experience by enhancing video quality. This technology has a wide range of applications in many fields, including television broadcasting, medical imaging, security monitoring, and consumer electronics.

[0003] At present, the more common video super-resolution technologies include a video super-resolution method based on a convolutional neural network (CNN) and a video super-resolution method based on a generative adversarial network (GAN); wherein the video super-resolution method based on a convolutional neural network (CNN) is specifically as follows: the convolutional neural network (CNN) is trained through a large number of low-resolution to high-resolution video samples, and then the trained convolutional neural network (CNN) is used to predict the high-resolution video corresponding to the low-resolution video; wherein the video super-resolution method based on a generative adversarial network (GAN) is specifically as follows: the generator in the generative adversarial network (GAN) generates a corresponding high-resolution video based on the low-resolution video, and then a network loss is generated based on the difference between the generated high-resolution video and the real high-resolution video, and then the generative adversarial network (GAN) is trained based on the network loss, and then the high-resolution video corresponding to the low-resolution video is predicted through the trained generative adversarial network (GAN).

[0004] However, convolutional neural networks (CNNs) mainly focus on feature extraction of low-resolution videos, that is, they focus on enhancing the spatial details of a single video frame, ignoring the temporal relationship between video frames, resulting in the final generated high-resolution video frames may have problems such as frame jumps and discontinuous motion; high-resolution videos generated by generative adversarial networks (GANs) often introduce textures or artifacts that are not truly linear, resulting in poor authenticity of the generated high-resolution videos; and, convolutional neural networks (CNNs) and generative adversarial networks (GANs) have relatively low computational efficiency due to their large number of model parameters; In summary, the existing technologies do not perform well in video super-resolution. Summary of the invention

[0005] In order to improve the effect of video super-resolution, the present application provides a video super-resolution method, apparatus, device and storage medium.

[0006] In a first aspect, the present application provides a video super-resolution method, comprising:

[0007] Perform multi-scale encoding on the video to be processed to obtain video features at multiple scales;

[0008] Fusion of video frame features corresponding to the same moment in the video features at different scales to obtain fused features;

[0009] Obtain a high-resolution video frame based on the fused feature and the up-sampled frame feature corresponding to the same time as the fused feature;

[0010] A target video is obtained based on the high-resolution video frames corresponding to each of the moments.

[0011] In a second aspect, the present application provides a video super-resolution device, comprising:

[0012] A multi-scale encoding module is used to perform multi-scale encoding on the video to be processed to obtain video features at multiple scales;

[0013] A feature fusion module, used for fusing the video frame features corresponding to the same moment in the video features at different scales to obtain a fused feature;

[0014] A decoding and reconstruction module, used to obtain a high-resolution video frame based on the fused features and the up-sampled frame features corresponding to the same time as the fused features;

[0015] The enhancement optimization module is used to obtain the target video based on the high-resolution video frames corresponding to each of the moments.

[0016] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0018] The above-mentioned video super-resolution method, device, equipment and storage medium obtain video features at multiple scales by multi-scale encoding of the video to be processed; fuse the video frame features corresponding to the same moment in the video features at different scales to obtain the fused features; obtain high-resolution video frames based on the fused features and the up-sampled frame features corresponding to the same moment as the fused features; obtain the target video based on the high-resolution video frames corresponding to each of the moments. Through the above implementation, by fusing the video features at the same moment in the video features at different scales, it is convenient to capture the moving and changing objects in the video to be processed, thereby improving the coherence between video frames, and generating high-resolution video frames by fusing features and up-sampled frame features, it is helpful to refine the video content and reduce the risk of information loss in the video frames, so as to prevent textures or artifacts that are not truly linear in the video frames; in addition, the above implementation avoids the participation of many model parameters, thereby improving the calculation speed of generating the target video; in summary, the present application can facilitate the improvement of the effect of video super-resolution.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flow chart of a video super-resolution method provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of a video super-resolution device provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application;

[0024] Figure 4 This is a diagram of the internal structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0027] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0028] Embodiment 1

[0029] Figure 1 A video super-resolution method flow chart provided in Example 1 of the present application, refer to Figure 1 The method may be performed by a device for performing the method, and the device may be implemented by software and / or hardware. The method includes:

[0030] S110, performing multi-scale encoding on the video to be processed to obtain video features at multiple scales.

[0031] The video to be processed is a low-resolution video that needs to be super-resolutioned, and the video to be processed is recorded as LR_video. Multi-scale encoding means first processing the video to be processed LR_video into videos at different video resolution scales, and then encoding the video at each video resolution scale. By encoding the video, various features in the video can be obtained, including color, texture, spatial relationship, etc. Various features are helpful to understand and reconstruct the video content in the video. In this embodiment, the scale specifically refers to the scale of video resolution. The video feature is the result obtained after encoding the video at the corresponding resolution scale, and the video feature is recorded as Encoded_Scales[i], where i represents the i-th scale.

[0032] Specifically, the video LR_video to be processed is processed into videos at different video resolution scales, each video resolution scale corresponds to one such video, and then the videos at different video resolution scales are encoded respectively to obtain video features corresponding to each video resolution scale.

[0033] S120: Fusion the video frame features corresponding to the same moment in the video features at different scales to obtain fusion features.

[0034] The video is composed of multiple temporally continuous video frames, each video frame has a corresponding video frame feature, and the video frame features of multiple continuous video frames constitute the video feature; the video frame has a corresponding time t in the video, and the time t is used to characterize the time sequence of the corresponding video frame in the video; video frame features at different scales all have corresponding video frame features at the same time t, and the video frame features include color, texture, spatial relationship, etc.; fusion is to add the video frame features corresponding to the video frame features at the same time t at different scales; the fused feature is the sum of the video frame features corresponding to the video frame features at the same time t at different scales, and the fused feature is recorded as Fused_feature[t], where t represents the time t in the video.

[0035] Specifically, the video frame features corresponding to the video frame features at different scales at the same time t are obtained, and then the obtained video frame features are added to obtain the fused feature Fused_feature[t].

[0036] It should be noted that by fusing the video frame features corresponding to the same time t at different scales, it is convenient to improve the coherence between video frames in the video to be processed when performing video super-resolution, so as to facilitate the capture of objects that move and / or change in the video to be processed, and then facilitate the processing of time-varying content in the video to reduce frame skipping or motion blur. For example, in the video analysis of live sports events, dynamic inter-frame feature fusion can help improve the ability to capture fast-moving objects and complex movements. When athletes move quickly, a single frame at a single moment may not be able to accurately capture all dynamic information. By fusing the features of continuous video frames, the motion details of each moment can be effectively reconstructed, thereby providing viewers with a smoother and more detailed visual experience.

[0037] S130: Obtain a high-resolution video frame based on the fused feature and the up-sampled frame feature corresponding to the same moment as the fused feature.

[0038] Among them, a clearer upsampled video can be obtained by upsampling the processed video. Similarly, the upsampled video also consists of multiple temporally continuous video frames, each of which has a corresponding video frame feature, and the video frame feature is recorded as an upsampled frame feature Up_LR_video[t]. The video frame corresponding to each upsampled frame feature has a corresponding time t in the upsampled video; by processing the fused feature Fused_feature[t] corresponding to the same time t and the upsampled frame feature Up_LR_video[t], a high-resolution video frame HR_video[t] can be obtained.

[0039] Specifically, the fused feature Fused_feature[t] and the upsampled frame feature Up_LR_video[t] corresponding to the same time t are processed to obtain a high-resolution video frame HR_video[t] corresponding to the time t.

[0040] It should be noted that by processing the fused feature Fused_feature[t] and the upsampled frame feature Up_LR_video[t] at the same time t, the content in the video to be processed can be gradually enhanced, which helps to refine the content in the video to be processed, and also helps to reduce the risk of information loss in the video to be processed, thereby facilitating the prevention of the phenomenon of refinement blur and edge artifacts in the video. Exemplarily, in medical image processing, the strategy of generating high-resolution video frames can be used to improve the clarity of scanned images, thereby helping doctors to make better diagnoses. For example, in MRI or CT scans, the strategy of generating high-resolution video frames can help doctors observe clearer images of tissues and organs, which is of great significance for diagnosing subtle lesions or early diseases.

[0041] S140: Obtain a target video based on the high-resolution video frames corresponding to each of the moments.

[0042] The target video can be obtained by processing the high-resolution video frames corresponding to each moment, and the target video is also the video to be processed after the video super-resolution is completed.

[0043] Specifically, the target video is obtained by processing the high-resolution video frames corresponding to each moment.

[0044] It should be noted that, in this embodiment, video features at multiple scales are obtained by multi-scale encoding of the video to be processed; fusion features are obtained by fusing the video frame features corresponding to the same moment in the video features at different scales; high-resolution video frames are obtained based on the fusion features and the up-sampled frame features corresponding to the same moment as the fusion features; and the target video is obtained based on the high-resolution video frames corresponding to each of the moments. Through the above implementation, by fusing the video features at the same moment in the video features at different scales, it is convenient to capture the moving and changing objects in the video to be processed, thereby improving the coherence between the video frames. Moreover, by generating high-resolution video frames through fusion features and up-sampled frame features, it is helpful to refine the video content and reduce the risk of information loss in the video frames, so as to prevent textures or artifacts that are not truly linear in the video frames; in addition, the above implementation avoids the participation of many model parameters, thereby improving the calculation speed of generating the target video; in summary, the present application can facilitate the improvement of the video super-resolution effect.

[0045] Embodiment 2

[0046] A video super-resolution method is provided in the second embodiment of the present application. The method optimizes the "multi-scale encoding of the processed video to obtain video features at multiple scales" in the first embodiment. It should be noted that for the parts not described in detail in this embodiment, reference can be made to the descriptions of other embodiments. The method includes:

[0047] S211 . Perform multi-scale downsampling on the video to be processed to obtain downsampled videos at multiple scales.

[0048] Among them, multi-scale downsampling, that is, reducing the video resolution of the video to be processed to multiple specified video resolutions through a preset downsampling function, thereby obtaining a video corresponding to each scale one by one, and the video is recorded as a downsampled video, and the downsampling function is recorded as Downsample(), wherein the input of the downsampling function is the video to be processed LR_video, and the downsampling function is also provided with a downsampling factor downfactor=2^i, i represents the scale ordinal number, the larger the scale i, the larger the downsampling factor, and the lower the video resolution to be processed will be; and the obtained downsampled video is recorded as Downsample(LR_video, downfactor=2^i)).

[0049] Specifically, according to the scale ordinal i, the corresponding downsampling factor downfactor=2^i is set for the downsampling function Downsample(), and then the video to be processed LR_video is input into the downsampling function with the downsampling factor for processing, so as to obtain the downsampled video Downsample(LR_video, downfactor=2^i) at the corresponding scale. Through the above method, different downsampling factors downfactor=2^i are set for the downsampling function according to different scale ordinals i, and then the video to be processed is processed by the downsampling function, so as to obtain downsampled videos at multiple scales.

[0050] It should be noted that by generating downsampled videos at multiple scales, it is convenient to obtain details at different levels in the video to be processed, thereby facilitating the acquisition of global specific texture and detail information of the video to be processed.

[0051] S212: Process the downsampled video based on a preset diffusion model and model parameters in the diffusion model that correspond one-to-one to the multiple scales to obtain video features at multiple scales.

[0052] In this embodiment, the preset diffusion model is specifically a dynamic progressive diffusion network (DPDN). In other embodiments, the diffusion model can also adopt a recurrent neural network (RNN), especially a long short-term memory network (LSTM) or a gated recurrent unit (GRU) in the recurrent neural network (RNN) type, without specific limitation; a model parameter Scale_Model[i] corresponding to each scale is provided in the diffusion network, wherein i represents the scale order; the diffusion model is used to process the downsampled video at each scale, thereby simulating the gradual recovery process of the noise in the downsampled video from blur to clarity, and the diffusion model function corresponding to the diffusion model is recorded as Diffuse(), and the diffusion model processes the downsampled video at a specific scale (such as the i-th scale) to obtain the video feature Encoded_Scales[i] corresponding to the downsampled video at the specific scale, Encoded_Scales[i]=Diffuse(Scale_Model[i], Downsample(LR_video, downfactor=2^i)).

[0053] Specifically, for the i-th scale, the model parameter Scale_Model[i] corresponding to the i-th scale is selected in the diffusion model, and then the downsampled video Downsample(LR_video, downfactor=2^i) corresponding to the i-th scale is input into the diffusion model for processing. The diffusion model outputs the corresponding video feature Encoded_Scales[i], Encoded_Scales[i]=Diffuse(Scale_Model[i], Downsample(LR_video, downfactor=2^i)).

[0054] It should be noted that the diffusion model processes downsampled videos of different scales, that is, the process of encoding downsampled videos of different scales. By encoding downsampled videos of each scale, the corresponding scale of the sampled video can be optimized. For example, at a higher scale (i.e., at a lower resolution), the diffusion model mainly focuses on the global motion and macro layout of the downsampled video, while at a lower scale (i.e., at a higher resolution), it mainly focuses on the detailed processing of texture and detail information. In practical applications, such as online video conferencing, real-time sports events, and other scenarios, the diffusion model can dynamically adjust the encoding strategy according to different network conditions, give priority to ensuring the clarity of important visual content (such as faces, actions, etc.), and restore as many details as possible under an acceptable computing load, thereby providing a better viewing experience.

[0055] S220: Fusing video frame features corresponding to the same moment in the video features at different scales to obtain a fused feature.

[0056] S230: Obtain a high-resolution video frame based on the fused feature and an upsampled frame feature corresponding to the fused feature at the same time.

[0057] S240: Obtain a target video based on the high-resolution video frames corresponding to each of the moments.

[0058] Embodiment 3

[0059] A video super-resolution method is provided in Embodiment 3 of the present application. The method optimizes the “fusion of video frame features corresponding to the same moment in the video features at different scales to obtain fusion features” in Embodiment 1. It should be noted that for the parts not described in detail in this embodiment, reference may be made to the descriptions of other embodiments. The method includes:

[0060] S310 , performing multi-scale encoding on the video to be processed to obtain video features at multiple scales.

[0061] S321. Obtain video frame features corresponding to each moment in the video features and dynamic weights set for the video frame features.

[0062] The downsampled video is composed of video frames corresponding to a plurality of consecutive different moments, the video features include features of video frames at a plurality of consecutive different moments, and the features of the video frames are recorded as video frame features; the moment corresponding to each video frame is also the moment t corresponding to the video frame corresponding to the video frame feature in the downsampled video; the video frame features corresponding to the moment t include not only the video frame features corresponding to the moment t, but also the video frame features corresponding to the moment t-1 and the moment t+1, and the video frame features corresponding to the moment t are recorded as Encoded_Scales[i][t-1:t+1]; it should be noted that the importance of video frame features at different moments is different, and the importance is determined by the dynamics and importance of the content in the video frame corresponding to the video frame feature. In order to reflect the importance of different video frame features, it is necessary to dynamically set the weight of the video frame feature according to the dynamics and importance of the content in the video frame corresponding to the video frame feature, and the weight is recorded as the dynamic weight w[i,t], wherein i is the scale ordinal number and t is the moment t corresponding to the video frame feature.

[0063] Specifically, the video frame features corresponding to each moment in the video features are obtained, and the dynamic weights dynamically set for the video frame features at different moments are also obtained; illustratively, taking moment t as an example, the video frame features w[i, t]Encoded_Scales[i][t-1:t+1] corresponding to moment t are obtained, and further the dynamic weight w[i, t] of the video frame corresponding to moment t is obtained.

[0064] S322: Obtain fusion features based on the video frame features corresponding to the same moment in the video features at different scales and the dynamic weights corresponding to the video frame features.

[0065] Among them, the fused feature is the weighted sum of the video frame feature Encoded_Scale[i][t-1:t+1] corresponding to the time t in the video features at different scales and the dynamic weight w[i, t], and the fused feature corresponding to the time t is recorded as Fused_feature[t]; the fused feature is also the integration result of the video frame feature Encoded_Scales[i][t-1:t+11] in the video features at different scales at the time t; the fused feature is convenient for subsequent use in generating high-resolution video of the video to be processed.

[0066] Specifically, the weighted sum of the video frame feature Encoded_Scales[i][t-1:t+1] corresponding to time t in the video features at different scales and the dynamic weight w[i, t] is calculated to obtain the fused feature fused feature.

[0067] S330: Obtain a high-resolution video frame based on the fused feature and the up-sampled frame feature corresponding to the same moment as the fused feature.

[0068] S340: Obtain a target video based on the high-resolution video frames corresponding to each of the moments.

[0069] Embodiment 4

[0070] A video super-resolution method is provided in a fourth embodiment of the present application. The method optimizes the "obtaining fusion features based on the video frame features corresponding to the same moment in the video features at different scales and the dynamic weights corresponding to the video frame features" in the third embodiment; it should be noted that for the parts not described in detail in this embodiment, reference may be made to the descriptions of other embodiments. The method includes:

[0071] S410 , performing multi-scale encoding on the video to be processed to obtain video features at multiple scales.

[0072] S421. Obtain video frame features corresponding to each moment in the video features and dynamic weights set for the video frame features.

[0073] S422A: Process the video frame features corresponding to the same moment in the video features at different scales based on a preset conversion function to obtain converted features.

[0074] Among them, the same moment in the video features under different scales corresponds to multiple video frame features, and there may be video frame features whose important features are not prominent among the multiple video frame features, which will lead to poor effect of the subsequently generated fusion features. In order to facilitate improving the effect of the subsequently generated fusion features, it is necessary to first convert the multiple video frame features corresponding to the same moment in the video features under different scales through a preset conversion function, and the conversion function is recorded as Transform(). The conversion function is used to highlight the important features in the video frame. The important features may be important textures, important area brightness, key content color, etc.; and the video frame features processed by the conversion function are recorded as converted features, and the converted features are recorded as Transform(Encoded_Scales[i][t-1:t+1]).

[0075] Specifically, multiple video frame features Encoded_Scales[i][t-1:t+1] corresponding to the same moment in the video features at different scales are respectively input into the preset transformation function Tansform() for processing, and the transformation function Transform() outputs the transformed features Transform(Encoded_Scales[i][t-1:t+1]) corresponding to each video frame feature one by one.

[0076] S422B, calculating the product of the converted feature and the corresponding dynamic weight to obtain a weighted feature.

[0077] Among them, the transformed feature corresponding to a scale at time t is Transform(Encoded_Scales[i][t-1:t+1]), the dynamic weight corresponding to the video frame feature of the transformed feature is generated, that is, the dynamic weight w[i, t] corresponding to the transformed feature, and the weighted feature is the product w[i, t]*Transform(Encoded_Scales[i][t-1:t+1]) of the transformed feature corresponding to a scale at time t and its corresponding dynamic weight w[i, t].

[0078] Specifically, taking the transformed feature corresponding to a scale at time t as an example, the product of the transformed feature Transform(Encoded_Scales[i][t-1:t+1]) and its corresponding dynamic weight w[i, t] is calculated to obtain the weighted feature w[i, t]*Transform(Encoded_Scales[i][t-1:t+1]); according to the above method, the weighted features corresponding to different scales at time t are calculated;

[0079] S422C: Obtain fusion features based on the weighted features corresponding to the different scales.

[0080] Among them, the fused feature at time t is the sum of the weighted features corresponding to different scales. The sum of the weighted features corresponding to different scales can be calculated by the preset sum function Sum(), and the fused feature Fused_feature[t]=Sum(w[i, t]*Transform(Encoded_Scales[i][t-1:t+1])).

[0081] Specifically, the sum of the weighted features corresponding to different scales at time t is calculated to obtain the fusion features corresponding to time t.

[0082] S430: Obtain a high-resolution video frame based on the fused feature and the up-sampled frame feature corresponding to the same moment as the fused feature.

[0083] S440: Obtain a target video based on the high-resolution video frames corresponding to each of the moments.

[0084] Embodiment 5

[0085] A video super-resolution method is provided in Embodiment 5 of the present application. The method optimizes the "obtaining a high-resolution video frame based on the fusion feature and the up-sampled frame feature corresponding to the same time as the fusion feature" in Embodiment 1. It should be noted that for the part not described in detail in this embodiment, reference may be made to the description of other embodiments. The method includes:

[0086] S510 , performing multi-scale encoding on the video to be processed to obtain video features at multiple scales.

[0087] S520: Fusion the video frame features corresponding to the same moment in the video features at different scales to obtain fusion features.

[0088] S531. Obtain up-sampled frame features based on the video to be processed.

[0089] Among them, in this embodiment, an upsampling function Upsample() is preset, and the upsampling function Upsample() is used to process the video to be processed, so as to obtain an upsampled video Up_LR_video at a corresponding scale, and the upsampled video Up_LR_video has a higher video resolution than the video to be processed LR_video; further, the upsampled video is processed by a preset diffusion model to obtain the features in the video frame corresponding to each time t, that is, the upsampled frame feature Up_LR_video[t].

[0090] Specifically, the video to be processed LR_video is input into the upsampling function Upsample() for processing, thereby obtaining the upsampled video Up_LR_video; further, the upsampled video Up_LR_video is input into the diffusion model for processing, thereby obtaining the upsampled frame feature Up_LR_video[t].

[0091] S532: Merge the upsampled frame features and the fusion features corresponding to the same moment to obtain a merged feature.

[0092] Among them, in this embodiment, a connection function Concatenate() is preset, which is used to merge the up-sampled feature Up_LR_video[t] and the fused feature Fused_feature[t] corresponding to the same time, and obtain the corresponding merged result, and the merged result is recorded as the merged feature Concatenate(Fused_feature[t], Up_LR_video[t]).

[0093] Specifically, the up-sampled feature Up_LR_video[t] and the fused feature Fused_feature[t] corresponding to the same time t are input into the preset connection function Concatenate() for processing, so as to obtain the corresponding merged feature Concatenate(Fused_feature[t], Up_LR_video[t]).

[0094] It should be noted that the merged features are used to provide richer video information at time t, thereby facilitating subsequent reconstruction of a high-resolution target video.

[0095] S533: Obtain a high-resolution video frame based on the combined features.

[0096] Among them, by decoding the merged features at time t, a high-resolution video frame corresponding to time t can be obtained, and the high-resolution video frame is recorded as HR_video[t].

[0097] Specifically, the merged features at time t are decoded to obtain a high-resolution video frame HR_video[t] corresponding to time t.

[0098] S540: Obtain a target video based on the high-resolution video frames corresponding to each of the moments.

[0099] Embodiment 6

[0100] A video super-resolution method is provided in Embodiment 6 of the present application. The method optimizes the "obtaining a high-resolution video frame based on the fusion feature and the up-sampled frame feature corresponding to the same time as the fusion feature" in Embodiment 5. It should be noted that for the part not described in detail in this embodiment, reference may be made to the description of other embodiments. The method includes:

[0101] S610 , performing multi-scale encoding on the video to be processed to obtain video features at multiple scales.

[0102] S620: Fusion the video frame features corresponding to the same moment in the video features at different scales to obtain fusion features.

[0103] S631. Obtain up-sampled frame features based on the video to be processed.

[0104] S632: Merge the upsampled frame features and the fusion features corresponding to the same moment to obtain a merged feature.

[0105] S633: Process the merged features based on a preset decoding function to obtain a high-resolution video frame.

[0106] Among them, in this embodiment, a decoding function Decode() is preset, and the decoding function Decode is used to reconstruct the merged features into a high-resolution video frame, that is, a high-resolution video frame, and the high-resolution video frame corresponding to time t is recorded as HR_video[t].

[0107] Specifically, the merged features corresponding to time t and time t are input into the decoding function Decode() for processing, thereby obtaining a high-resolution video frame HR_video[t], HR_video[t]=

[0108] Decode(Concatenate(Fused_feature[t], Up_LR_cideo[t]), t).

[0109] S640: Obtain a target video based on the high-resolution video frames corresponding to each of the moments.

[0110] Embodiment 7

[0111] A video super-resolution method is provided in Embodiment 7 of the present application. The method optimizes the "obtaining a target video based on the high-resolution video frames corresponding to each of the moments" in Embodiment 1. It should be noted that for the parts not described in detail in this embodiment, reference may be made to the descriptions of other embodiments. The method includes:

[0112] S710 , performing multi-scale encoding on the video to be processed to obtain video features at multiple scales.

[0113] S720: Fusing video frame features corresponding to the same moment in the video features at different scales to obtain a fused feature.

[0114] S730: Obtain a high-resolution video frame based on the fused feature and the up-sampled frame feature corresponding to the same moment as the fused feature.

[0115] S741. Adaptively enhance the high-resolution video frame based on a preset adaptive enhancement function and a feature weight array to obtain an enhanced video frame.

[0116] Among them, the high-resolution video frame contains multiple frame features, such as video content texture, video content brightness, video content color, etc. The adaptive enhancement function is used to enhance the frame features of the high-resolution video frame, so that the high-resolution video frame has a better visual effect, and the adaptive enhancement function is recorded as AdaptiveEnhance(); for the multiple frame features in the high-resolution video frame, a feature weight array is set, and the feature weight array includes feature weights corresponding to the multiple frame features one by one. The feature weight is used to reflect the importance and influence of the corresponding frame feature in the frame feature enhancement process, and the feature weight array is recorded as Feature_Weights. It should be noted that each feature weight in the feature weight array Feature_Weights is updated in real time as the content of the video to be processed changes, so that videos in different scenes can achieve better visual effects. By processing the high-resolution video frame HR_video[t], time t, and the feature weight array Feature_Weights corresponding to the high-resolution video frame HR_video[t] through the adaptive enhancement function AdaptiveEnhance(), the high-resolution video frame can be adaptively enhanced, and the corresponding enhanced video frame can be obtained, and the enhanced video frame is recorded as Enhanced_video[t].

[0117] Specifically, the high-resolution video frame HR_video[t], time t, and the feature weight array Feature_Weights corresponding to the high-resolution video frame HR_video[t] are input into the adaptive enhancement function AdaptiveEnhance() for processing, thereby obtaining the enhanced video frame Enhanced_video[t], Enhanced_video[t] = AdaptiveEnhance(HR_video[t], t, Feature_Weights).

[0118] It should be noted that in video clips with fast scene changes or intense movements, the feature weights may tend to enhance edge clarity and dynamic response speed; in more static or detail-rich scenes, the feature weights are adjusted to enhance the clarity of textures and details. By adaptively enhancing high-resolution video frames, the clarity and coherence of motion capture can be optimized, blur and jitter can be reduced, and the video viewing experience can be improved.

[0119] S742: Optimize the enhanced video frames corresponding to the different moments based on a preset optimization function, a spatial detail control variable, and a temporal coherence control variable to obtain a target video.

[0120] Among them, the corresponding enhanced video Enhanced_video can be obtained by combining the enhanced video frames Enhanced_video[t] corresponding to different moments in time sequence. Since the optimization function optimizes the enhanced video Enhanced_video, specifically the spatial details of the enhanced video Enhanced_video and the coherence between the video frames are optimized. In the process of optimizing the enhanced video Enhanced_video by the optimization function, the spatial detail control variable and the temporal coherence control variable are also used to control the hanging process, and the spatial detail control variable is recorded as alpha, and the temporal coherence control variable is recorded as beta; among them, the spatial detail control variable alpha is used to control the optimization degree of the spatial details, and the temporal coherence control variable beta is used to control the video frame by adjusting the spatial detail control variable alpha and the temporal coherence control variable beta, the video clarity can be ensured while reducing the motion blur and frame-to-frame jump phenomenon in the video, thereby facilitating maintaining a high visual consistency in dynamic scenes; and the optimization function is recorded as Optimize(); after completing the adjustment of the spatial detail control variable alpha and the temporal coherence control variable beta in the optimization function Optimize(), and processing the enhanced video Enhanced_video composed of enhanced video frames at different times through the optimization function Optimize(), the target video Final_output can be obtained, Final_output = Optimize(Enhanced_video, alpha, beta).

[0121] Specifically, the spatial detail control variable alpha and the temporal coherence control variable beta in the optimization function Optimize() are first adjusted, and then the enhanced video Enhanced_video consisting of enhanced video frames at different times is input into the optimization function Optimize() for processing, so as to obtain the target video Final_output.

[0122] It should be noted that by optimizing the enhanced video, the color, contrast, brightness of the enhanced video can be fine-tuned, and possible denoising and sharpening processing can be performed to ensure that each frame of the enhanced video can meet the optimal standard of visual effects.

[0123] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0124] Embodiment 8

[0125] Based on the same inventive concept, this embodiment also provides a video super-resolution device for implementing the video super-resolution method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more video super-resolution device embodiments provided below can refer to the above limitations on the video super-resolution method, which will not be repeated here.

[0126] In this embodiment, Figure 2 As shown, a video super-resolution device is provided, comprising:

[0127] A multi-scale encoding module is used to perform multi-scale encoding on the video to be processed to obtain video features at multiple scales;

[0128] A feature fusion module, used for fusing the video frame features corresponding to the same moment in the video features at different scales to obtain a fused feature;

[0129] A decoding and reconstruction module, used to obtain a high-resolution video frame based on the fused features and the up-sampled frame features corresponding to the same time as the fused features;

[0130] The enhancement optimization module is used to obtain the target video based on the high-resolution video frames corresponding to each of the moments.

[0131] Each module in the above video super-resolution device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0132] It should be noted that, in this embodiment, video features at multiple scales are obtained by multi-scale encoding of the video to be processed; fusion features are obtained by fusing the video frame features corresponding to the same moment in the video features at different scales; high-resolution video frames are obtained based on the fusion features and the up-sampled frame features corresponding to the same moment as the fusion features; and the target video is obtained based on the high-resolution video frames corresponding to each of the moments. Through the above implementation, by fusing the video features at the same moment in the video features at different scales, it is convenient to capture the moving and changing objects in the video to be processed, thereby improving the coherence between the video frames. Moreover, by generating high-resolution video frames through fusion features and up-sampled frame features, it is helpful to refine the video content and reduce the risk of information loss in the video frames, so as to prevent textures or artifacts that are not truly linear in the video frames; in addition, the above implementation avoids the participation of many model parameters, thereby improving the calculation speed of generating the target video; in summary, the present application can facilitate the improvement of the video super-resolution effect.

[0133] Embodiment 9

[0134] In this embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video super-resolution method is implemented.

[0135] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] Embodiment 10

[0137] In this embodiment, a computer readable storage medium is provided. Figure 4 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0138] Embodiment 11

[0139] In this embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A video super-resolution method, characterized in that: include: Perform multi-scale encoding on the video to be processed to obtain video features at multiple scales; Fusion of video frame features corresponding to the same moment in the video features at different scales to obtain fused features; Obtain a high-resolution video frame based on the fused feature and the up-sampled frame feature corresponding to the same time as the fused feature; A target video is obtained based on the high-resolution video frames corresponding to each of the moments.

2. The method according to claim 1, characterized in that The multi-scale encoding of the video to be processed is performed to obtain video features at multiple scales, including: Performing multi-scale downsampling on the video to be processed to obtain downsampled videos at multiple scales; The downsampled video is processed based on a preset diffusion model and model parameters in the diffusion model that correspond one-to-one to the multiple scales to obtain video features at multiple scales.

3. The method according to claim 1, characterized in that: The fusing the video frame features corresponding to the same moment in the video features at different scales to obtain the fused features includes: Obtaining the video frame features corresponding to each moment in the video features and the dynamic weights set for the video frame features; A fusion feature is obtained based on the video frame features corresponding to the same moment in the video features at different scales and the dynamic weights corresponding to the video frame features.

4. The method according to claim 3, characterized in that The obtaining of fusion features based on the video frame features corresponding to the same moment in the video features at different scales and the dynamic weights corresponding to the video frame features includes: Processing the video frame features corresponding to the same moment in the video features at different scales based on a preset conversion function to obtain converted features; Calculating the product of the converted feature and the corresponding dynamic weight to obtain a weighted feature; A fusion feature is obtained based on the weighted features corresponding to the different scales.

5. The method according to claim 1, characterized in that: The obtaining of a high-resolution video frame based on the fusion feature and the up-sampled frame feature corresponding to the same time as the fusion feature includes: Obtaining upsampled frame features based on the video to be processed; Merging the upsampled frame features and the fusion features corresponding to the same moment to obtain a merged feature; A high-resolution video frame is obtained based on the combined features.

6. The method according to claim 5, characterized in that The step of obtaining a high-resolution video frame based on the combined features includes: The combined features are processed based on a preset decoding function to obtain a high-resolution video frame.

7. The method according to claim 1, characterized in that The obtaining of the target video based on the high-resolution video frames corresponding to each of the moments comprises: Adaptively enhancing the high-resolution video frame based on a preset adaptive enhancement function and a feature weight array to obtain an enhanced video frame; Based on a preset optimization function, a spatial detail control variable, and a temporal coherence control variable, the enhanced video frames corresponding to the different moments are optimized to obtain a target video.

8. A video super-resolution device, characterized in that: The device comprises: A multi-scale encoding module is used to perform multi-scale encoding on the video to be processed to obtain video features at multiple scales; A feature fusion module, used for fusing the video frame features corresponding to the same moment in the video features at different scales to obtain a fused feature; A decoding and reconstruction module, used to obtain a high-resolution video frame based on the fused features and the up-sampled frame features corresponding to the same time as the fused features; The enhancement optimization module is used to obtain the target video based on the high-resolution video frames corresponding to each of the moments.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images