Video super-resolution method, system, device and medium based on diffusion model

Abstract

The present application provides a method, system, device, and medium for video super-resolution based on a diffusion model. A method for video super-resolution based on a diffusion model includes: constructing a training sample set, wherein the training sample set includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video; constructing an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module, and a decoder module; performing multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model; and performing video super-resolution processing on the low-resolution video to be processed using the target diffusion model to obtain a high-resolution video. By introducing the time embedding module into the initial diffusion model, the randomness and uncertainty of the image generation process in the diffusion model are suppressed, enabling the model to learn the characteristics of video frames at different time steps.

Description

Technical Field

[0001] The application relates to the field of video image resolution technology, and in particular to a video super-resolution method, system, device and medium based on a diffusion model. Background Art

[0002] The Diffusion Model (DM) is a probabilistic generative model whose principle is derived from the diffusion process in physics. In the video super-resolution task, the Diffusion Model learns how to generate high-resolution video from low-resolution video by simulating a gradual "diffusion" process from low-resolution video frames to high-resolution video frames.

[0003] However, during the image generation process, the diffusion model introduces a certain amount of random noise to simulate the diffusion process from the data distribution to the noise distribution. While this randomness improves the model's generative capabilities to a certain extent, it also leads to instability and unpredictability in the generated results. This leads to randomness and uncertainty in the image generation process of the diffusion model in video super-resolution tasks. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a video super-resolution method, system, device and medium based on a diffusion model, aiming to suppress the randomness and uncertainty of the image generation process in the diffusion model so that the model can learn the video frame features of different time steps.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a video super-resolution method based on a diffusion model, the method comprising:

[0006] Constructing a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video;

[0007] Constructing an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module;

[0008] The initial diffusion model is subjected to multiple rounds of iterative training using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining a model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss;

[0009] The target diffusion model is used to perform video super-resolution processing on the low-resolution video to be processed to obtain a high-resolution video.

[0010] The method provided in the first aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task. The purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model is achieved, so that the model can learn the video frame features of different time steps.

[0011] In one possible implementation, the encoder module includes: an input layer, a convolutional layer, and a fully connected layer;

[0012] The step of inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector includes:

[0013] The input layer obtains the video frame of the low-resolution sample video;

[0014] The convolutional layer extracts video data features of the video frame of the low-resolution sample video;

[0015] The fully connected layer converts the video data features into the first high-dimensional vector.

[0016] In one possible implementation, the convolution layer includes a normal convolution layer, a dynamic convolution layer, and a dimensionality reduction convolution layer;

[0017] The step of extracting video data features of the video frame of the low-resolution sample video by the convolutional layer includes:

[0018] The common convolution layer extracts a first data feature of a video frame of the low-resolution sample video;

[0019] The dynamic convolution layer adjusts the convolution kernel parameters based on the first data feature to obtain a second data feature;

[0020] The dimensionality reduction convolution layer reduces the data dimension of the second data feature to obtain the video data feature.

[0021] In one possible implementation, the dynamic convolution layer includes dynamic convolution, BN layer and Si LU activation function;

[0022] The dynamic convolution is used to calculate an attention weight based on the first data feature and adjust the size of the convolution kernel according to the attention weight;

[0023] The BN layer is used to perform normalization processing on the first data feature;

[0024] The Si LU activation function is used to perform a nonlinear transformation on the first data feature.

[0025] In one possible implementation, the time embedding module includes: a position encoding layer, a linear layer, a ReLU activation function layer, and a normalization processing layer;

[0026] The step of inputting the time step information into the time embedding module to obtain a second high-dimensional vector comprises:

[0027] The position encoding layer generates a position vector based on the time step information;

[0028] The linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector;

[0029] The ReLU activation function performs a nonlinear transformation on the linearly transformed vector to obtain a nonlinearly transformed vector;

[0030] The normalization processing layer performs normalization processing on the vector after the nonlinear transformation to obtain the second high-dimensional vector.

[0031] In one possible implementation, the position encoding layer generates a position vector based on the time step information, including: the position encoding layer captures periodic information of the time step information based on a sine-cosine function, and generates a position vector corresponding to the time step information based on the periodic information.

[0032] In one possible implementation, the step of adjusting model parameters of the initial diffusion model based on the model loss includes:

[0033] The parameters of the initial diffusion model are calculated based on the model loss and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimizer.

[0034] To achieve the above objectives, a second aspect of an embodiment of the present application provides a video super-resolution system based on a diffusion model, the system comprising:

[0035] A training sample set construction module is used to construct a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video;

[0036] Initial diffusion model construction module: used to construct an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module;

[0037] Model training module: used to perform multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining a model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss;

[0038] Video super-resolution processing module: used to perform video super-resolution processing on the low-resolution video to be processed using the target diffusion model to obtain a high-resolution video.

[0039] The system provided in the second aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task. The purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model is achieved, so that the model can learn the video frame features of different time steps.

[0040] In a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the diffusion model-based video super-resolution method as described in any possible implementation method in the first aspect is implemented.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the diffusion model-based video super-resolution method as described in any possible implementation manner in the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the description of one or more embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flow chart of a video super-resolution method based on a diffusion model provided in an embodiment of the present application.

[0044] Figure 2 A schematic diagram of a process for performing multiple rounds of iterative training of the initial diffusion model provided in an embodiment of the present application.

[0045] Figure 3 A schematic diagram of a framework of a diffusion model-based video super-resolution method provided in an embodiment of the present application.

[0046] Figure 4 Schematic diagram of the encoder network structure provided in an embodiment of the present application.

[0047] Figure 5 A schematic diagram of the structure of the time embedding module provided in an embodiment of the present application.

[0048] Figure 6 A schematic diagram of the flow of interaction between sub-network modules of the diffusion model provided in an embodiment of the present application.

[0049] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the one or more embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0051] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0053] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0054] Figure 1 This is an optional flowchart of the video super-resolution method based on the diffusion model provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S400.

[0055] First, as Figure 1 As shown, a video super-resolution method based on a diffusion model is provided, the method comprising:

[0056] S100: Construct a training sample set, wherein the training sample set includes multiple sample videos, and each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video.

[0057] It should be noted that, in some embodiments, the number of samples of the high-resolution sample video is greater than the number of samples of the low-resolution sample video, which enhances the generalization ability of the diffusion model at different resolutions, helps to provide a large amount of training information and guidance for the diffusion model in the future, and enables the diffusion model to learn rich mapping relationships, so that the diffusion model can accurately restore high-resolution details.

[0058] S200 , constructing an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module, and a decoder module.

[0059] S300 , performing multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model.

[0060] It's important to note that each round of iterative training gives the diffusion model an opportunity to further learn the characteristics and patterns in the training dataset. As the number of iterations increases, the model gradually adjusts its internal parameters to better fit the training data. However, in practice, excessive iterations can lead to overfitting of the model, preventing it from generalizing well to new data. Therefore, it's important to select the number of iterations based on actual needs and monitor model performance using validation sets or cross-validation to ensure good generalization of the diffusion model.

[0061] S400 , performing video super-resolution processing on the low-resolution video to be processed using a target diffusion model to obtain a high-resolution video.

[0062] Specifically, for step S300, if Figure 2 As shown, the initial diffusion model is iteratively trained for multiple rounds using the training sample set to obtain the target diffusion model, and further includes steps S301 to S306.

[0063] S301: Input a video frame of a low-resolution sample video into an encoder module to obtain a first high-dimensional vector.

[0064] S302: Determine time step information based on multiple sample videos, input the time step information into a time embedding module, and obtain a second high-dimensional vector.

[0065] It should be noted that each frame of multiple sample videos corresponds to separate time step information. The time embedding module is used to process time-related information and embed timestamps or time series data into the initial diffusion model so that the initial diffusion model can effectively control the time step length of diffusion during multiple rounds of iterative training and suppress the uncertainty and randomness in the diffusion process.

[0066] S303: Fuse the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector.

[0067] It is important to note that fusing the first and second high-dimensional vectors to obtain a fused vector reduces the dimensionality and number of vector features, simplifying the diffusion model structure and thus reducing the computational complexity of the diffusion model, thereby improving its training efficiency. Furthermore, vector fusion enhances the diffusion model's resistance to noise and outliers, helping it maintain stable performance in complex and changing real-world application scenarios.

[0068] S304: Input the fusion vector into the Unet network module to obtain video coding data.

[0069] Among them, such as Figure 3 As shown, Figure 3 This figure shows the framework of a diffusion model-based video super-resolution method. Specifically, the Unet network has 23 layers, consisting of three parts: a downsampling layer, an intermediate layer, and an upsampling layer. The output of each downsampling layer is passed to the upsampling layer as it is passed down. Upsampling and downsampling have similar network structures. Downsampling includes two residual attention blocks, with a downsampling layer following each residual attention block. Within the residual attention blocks, the attention mechanism learns the relationships between video frames, adjusts the weights between pixel points, and reassigns them, improving the diffusion model's ability to generate detailed textures in the video.

[0070] S305 , input the video encoding data into a decoder module to obtain a high-resolution predicted video.

[0071] It should be noted that the decoder module gradually restores the low-resolution features of the video encoding data to the original resolution through the upsampling operation, and combines the jump connection part of the encoder to enable the decoder to refer to the high-resolution video features in the encoder in the process of gradually restoring the low-resolution features of the video encoding data to the original resolution, and finally obtain a high-resolution predicted video.

[0072] S306: Determine the model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each low-resolution sample video, and adjust the model parameters of the initial diffusion model based on the model loss.

[0073] The goal of tuning the initial diffusion model parameters based on model loss is to minimize the initial diffusion model's loss on the training data. Each iteration updates the model parameters based on the current loss value, aiming to reduce the model's loss in the next iteration. This gradual loss reduction process helps improve the diffusion model's performance. As the number of iterations increases, the diffusion model's loss on the training data decreases to an ideal value, resulting in a target diffusion model with good generalization capabilities.

[0074] The method provided in the first aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task. The purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model is achieved, so that the diffusion model can learn the characteristics of video frames at different time steps.

[0075] In one possible implementation, the encoder module includes: an input layer, a convolutional layer, and a fully connected layer; the step of inputting the video frame of the low-resolution sample video into the encoder module to obtain the first high-dimensional vector includes: the input layer obtains the video frame of the low-resolution sample video; the convolutional layer extracts the video data features of the video frame of the low-resolution sample video; and the fully connected layer converts the video data features into the first high-dimensional vector.

[0076] It should be noted that the video frame of the low-resolution sample video is input into the encoder module to obtain a first high-dimensional vector. The high-dimensional vector can comprehensively describe the content of the video frame of the low-resolution sample video, providing a basis for subsequent video processing and analysis tasks. In addition, the encoder module can reduce the redundant information of the video data by converting the video frame into a high-dimensional vector, thereby reducing storage requirements, improving the efficiency of video compression, and reducing the computational complexity of subsequent processing tasks.

[0077] In one possible implementation, Figure 4 As shown, the convolution layer includes a normal convolution layer, a dynamic convolution layer and a dimensionality reduction convolution layer; the step of the convolution layer extracting the video data features of the video frame of the low-resolution sample video includes: the normal convolution layer extracts the first data feature of the video frame of the low-resolution sample video; the dynamic convolution layer adjusts the convolution kernel based on the first data feature to obtain the second data feature; the dimensionality reduction convolution layer reduces the data dimension of the second data feature to obtain the video data feature.

[0078] The standard convolutional layer performs a convolution operation with the input video frame data using a convolution kernel, extracting local features from the input data and obtaining the first data feature. The standard convolutional layer maintains the spatial structure of the input video frame data, preserving the relative positions of video pixels. The convolution kernels in the standard convolutional layer share weights across different locations in the input video frame data, reducing the number of parameters in the initial diffusion model and improving the sophistication, learning efficiency, and accuracy of the diffusion model's video feature extraction. The dynamic convolutional layer uses filters that vary based on the input data. Therefore, the dynamic convolutional layer can adaptively adjust its feature extraction method based on the characteristics of the first data feature output by the standard convolutional layer. This adaptive nature enables the dynamic convolutional layer to handle complex and changing input data with greater flexibility and accuracy. The dynamic convolutional layer also improves the generalization of the diffusion model, enabling the model to better extract features and perform classification or recognition when processing unseen input data. The dimensionality reduction convolutional layer typically reduces the dimensionality of the input data features using a 1×1 convolution kernel, helping to reduce the computational complexity and parameter count of the diffusion model and improve model processing speed.

[0079] In one possible implementation, Figure 4 As shown, the dynamic convolution layer includes dynamic convolution, BN layer and SiLU activation function; wherein, the dynamic convolution is used to calculate the attention weight based on the first data feature and adjust the size of the convolution kernel according to the attention weight; the BN layer is used to normalize the first data feature; the SiLU activation function is used to perform nonlinear transformation on the first data feature.

[0080] Among them, it should be noted that dynamic convolution is used to cope with the diversity of input video frame features. By dynamically adjusting the size of the convolution kernel, the adaptive ability of the diffusion model to different image features is improved. In addition, the present application introduces dynamic convolution in each encoder module, and uses dynamic convolution to adjust the size of the convolution kernel so that it can automatically select a more efficient feature extraction strategy among different input image features, thereby enhancing the adaptability of the diffusion model to complex video scenes. In addition, by utilizing the variability of the kernel size of the dynamic convolution kernel, the feature information of the video frame can be better extracted, thereby obtaining more detailed features of the video frame and the temporal relationship between the video frames. The BN layer is used to normalize the first data feature to improve the stability and training efficiency of the diffusion model. The SiLU activation function performs a nonlinear transformation on the first data feature, which can enhance the nonlinear expression ability of the diffusion model while maintaining the smoothness of the gradient flow.

[0081] In one possible implementation, Figure 5As shown, the time embedding module includes: a position encoding layer, a linear layer, a ReLU activation function layer and a normalization processing layer; the step of inputting the time step information into the time embedding module to obtain a second high-dimensional vector includes: the position encoding layer generates a position vector based on the time step information; the linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector; the ReLU activation function performs a nonlinear transformation on the linearly transformed vector to obtain a nonlinearly transformed vector; the normalization processing layer performs normalization on the nonlinearly transformed vector to obtain the second high-dimensional vector.

[0082] It should be noted that, in some embodiments, the position encoding layer maps each time step information in the time series to a vector with position information, which can capture the relative position and order relationship between the time steps, and helps the diffusion model process time series data. The linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector to extract higher-level features, so that the linearly transformed vector can better reflect the characteristics of the input data. The ReLU activation function performs a nonlinear transformation on the linearly transformed vector, so that the diffusion model can learn more complex mapping relationships, enhance the expressive power of the model, and enable the diffusion model to better fit complex data distributions. It should also be noted that the ReLU activation function outputs equal to the input when the input is positive, and outputs zero when the input is negative. This unilateral inhibition effect helps the diffusion model learn a sparse representation of the data.

[0083] In one possible implementation, the position encoding layer generates a position vector based on the time step information, including: the position encoding layer captures periodic information of the time step information based on a sine-cosine function, and generates a position vector corresponding to the time step information based on the periodic information.

[0084] In some embodiments, the position vector generated by the position encoding layer based on the sine-cosine function can provide a unique position for each position in the input sequence. This uniqueness ensures that the diffusion model can accurately distinguish different positions in the input time series, thereby enhancing the model's understanding of the time series structure, enabling the diffusion model to learn video frame features of different time steps, and because the sine-cosine function has the characteristics of continuity, differentiability and periodicity, these characteristics enable the generated position vector to capture the continuous structure and relative position information of the sequence.

[0085] In one possible implementation, the step of adjusting the model parameters of the initial diffusion model based on the model loss includes: calculating the parameters of the initial diffusion model based on the model loss and the back propagation algorithm, obtaining the gradient of each parameter, and optimizing and updating the gradient of each parameter using an optimizer.

[0086] It should be noted that model loss is an important metric that measures the difference between the model's predictions and the actual results. By calculating the loss value, one can intuitively understand the model's performance on the training data. As the loss value gradually decreases, it indicates that the diffusion model is gradually approaching the distribution of the real data. At this point, the change in the loss value can be used to guide parameter adjustments. The backpropagation algorithm calculates the gradients between each layer in the network, propagating the error signal from the output layer to the input layer layer by layer, thereby obtaining the gradient of each parameter. After obtaining the gradient of each parameter, the optimizer is used to optimize and update the parameters. Through multiple iterations of backpropagation and parameter updates, the diffusion model can better fit the training data, optimize its performance, improve its generalization ability, accelerate the model training process, and improve its efficiency in handling complex tasks.

[0087] To achieve the above objectives, a second aspect of an embodiment of the present application provides a video super-resolution system based on a diffusion model, the system comprising:

[0088] Training sample set construction module: used to construct a training sample set, wherein the training sample set includes multiple sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video.

[0089] It should be noted that, in some embodiments, the number of samples of the high-resolution sample video is greater than the number of samples of the low-resolution sample video, which enhances the generalization ability of the diffusion model at different resolutions, helps to provide a large amount of training information and guidance for the diffusion model in the future, and enables the diffusion model to learn rich mapping relationships, thereby enabling the diffusion model to accurately restore high-resolution details.

[0090] Initial diffusion model construction module: used to construct an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module.

[0091] It should be noted that if Figure 6As shown, the embodiment of the present application provides four interacting sub-network modules, wherein, in order to speed up the training speed of the diffusion model and reduce the number of parameters of the model, the present application introduces an encoder and a decoder module, the encoder module is used to extract low-resolution features of each frame or multiple frames of video, encode the low-resolution video frames into high-dimensional feature representations, and use the decoder to convert the features extracted by the encoder into high-resolution output frames, thereby effectively reducing the computational complexity and training time of the diffusion model; in order to control the randomness in the image generation process in the diffusion model, a time embedding module is introduced to encode the diffusion step information so that the diffusion model can learn the video frame features of different time steps; the Unet network based on the diffusion model trains the model in the latent space, so that the model learns the features of the high-resolution video in the latent space.

[0092] Model training module: used to perform multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining the time step information based on the multiple sample videos, and inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining the model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss.

[0093] It should be noted that the time embedding module is used to process time-related information and embed timestamps or time series data into the initial diffusion model so that the initial diffusion model can effectively control the time step length of the diffusion during multiple rounds of iterative training and suppress the uncertainty and randomness in the diffusion process.

[0094] Video super-resolution processing module: used to perform video super-resolution processing on the low-resolution video to be processed using the target diffusion model to obtain a high-resolution video.

[0095] The system provided in the second aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task. The purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model is achieved, so that the model can learn the video frame features of different time steps.

[0096] The present application also provides an electronic device, such as Figure 7 As shown, the electronic device 1400 includes:

[0097] one or more processors 1410;

[0098] The memory 1420 stores one or more programs. When the one or more programs are executed by the one or more processors 1410, the one or more processors 1410 implement the diffusion model-based video super-resolution method provided in any embodiment of the present application.

[0099] The memory 1420 is a non-transient network system that can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 1420 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1420 may optionally include a memory 1420 remotely located relative to the processor 1410, and these remote memories 1420 may be connected to the processor 1410 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0100] The memory 1420 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1420 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1420 and is called by the processor 1410 to execute the methods of the embodiments of this application.

[0101] The processor 1410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0102] In some embodiments, the electronic device further comprises:

[0103] Input / output interface, used to realize information input and output;

[0104] Communication interface, used to enable communication between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0105] A bus that transmits information between various components of the device (e.g., the processor 1410, memory 1420, input / output interfaces, and communication interfaces);

[0106] The processor 1410 , the memory 1420 , the input / output interface, and the communication interface can be communicatively connected to each other within the device via a bus.

[0107] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the diffusion model-based video super-resolution method provided in any embodiment of the present application.

[0108] An embodiment of the present application further provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the diffusion model-based video super-resolution method provided in any embodiment of the present application.

[0109] The system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0110] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0111] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0112] The above description of some embodiments of the present application with reference to the accompanying drawings does not limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present application.

[0113] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0114] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0115] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0116] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0117] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A video super-resolution method based on a diffusion model, characterized in that: The method comprises: Constructing a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video; Constructing an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module; The initial diffusion model is subjected to multiple rounds of iterative training using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining a model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss; The target diffusion model is used to perform video super-resolution processing on the low-resolution video to be processed to obtain a high-resolution video.

2. The method according to claim 1, characterized in that The encoder module includes: an input layer, a convolutional layer, and a fully connected layer; The step of inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector includes: The input layer obtains the video frame of the low-resolution sample video; The convolutional layer extracts video data features of the video frame of the low-resolution sample video; The fully connected layer converts the video data features into the first high-dimensional vector.

3. The method according to claim 2, characterized in that The convolutional layer includes a normal convolutional layer, a dynamic convolutional layer and a dimensionality reduction convolutional layer; The step of extracting video data features of the video frame of the low-resolution sample video by the convolutional layer includes: The common convolution layer extracts a first data feature of a video frame of the low-resolution sample video; The dynamic convolution layer adjusts the convolution kernel parameters based on the first data feature to obtain a second data feature; The dimensionality reduction convolution layer reduces the data dimension of the second data feature to obtain the video data feature.

4. The method according to claim 3, characterized in that The dynamic convolution layer includes dynamic convolution, BN layer and SiLU activation function; The dynamic convolution is used to calculate an attention weight based on the first data feature and adjust the size of the convolution kernel according to the attention weight; The BN layer is used to perform normalization processing on the first data feature; The SiLU activation function is used to perform a nonlinear transformation on the first data feature.

5. The method according to claim 1, wherein The time embedding module includes: a position encoding layer, a linear layer, a ReLU activation function layer and a normalization processing layer; The step of inputting the time step information into the time embedding module to obtain a second high-dimensional vector comprises: The position encoding layer generates a position vector based on the time step information; The linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector; The ReLU activation function performs a nonlinear transformation on the linearly transformed vector to obtain a nonlinearly transformed vector; The normalization processing layer performs normalization processing on the vector after the nonlinear transformation to obtain the second high-dimensional vector.

6. The method according to claim 5, characterized in that The position encoding layer generates a position vector based on the time step information, including: the position encoding layer captures period information of the time step information based on a sine-cosine function, and generates a position vector corresponding to the time step information based on the period information.

7. The method according to claim 1, characterized in that The step of adjusting the model parameters of the initial diffusion model based on the model loss includes: The parameters of the initial diffusion model are calculated based on the model loss and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimizer.

8. A video super-resolution system based on a diffusion model, characterized in that: The system comprises: A training sample set construction module is used to construct a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video; Initial diffusion model construction module: used to construct an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module; Model training module: used to perform multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining a model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss; Video super-resolution processing module: used to perform video super-resolution processing on the low-resolution video to be processed using the target diffusion model to obtain a high-resolution video.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the diffusion model-based video super-resolution method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the diffusion model-based video super-resolution method according to any one of claims 1 to 7 is implemented.

Images