Video super-resolution method, system and device based on diffusion model and medium

Abstract

The embodiment of the invention provides a video super-resolution method, system and device based on a diffusion model and a medium, and the method comprises the steps: constructing a training sample set which comprises a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video; an initial diffusion model is constructed, and the initial diffusion model comprises an encoder module, a time embedding module, a Unet network module and a decoder module; performing multi-round iterative training on the initial diffusion model by using the training sample set to obtain a target diffusion model; and performing video super-resolution processing on a to-be-processed low-resolution video by using the target diffusion model to obtain a high-resolution video. A time embedding module is introduced into an initial diffusion model, so that the randomness and uncertainty of an image generation process in the diffusion model are suppressed, and the model can learn video frame features of different time steps.

Description

Technical Field

[0001] The application relates to the field of video image resolution technology, and in particular to a video super-resolution method, system, device and medium based on a diffusion model. Background Art

[0002] The diffusion model is a generative model based on probability theory, and its principle is derived from the diffusion process in physics. In the video super-resolution task, the diffusion model learns how to generate high-resolution videos from low-resolution videos by simulating a gradual "diffusion" process from low-resolution video frames to high-resolution video frames.

[0003] However, in the process of generating images, the diffusion model will introduce a certain amount of random noise to simulate the diffusion process from data distribution to noise distribution. Although this randomness increases the generation ability of the model to a certain extent, it also leads to the instability and unpredictability of the generated results, which leads to the randomness and uncertainty of the image generation process of the diffusion model in the video super-resolution task. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a video super-resolution method, system, device and medium based on a diffusion model, aiming to suppress the randomness and uncertainty of the image generation process in the diffusion model so that the model can learn the video frame features of different time steps.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a video super-resolution method based on a diffusion model, the method comprising:

[0006] Constructing a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video;

[0007] Constructing an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module;

[0008] The initial diffusion model is subjected to multiple rounds of iterative training using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining a model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss;

[0009] The target diffusion model is used to perform video super-resolution processing on the low-resolution video to be processed to obtain a high-resolution video.

[0010] The method provided in the first aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task. The purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model is achieved, so that the model can learn the video frame features of different time steps.

[0011] In one possible implementation, the encoder module includes: an input layer, a convolutional layer, and a fully connected layer;

[0012] The step of inputting the video frame of the low-resolution sample video into the encoder module to obtain the first high-dimensional vector comprises:

[0013] The input layer obtains the video frame of the low-resolution sample video;

[0014] The convolution layer extracts video data features of the video frame of the low-resolution sample video;

[0015] The fully connected layer converts the video data features into the first high-dimensional vector.

[0016] In a possible implementation, the convolution layer includes a common convolution layer, a dynamic convolution layer and a dimension reduction convolution layer;

[0017] The step of extracting video data features of the video frame of the low-resolution sample video by the convolution layer comprises:

[0018] The common convolution layer extracts a first data feature of a video frame of the low-resolution sample video;

[0019] The dynamic convolution layer adjusts the convolution kernel parameters based on the first data feature to obtain a second data feature;

[0020] The dimension reduction convolution layer reduces the data dimension of the second data feature to obtain the video data feature.

[0021] In a possible implementation, the dynamic convolution layer includes a dynamic convolution, a BN layer, and a Si LU activation function;

[0022] The dynamic convolution is used to calculate an attention weight based on the first data feature, and adjust the size of the convolution kernel according to the attention weight;

[0023] The BN layer is used to perform normalization processing on the first data feature;

[0024] The Si LU activation function is used to perform a nonlinear transformation on the first data feature.

[0025] In a possible implementation, the time embedding module includes: a position encoding layer, a linear layer, a ReLU activation function layer and a normalization processing layer;

[0026] The step of inputting the time step information into the time embedding module to obtain a second high-dimensional vector comprises:

[0027] The position encoding layer generates a position vector based on the time step information;

[0028] The linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector;

[0029] The ReLU activation function performs a nonlinear transformation on the linearly transformed vector to obtain a nonlinearly transformed vector;

[0030] The normalization processing layer performs normalization processing on the vector after the nonlinear transformation to obtain the second high-dimensional vector.

[0031] In one possible implementation, the position encoding layer generates a position vector based on the time step information, including: the position encoding layer captures periodic information of the time step information based on a sine-cosine function, and generates a position vector corresponding to the time step information based on the periodic information.

[0032] In a possible implementation manner, the step of adjusting model parameters of the initial diffusion model based on the model loss includes:

[0033] The parameters of the initial diffusion model are calculated based on the model loss and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimizer.

[0034] To achieve the above object, a second aspect of an embodiment of the present application proposes a video super-resolution system based on a diffusion model, the system comprising:

[0035] A training sample set construction module: used to construct a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video;

[0036] An initial diffusion model construction module: used to construct an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module;

[0037] Model training module: used to perform multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining the model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss;

[0038] Video super-resolution processing module: used to perform video super-resolution processing on the low-resolution video to be processed by using the target diffusion model to obtain a high-resolution video.

[0039] The system provided in the second aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task. The purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model is achieved, so that the model can learn the video frame features of different time steps.

[0040] In a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the diffusion model-based video super-resolution method as described in any possible implementation method in the first aspect is implemented.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the diffusion model-based video super-resolution method as described in any possible implementation manner in the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the description of one or more embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A flowchart of a video super-resolution method based on a diffusion model is provided in an embodiment of the present application.

[0044] Figure 2 A schematic diagram of a process for performing multiple rounds of iterative training on the initial diffusion model provided in an embodiment of the present application.

[0045] Figure 3 A schematic diagram of a framework of a diffusion model-based video super-resolution method provided in an embodiment of the present application.

[0046] Figure 4 A schematic diagram of the encoder network structure provided in an embodiment of the present application.

[0047] Figure 5 A schematic diagram of the structure of the time embedding module provided in an embodiment of the present application.

[0048] Figure 6 A schematic diagram of the flow of interaction between sub-network modules of the diffusion model provided in an embodiment of the present application.

[0049] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the one or more embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0054] Figure 1 is an optional flowchart of a video super-resolution method based on a diffusion model provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S400.

[0055] First, as Figure 1 As shown, a video super-resolution method based on a diffusion model is provided, the method comprising:

[0056] S100: Construct a training sample set, wherein the training sample set includes a plurality of sample videos, and each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video.

[0057] It should be noted that, in some embodiments, the number of samples of the high-resolution sample video is greater than the number of samples of the low-resolution sample video, which enhances the generalization ability of the diffusion model at different resolutions, helps to provide a large amount of training information and guidance for the diffusion model in the future, and enables the diffusion model to learn rich mapping relationships, so that the diffusion model can accurately restore high-resolution details.

[0058] S200, constructing an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module.

[0059] S300, performing multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model.

[0060] It should be noted that each round of iterative training gives the diffusion model the opportunity to further learn the features and patterns in the training data set. As the number of iterations increases, the model gradually adjusts its internal parameters to better fit the training data. However, in practical applications, too many iterations may cause the model to overfit, so that it cannot generalize well to new data. Therefore, it is necessary to select the number of iterations according to actual needs and use methods such as validation sets or cross-validation to monitor the performance of the model to ensure that the diffusion model has good generalization ability.

[0061] S400, performing video super-resolution processing on the low-resolution video to be processed by using the target diffusion model to obtain a high-resolution video.

[0062] Specifically, for step S300, Figure 2 As shown, the initial diffusion model is iteratively trained for multiple rounds using the training sample set to obtain a target diffusion model, and also includes steps S301 to S306.

[0063] S301: Input a video frame of a low-resolution sample video into an encoder module to obtain a first high-dimensional vector.

[0064] S302: Determine time step information based on multiple sample videos, input the time step information into a time embedding module, and obtain a second high-dimensional vector.

[0065] It should be noted that each frame of multiple sample videos corresponds to separate time step information. The time embedding module is used to process time-related information and embed timestamps or time series data into the initial diffusion model so that the initial diffusion model can effectively control the time step length of the diffusion when performing multiple rounds of iterative training, thereby suppressing the uncertainty and randomness in the diffusion process.

[0066] S303: Fuse the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector.

[0067] It should be noted that by fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector, the dimension and number of vector features can be reduced, the diffusion model structure can be simplified, and the computational complexity of the diffusion model can be reduced, which helps to improve the training efficiency of the diffusion model. In addition, through vector fusion, the diffusion model's anti-interference ability to noise and outliers is improved, which helps the diffusion model maintain stable performance in complex and changeable practical application scenarios.

[0068] S304: Input the fusion vector into the Unet network module to obtain video coding data.

[0069] Among them, Figure 3 As shown, Figure 3 The figure is a schematic diagram of the framework of the video super-resolution method based on the diffusion model. Specifically, the Unet network has a total of 23 layers, including three parts: the downsampling part, the middle part, and the upsampling part. The output of the downsampling part is transmitted to the upsampling part while being transmitted downward in each layer. Upsampling and downsampling have similar network structures. Downsampling includes two residual attention blocks, and there is a downsampling layer after every two residual attention blocks. In the residual attention block, the attention mechanism can improve the generation effect of the diffusion model on the video detail texture by learning the relationship between video frames, adjusting the weights between the pixels of the video frames, and reassigning them.

[0070] S305 , input the video encoding data into a decoder module to obtain a high-resolution predicted video.

[0071] It should be noted that the decoder module gradually restores the low-resolution features of the video encoding data to the original resolution through upsampling operations, and combines the jump connection part of the encoder to enable the decoder to refer to the high-resolution video features in the encoder while gradually restoring the low-resolution features of the video encoding data to the original resolution, and finally obtain a high-resolution predicted video.

[0072] S306: Determine the model loss based on the high-resolution predicted videos and the high-resolution sample videos corresponding to each low-resolution sample video, and adjust the model parameters of the initial diffusion model based on the model loss.

[0073] The purpose of adjusting the parameters of the initial diffusion model based on the model loss is to minimize the loss data of the initial diffusion model on the training data. Each round of iteration will update the model parameters according to the current loss value to reduce the loss of the model in the next round of iteration. This process of gradually reducing the loss helps to improve the performance of the diffusion model. As the number of iterations increases, the loss value of the diffusion model on the training data decreases to an ideal value, thereby obtaining a target diffusion model with good generalization ability.

[0074] The method provided in the first aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task, and achieves the purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model, so that the diffusion model can learn the video frame features of different time steps.

[0075] In one possible implementation, the encoder module includes: an input layer, a convolutional layer and a fully connected layer; the step of inputting the video frame of the low-resolution sample video into the encoder module to obtain the first high-dimensional vector includes: the input layer obtains the video frame of the low-resolution sample video; the convolutional layer extracts video data features of the video frame of the low-resolution sample video; and the fully connected layer converts the video data features into the first high-dimensional vector.

[0076] It should be noted that the video frame of the low-resolution sample video is input into the encoder module to obtain the first high-dimensional vector, which can comprehensively describe the content of the video frame of the low-resolution sample video and provide a basis for subsequent video processing and analysis tasks. In addition, the encoder module can reduce the redundant information of the video data by converting the video frame into a high-dimensional vector, thereby reducing storage requirements, improving the efficiency of video compression, and reducing the computational complexity of subsequent processing tasks.

[0077] In one possible implementation, Figure 4 As shown, the convolution layer includes a normal convolution layer, a dynamic convolution layer and a dimensionality reduction convolution layer; the step of extracting the video data features of the video frame of the low-resolution sample video by the convolution layer includes: the normal convolution layer extracts the first data feature of the video frame of the low-resolution sample video; the dynamic convolution layer adjusts the convolution kernel based on the first data feature to obtain the second data feature; the dimensionality reduction convolution layer reduces the data dimension of the second data feature to obtain the video data feature.

[0078] Among them, the ordinary convolution layer performs convolution operation with the input video frame data through the convolution kernel, extracts local features in the input data, and obtains the first data feature. The ordinary convolution layer can maintain the spatial structure of the input video frame data, that is, it can retain the relative position between the video pixels, and the convolution kernel of the ordinary convolution layer shares weights in different positions of the input video frame data, reducing the number of parameters of the initial diffusion model and improving the precision, learning efficiency and accuracy of the diffusion model in extracting video features. The filter used by the dynamic convolution layer can change according to the different input data. Therefore, the dynamic convolution layer can adaptively adjust the feature extraction method according to the characteristics of the first data feature output by the ordinary convolution layer. This adaptive characteristic enables the dynamic convolution layer to have higher flexibility and accuracy when processing complex and changeable input data, and the dynamic convolution layer can improve the generalization ability of the diffusion model, so that the model can better extract features and classify or identify when processing untouched input data. The dimension reduction convolution layer usually reduces the input data feature dimension through a 1×1 convolution kernel, which helps to reduce the amount of calculation and parameter of the diffusion model and improve the model processing speed.

[0079] In one possible implementation, Figure 4 As shown, the dynamic convolution layer includes dynamic convolution, BN layer and SiLU activation function; wherein, the dynamic convolution is used to calculate the attention weight based on the first data feature, and adjust the size of the convolution kernel according to the attention weight; the BN layer is used to normalize the first data feature; the SiLU activation function is used to perform nonlinear transformation on the first data feature.

[0080] Among them, it should be noted that dynamic convolution is used to cope with the diversity of input video frame features. By dynamically adjusting the size of the convolution kernel, the adaptive ability of the diffusion model to different image features is improved. In addition, the present application introduces dynamic convolution in each encoder module, and uses dynamic convolution to adjust the size of the convolution kernel so that it can automatically select a more efficient feature extraction strategy among different input image features, thereby enhancing the adaptability of the diffusion model to complex video scenes. In addition, by utilizing the variability of the kernel size of the dynamic convolution kernel, the feature information of the video frame can be better extracted, thereby obtaining more video frame detail features and the temporal relationship between video frames. The BN layer is used to normalize the first data feature to improve the stability and training efficiency of the diffusion model. The SiLU activation function performs a nonlinear transformation on the first data feature, which can enhance the nonlinear expression ability of the diffusion model while maintaining the smoothness of the gradient flow.

[0081] In one possible implementation, Figure 5As shown, the time embedding module includes: a position encoding layer, a linear layer, a ReLU activation function layer and a normalization processing layer; the step of inputting the time step information into the time embedding module to obtain a second high-dimensional vector includes: the position encoding layer generates a position vector based on the time step information; the linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector; the ReLU activation function performs a nonlinear transformation on the linearly transformed vector to obtain a nonlinearly transformed vector; the normalization processing layer performs normalization on the nonlinearly transformed vector to obtain the second high-dimensional vector.

[0082] It should be noted that, in some embodiments, the position encoding layer maps each time step information in the time series to a vector with position information, which can capture the relative position and order relationship between the time steps, and helps the diffusion model process time series data. The linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector to extract higher-level features, so that the linearly transformed vector can better reflect the characteristics of the input data. The ReLU activation function performs a nonlinear transformation on the linearly transformed vector, so that the diffusion model can learn more complex mapping relationships, enhance the expressive power of the model, and enable the diffusion model to better fit complex data distributions. It should also be noted that the ReLU activation function outputs equal to the input when the input is positive, and outputs zero when the input is negative. This unilateral inhibition effect helps the diffusion model learn the sparse representation of the data.

[0083] In one possible implementation, the position encoding layer generates a position vector based on the time step information, including: the position encoding layer captures periodic information of the time step information based on a sine-cosine function, and generates a position vector corresponding to the time step information based on the periodic information.

[0084] Among them, in some embodiments, the position vector generated by the position encoding layer based on the sine-cosine function can provide a unique position for each position in the input sequence. This uniqueness ensures that the diffusion model can accurately distinguish different positions in the input time series, thereby enhancing the model's understanding of the time series structure, enabling the diffusion model to learn video frame features at different time steps, and because the sine-cosine function has the characteristics of continuity, differentiability and periodicity, these characteristics enable the generated position vector to capture the continuous structure and relative position information of the sequence.

[0085] In one possible implementation, the step of adjusting the model parameters of the initial diffusion model based on the model loss includes: calculating the parameters of the initial diffusion model based on the model loss and a back propagation algorithm, obtaining the gradient of each parameter, and optimizing and updating the gradient of each parameter using an optimizer.

[0086] It should be noted that model loss is an important indicator to measure the difference between the model's predicted results and the actual results. By calculating the loss value, you can intuitively understand the performance of the model on the training data. When the loss value gradually decreases, it means that the diffusion model is gradually approaching the distribution of the real data. At this time, the change in the loss value can be used to guide the adjustment of the parameters. The back propagation algorithm calculates the gradient between each layer in the network and propagates the error signal from the output layer to the input layer layer by layer to obtain the gradient of each parameter. After obtaining the gradient of each parameter, the optimizer is used to optimize and update the parameters. Through multiple iterations of back propagation and parameter update, the diffusion model can better fit the training data, optimize the performance of the diffusion model, improve the generalization ability of the model, accelerate the model training process, and improve the efficiency of the model in handling complex tasks.

[0087] To achieve the above object, a second aspect of an embodiment of the present application proposes a video super-resolution system based on a diffusion model, the system comprising:

[0088] Training sample set construction module: used to construct a training sample set, wherein the training sample set includes multiple sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video.

[0089] It should be noted that, in some embodiments, the number of samples of the high-resolution sample video is greater than the number of samples of the low-resolution sample video, which enhances the generalization ability of the diffusion model at different resolutions, helps to provide a large amount of training information and guidance for the diffusion model in the future, and enables the diffusion model to learn rich mapping relationships, thereby enabling the diffusion model to accurately restore high-resolution details.

[0090] Initial diffusion model construction module: used to construct an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module.

[0091] It should be noted that if Figure 6As shown, the embodiment of the present application provides four interacting sub-network modules, wherein, in order to speed up the training speed of the diffusion model and reduce the number of parameters of the model, the present application introduces an encoder and a decoder module, the encoder module is used to extract low-resolution features of each frame or multiple frames of video, encode the low-resolution video frames into high-dimensional feature representations, and use a decoder to convert the features extracted by the encoder into high-resolution output frames, thereby effectively reducing the amount of computation and training time of the diffusion model; in order to control the randomness of the image generation process in the diffusion model, a time embedding module is introduced to encode the diffusion step information so that the diffusion model can learn the video frame features of different time steps; the Unet network based on the diffusion model trains the model in the latent space, so that the model learns the features of the high-resolution video in the latent space.

[0092] Model training module: used to perform multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining the time step information based on the multiple sample videos, and inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining the model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss.

[0093] It should be noted that the time embedding module is used to process time-related information and embed timestamps or time series data into the initial diffusion model so that the initial diffusion model can effectively control the time step length of the diffusion when performing multiple rounds of iterative training, thereby suppressing the uncertainty and randomness in the diffusion process.

[0094] Video super-resolution processing module: used to perform video super-resolution processing on the low-resolution video to be processed by using the target diffusion model to obtain a high-resolution video.

[0095] The system provided in the second aspect can solve the problem that the randomness of the diffusion model in the process of generating images in the prior art leads to the instability and unpredictability of the generated results, thereby causing the randomness and uncertainty of the process of generating images by the diffusion model in the video super-resolution task. The purpose of suppressing the randomness and uncertainty of the image generation process in the diffusion model is achieved, so that the model can learn the video frame features of different time steps.

[0096] The present application also provides an electronic device, such as Figure 7 As shown, the electronic device 1400 includes:

[0097] one or more processors 1410;

[0098] The memory 1420 stores one or more programs. When the one or more programs are executed by the one or more processors 1410, the one or more processors 1410 implement the diffusion model-based video super-resolution method provided in any embodiment of the present application.

[0099] The memory 1420, as a non-transient network system, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 1420 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1420 may optionally include a memory 1420 remotely arranged relative to the processor 1410, and these remote memories 1420 may be connected to the processor 1410 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0100] The memory 1420 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1420 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1420, and the processor 1410 calls and executes the methods of the embodiments of this application.

[0101] The processor 1410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0102] In some embodiments, the electronic device further comprises:

[0103] Input / output interface, used to realize information input and output;

[0104] Communication interface, used to realize communication interaction between this device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0105] A bus that transmits information between various components of the device (e.g., processor 1410, memory 1420, input / output interface, and communication interface);

[0106] The processor 1410 , the memory 1420 , the input / output interface and the communication interface can be connected to each other in communication within the device via a bus.

[0107] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the diffusion model-based video super-resolution method provided by any embodiment of the present application.

[0108] An embodiment of the present application also provides a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the diffusion model-based video super-resolution method provided in any embodiment of the present application.

[0109] The system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0110] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0111] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0112] The above describes some embodiments of the present application with reference to the accompanying drawings, but does not limit the scope of the present invention. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present application.

[0113] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0114] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0117] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video super-resolution method based on a diffusion model, characterized in that: The method comprises: Constructing a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video; Constructing an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module; The initial diffusion model is subjected to multiple rounds of iterative training using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining a model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss; The target diffusion model is used to perform video super-resolution processing on the low-resolution video to be processed to obtain a high-resolution video.

2. The method according to claim 1, characterized in that The encoder module includes: an input layer, a convolutional layer, and a fully connected layer; The step of inputting the video frame of the low-resolution sample video into the encoder module to obtain the first high-dimensional vector comprises: The input layer obtains the video frame of the low-resolution sample video; The convolution layer extracts video data features of the video frame of the low-resolution sample video; The fully connected layer converts the video data features into the first high-dimensional vector.

3. The method according to claim 2, characterized in that The convolutional layer includes a common convolutional layer, a dynamic convolutional layer and a dimension reduction convolutional layer; The step of extracting video data features of the video frame of the low-resolution sample video by the convolution layer comprises: The common convolution layer extracts a first data feature of a video frame of the low-resolution sample video; The dynamic convolution layer adjusts the convolution kernel parameters based on the first data feature to obtain a second data feature; The dimension reduction convolution layer reduces the data dimension of the second data feature to obtain the video data feature.

4. The method according to claim 3, characterized in that The dynamic convolution layer includes dynamic convolution, BN layer and SiLU activation function; The dynamic convolution is used to calculate an attention weight based on the first data feature, and adjust the size of the convolution kernel according to the attention weight; The BN layer is used to perform normalization processing on the first data feature; The SiLU activation function is used to perform a nonlinear transformation on the first data feature.

5. The method according to claim 1, characterized in that The time embedding module includes: a position encoding layer, a linear layer, a ReLU activation function layer and a normalization processing layer; The step of inputting the time step information into the time embedding module to obtain a second high-dimensional vector comprises: The position encoding layer generates a position vector based on the time step information; The linear layer performs a linear transformation on the position vector to obtain a linearly transformed vector; The ReLU activation function performs a nonlinear transformation on the linearly transformed vector to obtain a nonlinearly transformed vector; The normalization processing layer performs normalization processing on the vector after the nonlinear transformation to obtain the second high-dimensional vector.

6. The method according to claim 5, characterized in that The position encoding layer generates a position vector based on the time step information, including: the position encoding layer captures period information of the time step information based on a sine-cosine function, and generates a position vector corresponding to the time step information based on the period information.

7. The method according to claim 1, characterized in that The step of adjusting the model parameters of the initial diffusion model based on the model loss includes: The parameters of the initial diffusion model are calculated based on the model loss and the back propagation algorithm to obtain the gradient of each parameter, and the gradient of each parameter is optimized and updated using an optimizer.

8. A video super-resolution system based on a diffusion model, characterized in that: The system comprises: A training sample set construction module: used to construct a training sample set, wherein the training sample set includes a plurality of sample videos, each sample video includes a low-resolution sample video and a high-resolution sample video corresponding to the low-resolution sample video; An initial diffusion model construction module: used to construct an initial diffusion model, wherein the initial diffusion model includes an encoder module, a time embedding module, a Unet network module and a decoder module; Model training module: used to perform multiple rounds of iterative training on the initial diffusion model using the training sample set to obtain a target diffusion model, wherein each round of iterative training includes: inputting the video frame of the low-resolution sample video into the encoder module to obtain a first high-dimensional vector; determining time step information based on the multiple sample videos, inputting the time step information into the time embedding module to obtain a second high-dimensional vector; fusing the first high-dimensional vector and the second high-dimensional vector to obtain a fused vector; inputting the fused vector into the Unet network module to obtain video encoding data; inputting the video encoding data into the decoder module to obtain a high-resolution predicted video; determining the model loss based on the high-resolution predicted video and the high-resolution sample video corresponding to each of the low-resolution sample videos, and adjusting the model parameters of the initial diffusion model based on the model loss; Video super-resolution processing module: used to perform video super-resolution processing on the low-resolution video to be processed by using the target diffusion model to obtain a high-resolution video.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the diffusion model-based video super-resolution method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the diffusion model-based video super-resolution method according to any one of claims 1 to 7 is implemented.

Images