Video deblurring method based on memory diffusion network

Abstract

The application provides a video deblurring method based on a memory diffusion network, and the core of the method comprises a deblurring branch and a diffusion branch. The deblurring branch has a bidirectional circulation structure and maintains a compact and effective memory bank comprising a specific updating mechanism, which can effectively utilize the space-time relationship between video frames and further ensure the time consistency between the frames in the final deblurred video. The diffusion branch comprises a lightweight diffusion network to generate multi-scale diffusion prior features. The application improves the effect of the existing regression model on video deblurring by combining the diffusion prior generated by the diffusion network. A compact and effective memory bank is constructed to filter and store time frame information to accurately capture the complex space-time local relationship in the blurred video to achieve finer-grained deblurring. A specific memory updating algorithm is designed to refresh the memory bank, so as to better utilize the space-time information between continuous video frames and ensure the time consistency between the frames.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to a video deblurring method based on a memory diffusion network. Background Technology

[0002] In reality, blur in videos can originate from various sources, such as object motion, camera shake, and changes in scene depth. These factors are often intertwined, making it difficult to accurately capture the complex spatiotemporal relationships in blurry videos, thus posing a challenge to subsequent deblurring.

[0003] Several deep learning-based video deblurring methods have been proposed to leverage complex spatiotemporal information in the input frame sequence from different perspectives. On one hand, CNN-based models typically feed the current frame and adjacent consecutive blurred frames into an encoder-decoder to directly recover the sharp frame. On the other hand, RNN-based models maintain hidden states between video frames to sequentially propagate features from the first frame to the last. However, this approach relies solely on information from previous frames and cannot capture more complex motion relationships. Recently, there has been an increase in work utilizing self-attention mechanisms to address the complexity of video blur (such as the spatiotemporal inhomogeneity of blur). However, these networks typically only utilize coarse attention mechanisms and cannot effectively capture local motion-related features between video frames.

[0004] Existing methods based on CNNs, RNNs, and self-attention mechanisms generally only utilize information from a subset of input frames and cannot effectively handle spatiotemporal local features. Adjacent frames are often utilized through simple concatenation or frame alignment, leading to either performance limitations or high computational complexity. Recent work has explored the potential of memory networks for video deblurring, which can not only capture spatiotemporal local features more effectively but also utilize the spatiotemporal information of all input frames—a deficiency in other methods. However, existing memory network-based methods simply maintain a certain number of memory frames, resetting them upon expiration and discarding all previous memory information, thus failing to achieve effective utilization. This is not an effective memory update mechanism.

[0005] Most recent methods (including recent memory network-based methods) are based on regression models, which tend to be highly sensitive to degradation or have limited expressive power because these networks essentially still rely on information from highly degraded LQ input frames. Generative priors can alleviate this situation to some extent. For example, some image restoration methods incorporate diffusion networks during training to further introduce ground truth rather than just degraded LQ frames. The diffusion priors generated by adding and denoising in the latent space effectively complement the regression models used, a feature that current video deblurring methods have not yet explored in depth. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a video deblurring method based on a memory-diffusion network. This invention further explores the potential of combining existing regression models with diffusion priors generated by diffusion networks for video deblurring. Its core comprises two branches: a deblurring branch and a diffusion branch. The deblurring branch has a bidirectional cyclic structure and maintains a concise and efficient memory bank with a specific update mechanism, which effectively utilizes the spatiotemporal relationships between video frames to further ensure temporal consistency among frames in the final deblurred video. The diffusion branch contains a lightweight diffusion network to generate multi-scale diffusion prior features.

[0007] The technical means employed in this invention are as follows:

[0008] A video deblurring method based on memory diffusion networks includes:

[0009] S1. Select a publicly available video deblurring dataset as the ground truth, preprocess the input blurred video frames using residual dense blocks, and downsample to obtain the initial downsampled features x. i ;

[0010] S2. The blurred input frame and the corresponding Ground Truth frame are concatenated together, encoded by a latent encoder, and then subjected to a forward diffusion noise generation process to generate random noise. This noise is then reversed by a diffusion denoising network to generate multi-scale diffusion prior features.

[0011] S3. The downsampled feature x obtained in step S1 i The features are fed into the feature encoder for encoding to obtain multi-scale encoded features; during encoding, the multi-scale diffusion prior features generated in step S2 are... It is integrated into the encoder through a layered integration module;

[0012] S4. Before decoding, the multi-scale coding features obtained in step S3 are used to construct a query key to generate a query for memory feature retrieval in the memory bank. The retrieved memory features, the maintained transient perceptual memory, and the third-scale coding features are sent to the memory decoder to enhance the coding features in the early stage.

[0013] S5. Input the enhanced encoded features from step S4 into the feature decoder for decoding to obtain multi-scale decoded features;

[0014] S6. Input the obtained first-scale decoding features, the features extracted by the previous loop feature extraction module, and the initial downsampled features into the loop feature extraction module to further enhance the decoding features;

[0015] S7. Input the previously encoded features, enhanced decoding features and the initial downsampling features into the target frame reconstruction module to reconstruct the target frame. Then, after transpose convolution upsampling, it is fused with the original blurred input to obtain the final deblurring result.

[0016] S8. Use L1 Charbonnier loss and L1 loss during the diffusion process to jointly train the entire network.

[0017] Furthermore, in step S1, the initial downsampled feature x is obtained. i ,as follows:

[0018] x i =RDB(I i ), i = 1, 2, ..., N

[0019] Where, x i I represents the downsampling feature. i Indicates fuzzy input frame I Blur RDB represents the residual dense block used for downsampling.

[0020] Further, step S2 specifically includes:

[0021] S21. During the forward diffusion process, z is transformed into Gaussian noise through T iterations, as shown in the following equation:

[0022]

[0023] Where t = 1, 2, ..., T; β t ∈(0,1) represents the hyperparameter used to control the noise variance; N represents the Gaussian distribution; I represents the standard normal distribution;

[0024] S22, Let α t =1-β t , The formula in step S21 can be rewritten through iterative derivation with reparameterization, as follows:

[0025]

[0026] Where z0 = z;

[0027] S23. During the reverse denoising process, DM samples a Gaussian random noise z. T Then gradually z T Denoising back to z0 is as follows:

[0028]

[0029] in,

[0030] S24. Use formula Replacing z0 yields:

[0031]

[0032] Remove The variance estimate in the middle is obtained as follows:

[0033]

[0034] Among them, noise ε is the only uncertain variable.

[0035] Further, step S3 specifically includes:

[0036]

[0037] Among them, f i ej This represents the multi-scale forward coding features encoded by the encoder. x represents the multi-scale backward coding feature encoded by the encoder. i Represents downsampling features, The multi-scale diffusion prior features generated by the diffusion network are represented by j = 1, 2, 3, which represent three different scales. This represents the encoding / decoding features obtained in the previous step during backpropagation. This represents the encoding / decoding features obtained in the previous step of the forward propagation; the cross-attention principle involved in the hierarchical integration module is shown in the following equation:

[0038]

[0039] in, T represents the scaling factor and transpose operation; Q is obtained by mapping encoded or decoded features of different scales through a linear layer; K and V are obtained by mapping diffusion priors of different scales through a linear layer.

[0040] Further, step S4 specifically includes:

[0041]

[0042] Where f on the left side of the formula i e3 This represents the third-scale forward encoded features enhanced by the MemoryDecoder, as shown on the left side of the equation. This represents the third-scale backward encoded feature enhanced by the MemoryDecoder; f on the right-hand side of the formula i e3 The third-scale forward feature encoded by the encoder is represented on the right side of the formula. This represents the third-scale backward feature encoded by the encoder. This represents the forward defuzzification memory features retrieved from memory. This represents the backward defuzzification memory features retrieved from memory. This indicates the maintenance of forward transient perceptual memory. This indicates the maintenance of backward transient perceptual memory.

[0043] Furthermore, in the memory retrieval process in step S4, the query key is constructed using the encoded features of the last scale output by the encoder, and key projection is further used to reduce the channel dimension to reduce computational overhead. Then, the query key is regarded as query q, and the memory features m retrieved from the memory can be calculated through the readout operation, as shown in the following formula:

[0044] m = vW(k,q)

[0045] Here, k and v are memory key-value pairs stored in the memory bank; W(k,q) is an affinity matrix of size N×HW, representing the read operation controlled by the key and query; the read operation maps each query element to all memory elements and aggregates their values ​​v accordingly.

[0046] Further, step S5 specifically includes:

[0047] During decoding, similar to encoding, the multi-scale diffused prior features are again fused into the decoder through a hierarchical ensemble module. Furthermore, the transient perceptual memory maintained in the memory bank is updated using a gated recurrent unit (GRU) mechanism, involving the following formulas:

[0048]

[0049] Among them, f i ej and f represents the encoded features enhanced by the memory decoder. i dj This represents the multi-scale forward decoding features of the decoder. The multi-scale backward decoding features of the decoder are represented; the cross-attention principle involved in the hierarchical ensemble module is shown in the following equation:

[0050]

[0051] in, T represents the scaling factor and transpose operation; Q is obtained by mapping encoded or decoded features of different scales through a linear layer; K and V are obtained by mapping diffusion priors of different scales through a linear layer.

[0052] Further, step S6 specifically includes:

[0053] A discriminative feature fusion module is designed, which preprocesses relevant features before the cyclic feature extraction module. During backpropagation, the enhanced decoded features are encoded by a value encoder to obtain the memory value of the back memory, while the previous query key is reused as the memory key of the back memory. The formulas involved are as follows:

[0054]

[0055] Where, x i Representing the downsampling feature, f on the right side of the formula i d1 This represents the first-scale forward decoding feature of the decoder, as shown on the right side of the formula. This represents the first-scale backward decoding feature of the decoder; f on the left side of the formula i d1 and These all represent the enhanced decoding features obtained. This represents the features extracted by the previous loop feature extraction module during forward propagation. The first part represents the features extracted by the previous loop feature extraction module in the backpropagation; the second part represents the distinguishable feature fusion module; the third part represents the loop feature extraction module, which has a loop structure and the currently extracted features depend on the previously extracted features.

[0056] Further, step S7 specifically includes:

[0057] During the forward propagation process, the final deblurred features are encoded by the Value Encoder to obtain the memory value of the forward memory. The previous query key is also reused as the memory key of the forward memory. The formulas involved are as follows:

[0058]

[0059] Among them, I i and R i They represent the fuzzy input frames I respectively Blur And the finally recovered deblurred frame I DB x i f represents the downsampling feature. i ej This represents the multi-scale forward coding features encoded by the early encoder. This represents the multi-scale backward coding features encoded by the earlier encoder; f i dj This represents the enhanced multi-scale forward decoding features. This represents the enhanced multi-scale backward decoding features; U represents the upsampling module, and TFR represents the target frame reconstruction module.

[0060] Further, step S8 specifically includes:

[0061] The entire network is jointly trained using L1 Charbonnier loss and L1 loss during the diffusion process, as shown in the following equation:

[0062]

[0063] L all =L char +L diff

[0064] In the Charbonnier loss, R t G represents the recovered deblurred frame. t This represents the corresponding real, sharp frame; ε is a constant used for stable training; in the diffusion loss, denoted as the prior features predicted by the diffusion network, and z represents the original prior features obtained by concatenating the fuzzy input frame and the corresponding Ground Truth frame and encoding them by the latent encoder.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] 1. The present invention provides a video deblurring method based on memory diffusion network, which improves the effect of existing regression models on video deblurring by combining diffusion priors generated by diffusion network.

[0067] 2. The present invention provides a video deblurring method based on memory diffusion network, which constructs a concise and effective memory bank to filter and store time frame information to accurately capture complex spatiotemporal local relationships in blurred videos to achieve finer-grained deblurring. It also improves upon existing memory network-based work by designing a specific memory update algorithm to refresh the memory bank, thereby better utilizing the spatiotemporal information between consecutive video frames and ensuring the temporal consistency between frames.

[0068] Based on the above reasons, this invention can be widely applied in fields such as image processing. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart of the method of the present invention.

[0071] Figure 2 The memory provided for embodiments of the present invention.

[0072] Figure 3 The perceptual memory provided in the embodiments of the present invention.

[0073] Figure 4 The distinguishable feature fusion module provided in the embodiments of the present invention. Detailed Implementation

[0074] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0075] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0076] This invention proposes a video deblurring method based on a memory diffusion network, further exploring the potential of combining existing regression models with diffusion priors generated by diffusion networks for video deblurring. Figure 1 As shown, its core consists of two branches: a deblurring branch and a diffusion branch. The deblurring branch has a bidirectional recurrent structure and maintains a concise and efficient memory bank with a specific update mechanism, which can effectively utilize the spatiotemporal relationships between video frames to further ensure the temporal consistency between frames in the final deblurred video. The diffusion branch contains a lightweight diffusion network to generate multi-scale diffusion prior features.

[0077] The overall process is as follows:

[0078] S1. Select a publicly available video deblurring dataset as the ground truth, preprocess the input blurred video frames using Residual Dense Blocks (RDB), and downsample to obtain the initial downsampled features x. i ;

[0079] S2. The blurred input frame and the corresponding Ground Truth frame are concatenated together, encoded by a Latent Encoder, and then subjected to a forward diffusion noise generation process to generate random noise. This noise is then reversed by a diffusion denoising network to generate multi-scale diffusion prior features.

[0080] S3. The downsampled feature x obtained in step S1 i The features are sent to the feature encoder for encoding to obtain multi-scale encoded features; during encoding, the multi-scale diffused prior features generated in step S2 are... It is fused into the encoder via a Hierarchical Integration Module (HIM);

[0081] S4. Before decoding, a query key is constructed using the multi-scale coding features obtained in step S3 to generate a query for retrieving memory features from the memory bank. The retrieved memory features, the maintained transient perceptual memory, and the third-scale coding features are then sent to the memory decoder to enhance the coding features from the previous stage.

[0082] S5. Input the enhanced encoded features from step S4 into the feature decoder to decode and obtain multi-scale decoded features;

[0083] S6. Input the obtained first-scale decoding features, the features extracted by the previous loop feature extraction module, and the initial downsampled features into the loop feature extraction module (RFE) to further enhance the decoding features;

[0084] S7. Input the previously encoded features, enhanced decoding features and the initial downsampling features into the target frame reconstruction module (TFR) to reconstruct the target frame. Then, after transpose convolution upsampling, it is fused with the original blurred input to obtain the final deblurring result.

[0085] S8. Use L1 Charbonnier loss and L1 loss during the diffusion process to jointly train the entire network.

[0086] In a specific implementation, as a preferred embodiment of the present invention, in step S1, the initial downsampling feature x is obtained. i ,as follows:

[0087] x i =RDB(I i ), i = 1, 2, ..., N

[0088] Where, x i I represents the downsampling feature. i Indicates fuzzy input frame I Blur RDB represents the residual dense block used for downsampling.

[0089] In a specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0090] S21. During the forward diffusion process, z is transformed into Gaussian noise through T iterations, as shown in the following equation:

[0091]

[0092] Where t = 1, 2, ..., T; β t ∈(0,1) represents the hyperparameter used to control the noise variance; N represents the Gaussian distribution; I represents the standard normal distribution;

[0093] S22, Let α t =1-β t , The formula in step S21 can be rewritten through iterative derivation with reparameterization, as follows:

[0094]

[0095] Where z0 = z;

[0096] S23. During the reverse denoising process, DM samples a Gaussian random noise z. T Then gradually z T Denoising back to z0 is as follows:

[0097]

[0098] in,

[0099] S24. Use formula Replacing z0 yields:

[0100]

[0101] Remove The variance estimate in the middle is obtained as follows:

[0102]

[0103] Among them, noise ε is the only uncertain variable.

[0104] Following previous work, this embodiment uses a neural network to calculate the value of z. t The noise ε is predicted under the condition of c,t. Furthermore, following the general flow of the diffusion process, during test inference, randomly generated noise is directly input into the diffusion denoising network to generate multi-scale diffusion prior features. In this embodiment, as... Figure 1 As shown in (a), since the inherent randomness of the diffusion model may impair spatial fidelity and even generate some additional visual content, the corresponding fuzzy input frame is encoded by the latent encoder and then used as condition c to be input into the diffusion denoising network to constrain the generated prior features.

[0105] In a specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:

[0106]

[0107] Among them, f i ej This represents the multi-scale forward coding features encoded by the encoder. x represents the multi-scale backward coding feature encoded by the encoder. i Represents downsampling features, The multi-scale diffusion prior features generated by the diffusion network are represented by j = 1, 2, 3, which represent three different scales. This represents the encoding / decoding features obtained in the previous step during backpropagation. This represents the encoding / decoding features obtained in the previous step of the forward propagation; the cross-attention principle involved in the hierarchical integration module is shown in the following equation:

[0108]

[0109] in, T represents the scaling factor and transpose operation; Q is obtained by mapping encoded or decoded features of different scales through a linear layer; K and V are obtained by mapping diffusion priors of different scales through a linear layer.

[0110] In a specific implementation, as a preferred embodiment of the present invention, step S4 is as follows: Figure 2 As shown, before decoding, the obtained encoded features are used to construct the query key to generate a query for memory feature retrieval from the memory bank. Then, the retrieved memory features, the maintained transient perceptual memory, and the third-scale encoded features are sent together to the memory decoder to enhance the earlier encoded features. The formulas involved are as follows:

[0111]

[0112] Where f on the left side of the formula i e3 This represents the third-scale forward encoded features enhanced by the MemoryDecoder, as shown on the left side of the equation. This represents the third-scale backward encoded feature enhanced by the MemoryDecoder; f on the right-hand side of the formula i e3 The third-scale forward feature encoded by the encoder is represented on the right side of the formula. This represents the third-scale backward feature encoded by the encoder. This represents the forward defuzzification memory features retrieved from memory. This represents the backward defuzzification memory features retrieved from memory. This indicates the maintenance of forward transient perceptual memory. This represents the maintained backward transient perceptual memory. In this embodiment, the memory decoder consists of residual blocks and CBAM blocks.

[0113] In a specific implementation, as a preferred embodiment of the present invention, during the memory retrieval process in step S4, the query key is constructed using the encoded features of the last scale output by the encoder, and further key projection (which consists of a convolution) is used to reduce the channel dimension to reduce computational overhead. Then, the query key is regarded as query q, and the memory features m retrieved from the memory can be calculated through the readout operation, as shown in the following formula:

[0114] m = vW(k,q)

[0115] Here, k and v are memory key-value pairs stored in the memory bank; W(k,q) is an affinity matrix of size N×HW, representing the read operation controlled by the key and query; the read operation maps each query element to all memory elements and aggregates their values ​​v accordingly. In fact, this entire operation can be viewed as a spatiotemporal attention mechanism, and W can be seen as an attention graph, because we are searching the memory frames for the positions most relevant to retrieving the current query position to utilize their additional useful information.

[0116] In a specific implementation, as a preferred embodiment of the present invention, step S5 specifically includes:

[0117] During decoding, similar to encoding, the multi-scale diffused prior features are again fused into the decoder via a hierarchical ensemble module (HIM). Furthermore, the transient perceptual memory maintained in the memory bank is updated using a gated recurrent unit (GRU) mechanism, such as... Figure 3 As shown (HIM blocks are omitted for simplicity), the formulas involved are as follows:

[0118]

[0119] Among them, f i ej and f represents the encoded features enhanced by the memory decoder. i dj This represents the multi-scale forward decoding features of the decoder. The multi-scale backward decoding features of the decoder are represented; the cross-attention principle involved in the hierarchical ensemble module is shown in the following equation:

[0120]

[0121] in, T represents the scaling factor and transpose operation; Q is obtained by mapping encoded or decoded features of different scales through a linear layer; K and V are obtained by mapping diffusion priors of different scales through a linear layer.

[0122] In a specific implementation, as a preferred embodiment of the present invention, step S6 specifically includes:

[0123] The Discriminative Feature Fusion (DFF) module is designed to preprocess relevant features before the Recurrent Feature Extraction (RFE) module. During backpropagation, the enhanced decoded features are encoded by a Value Encoder to obtain the memory value of the back memory, while the previous lookup key is reused as the memory key of the back memory. The formulas involved are as follows:

[0124]

[0125] Where, x i Representing the downsampling feature, f on the right side of the formula i d1 This represents the first-scale forward decoding feature of the decoder, as shown on the right side of the formula. This represents the first-scale backward decoding feature of the decoder; f on the left side of the formula i d1 and These all represent the enhanced decoding features obtained. This represents the features extracted by the previous loop feature extraction module during forward propagation. The first part represents the features extracted by the previous loop feature extraction module during backpropagation; the second part represents the distinguishable feature fusion module; and the third part represents the loop feature extraction module, which has a loop structure where the currently extracted features depend on the previously extracted features. Furthermore, this embodiment also includes the previously downsampled features x. i-1 / x i+1 The input is fed into the Recurrent Feature Extraction (RFE) module, as this has proven effective. In this embodiment, the RFE module consists of a series of Channel Attention Blocks (CABs). The structure of the Discriminative Feature Fusion (DFF) module is as follows: Figure 4 As shown, its core lies in the further introduction of gating attention mechanism and dynamic filter to enhance feature fusion.

[0126] In a specific implementation, as a preferred embodiment of the present invention, step S7 specifically includes:

[0127] During the forward propagation process, the final deblurred features are encoded by the Value Encoder to obtain the memory value of the forward memory. The previous query key is also reused as the memory key of the forward memory. The formulas involved are as follows:

[0128]

[0129] Among them, Ii and R i They represent the fuzzy input frames I respectively Blur And the finally recovered deblurred frame I DB x i f represents the downsampling feature. i ej This represents the multi-scale forward coding features encoded by the early encoder. This represents the multi-scale backward coding features encoded by the earlier encoder; f i dj This represents the enhanced multi-scale forward decoding features. The symbol represents the enhanced multi-scale backward decoding feature; U represents the upsampling module, and TFR represents the target frame reconstruction module. In this embodiment, the target frame reconstruction module TFR consists of a series of channel attention blocks (CABs). The value encoder mainly consists of residual blocks and the first layer of a pre-trained ResNet50 network for memorizing value encoding.

[0130] In a preferred embodiment of this invention, the memory key-value pairs obtained in steps S6 and S7 are appended to the memory bank. For simplicity, this embodiment simply concatenates them together. Simultaneously, to construct a streamlined and efficient memory bank, and to avoid the ineffective utilization of memory information caused by simply resetting and re-memorizing during training (discarding all previous memory frames) as in existing works, this embodiment explores a better memory forgetting mechanism. To handle longer videos, inspired by the least-frequently-used (LFU) algorithm, this embodiment selects high-usage frames for retention. Specifically, this embodiment introduces the concept of "usage," which is derived row-by-row from the affinity matrix W and normalized by the duration of each memory frame in the memory bank. Finally, the top-k filters remove the k low-usage frames, which to some extent ensures a balance between training and inference, as a consistent strategy is employed.

[0131] In a specific implementation, as a preferred embodiment of the present invention, step S8 specifically includes:

[0132] The entire network is jointly trained using L1 Charbonnier loss and L1 loss during the diffusion process, as shown in the following equation:

[0133]

[0134] L all =L char +L diff

[0135] In the Charbonnier loss, Rt G represents the recovered deblurred frame. t This represents the corresponding real, sharp frame; ε is a constant used for stable training; in the diffusion loss, denoted as the prior features predicted by the diffusion network, while z represents the original prior features obtained by concatenating the fuzzy input frame and the corresponding Ground Truth frame and encoding them by the Latent Encoder.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video deblurring method based on memory diffusion networks, characterized in that, include: S1. Select a publicly available video deblurring dataset as the ground truth, preprocess the input blurred video frames using residual dense blocks, and downsample to obtain the initial downsampled features x. i ; S2. The blurred input frame and the corresponding Ground Truth frame are concatenated together, encoded by a latent encoder, and then subjected to a forward diffusion noise generation process to generate random noise. This noise is then reversed by a diffusion denoising network to generate multi-scale diffusion prior features. S3. The downsampled feature x obtained in step S1 i The features are fed into the feature encoder for encoding to obtain multi-scale encoded features; during encoding, the multi-scale diffusion prior features generated in step S2 are... It is integrated into the encoder through a layered integration module; S4. Before decoding, the multi-scale coding features obtained in step S3 are used to construct a query key to generate a query for memory feature retrieval in the memory bank. The retrieved memory features, the maintained transient perceptual memory, and the third-scale coding features are sent to the memory decoder to enhance the coding features in the early stage. S5. Input the enhanced encoded features from step S4 into the feature decoder for decoding to obtain multi-scale decoded features; S6. Input the obtained first-scale decoding features, the features extracted by the previous loop feature extraction module, and the initial downsampled features into the loop feature extraction module to further enhance the decoding features; S7. Input the previously encoded features, enhanced decoding features and the initial downsampling features into the target frame reconstruction module to reconstruct the target frame. Then, after transpose convolution upsampling, it is fused with the original blurred input to obtain the final deblurring result. S8. Use L1 Charbonnier loss and L1 loss during the diffusion process to jointly train the entire network.

2. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, In step S1, the initial downsampled feature x is obtained. i ,as follows: x i =RDB(I i ),i=1,2,...,N Where, x i I represents the downsampling feature. i Indicates fuzzy input frame I Blur RDB represents the residual dense block used for downsampling.

3. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, Step S2 specifically includes: S21. During the forward diffusion process, z is transformed into Gaussian noise through T iterations, as shown in the following equation: Where t = 1, 2, ..., T; β t ∈(0,1) represents the hyperparameter used to control the noise variance; N represents the Gaussian distribution; I represents the standard normal distribution; S22, Let α t =1-β t , The formula in step S21 can be rewritten through iterative derivation with reparameterization, as follows: Where z0 = z; S23. During the reverse denoising process, DM samples a Gaussian random noise z. T Then gradually z T Denoising back to z0 is as follows: in, S24. Use formula Replacing z0 yields: Remove The variance estimate in the middle is obtained as follows: Among them, noise ε is the only uncertain variable.

4. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, Step S3 specifically includes: Among them, f i ej This represents the multi-scale forward coding features encoded by the encoder. x represents the multi-scale backward coding feature encoded by the encoder. i Represents downsampling features, The multi-scale diffusion prior features generated by the diffusion network are represented by j = 1, 2, 3, which represent three different scales. This represents the encoding / decoding features obtained in the previous step during backpropagation. This represents the encoding / decoding features obtained in the previous step of the forward propagation; the cross-attention principle involved in the hierarchical integration module is shown in the following equation: in, T represents the scaling factor and transpose operation; Q is obtained by mapping encoded or decoded features of different scales through a linear layer; K and V are obtained by mapping diffusion priors of different scales through a linear layer.

5. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, Step S4 specifically includes: Where f on the left side of the formula i e3 This represents the third-scale forward encoded features enhanced by the MemoryDecoder, as shown on the left side of the equation. This represents the third-scale backward encoded feature enhanced by the MemoryDecoder; f on the right-hand side of the formula i e3 The third-scale forward feature encoded by the encoder is represented on the right side of the formula. This represents the third-scale backward feature encoded by the encoder. This represents the forward defuzzification memory features retrieved from memory. This represents the backward defuzzification memory features retrieved from memory. This indicates the maintenance of forward transient perceptual memory. This indicates the maintenance of backward transient perceptual memory.

6. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, In the memory retrieval process in step S4, the query key is constructed using the encoded features of the last scale output by the encoder, and key projection is further used to reduce the channel dimension to reduce computational overhead. Then, the query key is regarded as query q, and the memory features m retrieved from the memory can be calculated through the readout operation, as shown in the following formula: m = vW(k,q) Here, k and v are memory key-value pairs stored in the memory bank; W(k,q) is an affinity matrix of size N×HW, representing the read operation controlled by the key and query; the read operation maps each query element to all memory elements and aggregates their values ​​v accordingly.

7. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, Step S5 specifically includes: During decoding, similar to encoding, the multi-scale diffused prior features are again fused into the decoder through a hierarchical ensemble module. Furthermore, the transient perceptual memory maintained in the memory bank is updated using a gated recurrent unit (GRU) mechanism, involving the following formulas: Among them, f i ej and f represents the encoded features enhanced by the memory decoder. i dj This represents the multi-scale forward decoding features of the decoder. The multi-scale backward decoding features of the decoder are represented; the cross-attention principle involved in the hierarchical ensemble module is shown in the following equation: in, T represents the scaling factor and transpose operation; Q is obtained by mapping encoded or decoded features of different scales through a linear layer; K and V are obtained by mapping diffusion priors of different scales through a linear layer.

8. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, Step S6 specifically includes: A discriminative feature fusion module is designed, which preprocesses relevant features before the cyclic feature extraction module. During backpropagation, the enhanced decoded features are encoded by a value encoder to obtain the memory value of the back memory, while the previous query key is reused as the memory key of the back memory. The formulas involved are as follows: Where, x i Representing the downsampling feature, f on the right side of the formula i d1 This represents the first-scale forward decoding feature of the decoder, as shown on the right side of the formula. This represents the first-scale backward decoding feature of the decoder; f on the left side of the formula i d1 and These all represent the enhanced decoding features obtained. This represents the features extracted by the previous loop feature extraction module during forward propagation. The first part represents the features extracted by the previous loop feature extraction module in the backpropagation; the second part represents the distinguishable feature fusion module; the third part represents the loop feature extraction module, which has a loop structure and the currently extracted features depend on the previously extracted features.

9. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, Step S7 specifically includes: During the forward propagation process, the final deblurred features are encoded by the Value Encoder to obtain the memory value of the forward memory. The previous query key is also reused as the memory key of the forward memory. The formulas involved are as follows: Among them, I i and R i They represent the fuzzy input frames I respectively Blur And the finally recovered deblurred frame I DB x i f represents the downsampling feature. i ej This represents the multi-scale forward coding features encoded by the early encoder. This represents the multi-scale backward coding features encoded by the earlier encoder; f i dj This represents the enhanced multi-scale forward decoding features. This represents the enhanced multi-scale backward decoding features; U represents the upsampling module, and TFR represents the target frame reconstruction module.

10. The video deblurring method based on memory diffusion networks according to claim 1, characterized in that, Step S8 specifically includes: The entire network is jointly trained using L1 Charbonnier loss and L1 loss during the diffusion process, as shown in the following equation: L all =L char +L diff In the Charbonnier loss, R t G represents the recovered deblurred frame. t This represents the corresponding real, sharp frame; ε is a constant used for stable training; in the diffusion loss, denoted as the prior features predicted by the diffusion network, and z represents the original prior features obtained by concatenating the fuzzy input frame and the corresponding Ground Truth frame and encoding them by the latent encoder.

Images