Video deblurring method based on spatio-temporal feature fusion and bidirectional recurrent neural network
By employing spatiotemporal feature fusion and bidirectional recurrent neural networks, the problem of insufficient utilization of inter-frame dependencies in video deblurring is solved, achieving efficient video clarity enhancement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUILIN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2023-05-22
- Publication Date
- 2026-04-24
AI Technical Summary
Existing video deblurring methods fail to effectively utilize the temporal dependencies between adjacent frames and the current frame, and cannot properly weight spatial features during extraction, resulting in decreased information transmission accuracy.
A method based on spatiotemporal feature fusion and bidirectional recurrent neural networks is adopted. Features are extracted through forward and backward units, and gradient propagation is performed using RNN units with dense attention blocks. Combined with the spatiotemporal feature fusion reconstruction module, the temporal dependencies between frames are captured and information is reconstructed.
It achieves efficient information fusion and reconstruction between video frames, generating high-fidelity, high-quality clear video sequences.
Smart Images
Figure CN116612029B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and more specifically to a video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural networks. Background Technology
[0002] With the widespread use of high-speed motion photography, video image blurring caused by factors such as slow shutter speeds, motion blur, and camera shake is becoming increasingly common, directly impacting the user's visual experience. Simultaneously, with the continuous development of computer vision technology, the demand for clear images is also increasing. To improve visual quality and meet user needs, video deblurring has become an important visual task, aiming to reconstruct clear image sequences from low-quality, blurry video sequences.
[0003] With the continuous development of deep learning technology, deep learning-based video deblurring methods are constantly emerging. Compared with traditional methods, deep learning-based methods offer significant improvements in processing performance and speed. These methods primarily utilize deep neural networks to learn the mapping relationship between blurred and sharp images from image data, thereby achieving deblurring. Among them, recurrent neural networks (RNNs) serve as the main framework for solving video deblurring problems because RNNs can capture temporal dependencies and model the temporal features between video frames. Simultaneously, training multi-layer networks through backpropagation improves the model's robustness and accuracy. The main idea behind using RNNs for video deblurring is to construct the network structure by temporally concatenating features from different times. These real-time input sequences are fed into the network over time, allowing the entire network to track the changes in this sequence, thus achieving video deblurring. However, current methods fail to establish temporal dependencies between features when fusing adjacent and current frames, resulting in incomplete utilization of information from adjacent frames. Furthermore, because the model cannot perform appropriate weighting during the spatial feature extraction stage, it cannot learn the dynamic weights between different frames, leading to a decrease in accuracy when transferring information in the temporal domain. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention aims to provide a video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural networks.
[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems includes the following steps:
[0006] Step S1, Data preprocessing: Divide the input blurred video into a sequence with the current frame and the four frames before and after it as the basic unit, and use this as input into the video deblurring model based on spatiotemporal feature fusion and bidirectional recurrent neural network.
[0007] Step S2, constructing a video deblurring model: including a forward unit, a backward unit, and a spatiotemporal feature fusion and reconstruction module; the forward unit and the backward unit propagate the feature information of the current frame backward and forward in time, respectively; the spatiotemporal feature fusion and reconstruction module uses the features of the previous and next frames to effectively fuse the features with the current frame in time and reconstruct the model.
[0008] Step S3, train the video deblurring model: calculate the L1 loss based on the input blurred video and its corresponding clear video image and perform backpropagation to update the model parameters;
[0009] Step S4, Reconstruct Clear Video: Input the blurry video into the trained deblurring model, output and save the generated clear video.
[0010] Furthermore, step S2 specifically includes the following steps:
[0011] The input basic unit sequence is input into the forward unit to extract image features, thereby obtaining forward latent features at different times;
[0012] The forward latent features are input into the backward unit to continue extracting fuzzy features, thereby obtaining backward latent features at different times;
[0013] The spatiotemporal feature fusion and reconstruction module uses the backward latent feature input values from different times to reconstruct the current frame, and repeats this process to obtain a clear video sequence.
[0014] Furthermore, in step S2, the forward and backward units are composed of RNN units based on dense attention blocks (DAB), specifically including:
[0015] In the forward unit, features are first extracted from the input fuzzy frame, then concatenated with the hidden state output by the previous forward unit, and then fuzzy features are extracted by multiple cascaded DABs. The output hidden state is propagated to the next forward unit, and the output forward latent features are input to the backward unit.
[0016] In the backward unit, the forward latent features are received as input for fuzzy feature extraction, and then concatenated with the hidden state output by the next backward unit to extract features. The output hidden state is propagated to the previous backward unit, and the output latent features are input into the spatiotemporal feature fusion and reconstruction module to reconstruct a clear image.
[0017] Furthermore, residual connections are added between multiple DABs. Specifically, each DAB includes a dense feature extraction module, a channel attention mechanism, and residual connections. The dense feature extraction module extracts features using four convolutional groups and adds dense connections between different groups. The convolutional group includes a 3x3 convolutional layer, a ReLU activation function, a 3x3 dilated convolution, and a ReLU activation function. The channel attention block includes a global average pooling operation, a 1x1 convolutional layer, a ReLU activation function, a 1x1 convolutional layer, and a non-linear activation function sigmoid, which yields a weighted attention map and multiplies it with the extracted features to obtain weighted features. The residual connection refers to adding the input features to the weighted features.
[0018] Furthermore, in step S2, the spatiotemporal feature fusion and reconstruction module specifically includes:
[0019] The four potential features of the previous and next frames and the potential features of the current frame of the backward unit are linearly embedded through a 1x1 convolutional layer;
[0020] The embedded features are multiplied together and then passed through the softmax activation function to obtain the mapping features between the current frame and the previous and next frames.
[0021] The fused feature is obtained by averaging the four mapping features and multiplying them by the current frame features.
[0022] The fused features are passed through a 1x1 convolutional layer and added to the current frame to obtain the final reconstructed frame.
[0023] Furthermore, in step S3, the L1 loss between the reconstructed, deblurred video and the real, clear video is used as the loss function to train the model and update the model parameters.
[0024] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0025] (1) The video deblurring network designed in this invention based on spatiotemporal feature fusion and bidirectional recurrent neural network adopts forward connection and backward connection methods to effectively utilize bidirectional information for frame reconstruction.
[0026] (2) The RNN unit based on dense attention blocks designed in this invention uses dense connections and residual connections to enhance gradient propagation. Adding dilated convolutions in feature extraction expands the network's receptive field without increasing the number of parameters, capturing richer contextual information. Simultaneously, the added channel attention mechanism allows the network to focus on important high-frequency edge information.
[0027] (3) The spatiotemporal feature fusion reconstruction module designed in this invention can capture the temporal domain dependency between adjacent frames and the current frame, and fuse effective information to reconstruct the current frame. Attached Figure Description
[0028] Figure 1 This is a flowchart of the video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural network described in an embodiment of the present invention.
[0029] Figure 2 This is a diagram of the video deblurring network structure based on spatiotemporal feature fusion and bidirectional recurrent neural network as described in an embodiment of the present invention.
[0030] Figure 3 This is a diagram of the RNN unit framework based on dense attention blocks as described in an embodiment of the present invention.
[0031] Figure 4 This is a framework diagram of the spatiotemporal feature fusion and reconstruction module described in an embodiment of the present invention. Detailed Implementation
[0032] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0033] Example
[0034] like Figure 1 As shown, the video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural networks includes the following steps:
[0035] Step S1, Data Preprocessing: The input blurred video is divided into a sequence with the current frame and the four frames before and after it as the basic unit, and this sequence is used as input into a video deblurring model based on spatiotemporal feature fusion and bidirectional recurrent neural network.
[0036] Specifically, the basic unit of input can be represented as [I t-2 I t-1 I t I t+1 I t-2 ], where t represents different times.
[0037] Step S2, construct the video deblurring model: including forward units, backward units, and a spatiotemporal feature fusion and reconstruction module;
[0038] Specifically, the input basic unit sequence [I] t-2 I t-1 I t I t+1 I t-2 The image features are extracted from the input to the forward unit to obtain the forward latent features [f′] at different times. t-2 f′ t-1 f′ t f′ t+1 f′ t+2 ].
[0039] Specifically, the forward latent feature [f′] t-2 f′ t-1 f′ t f′ t+1 f′ t+2 The input is fed into the backward unit to continue extracting fuzzy features, resulting in backward latent features at different times [f]. t-2 f t-1 f t f t+1 f t+2 ].
[0040] Specifically, the spatiotemporal feature fusion and reconstruction module uses the backward latent feature input values from different times to perform the current frame O t Reconstruct, and repeat this process to obtain a clear video sequence.
[0041] (S21) The forward unit will move the current frame I t The characteristic information propagates backward in time.
[0042] The specific process is as follows: First, from the input fuzzy frame I... t Features are extracted from the hidden state h′ output by the previous feedforward unit. t-1 The components are concatenated, and then multiple cascaded DABs are used to extract fuzzy features f′. t The specific formula can be expressed as:
[0043] f′ t =F DABs (Concat[F Conv3 (I t ), h′ t-1 ])
[0044] Among them, F DABs (·) represents multiple cascaded DAB modules, F Conv3 (·) represents a 3x3 convolutional layer, and Concat represents the concatenation operation. The final output is the hidden state h′. t The latent feature f is propagated to the next feedforward unit. t The input to the backward unit can be expressed as follows:
[0045] h′ t =F Conv3 (F DAB (F Conv3 (f t ′)))
[0046] Where F DAB (·) represents a DAB module.
[0047] (S22) The backward unit will move the current frame I t The characteristic information propagates forward in time.
[0048] The specific process is as follows: First, the latent feature f output by the feedforward unit is received. t The hidden state h is used as input for fuzzy feature extraction, and then compared with the hidden state h output by the next backward unit. t+1 Concatenate and extract features, outputting the hidden state h. t The latent feature f is propagated to the previous backward unit and output. t Input into the global fusion reconstruction module to reconstruct a clear image.
[0049] More specifically, such as Figure 3 As shown, the DAB module includes a dense feature extraction module, a channel attention mechanism, and residual connections. The input features are first extracted through four convolutional groups. Each convolutional group consists of a 3x3 convolutional layer, a ReLU activation function, a 3x3 dilated convolution, and another ReLU activation function. Unlike traditional dense residual blocks, the added dilated convolution expands the network's receptive field without increasing the number of parameters, allowing it to capture richer contextual information. Next, a channel attention mechanism is applied, enabling the network to focus on important high-frequency edge information. The channel attention block first uses global average pooling to obtain global channel statistics, then uses a 1x1 convolutional layer, a ReLU activation function, another convolutional layer, and a non-linear activation function (sigmoid) to obtain a weighted attention map. Finally, the weighted attention map is multiplied by the extracted features to obtain the weighted features. The input features and weighted features are then added together as the output and fed into the next DAB for fuzzy feature extraction.
[0050] (S23) The spatiotemporal feature fusion and reconstruction module uses the features of the previous and next frames to perform effective temporal feature fusion and reconstruction with the current frame.
[0051] The specific process is as follows: First, the four potential features [f] obtained by the backward unit at adjacent time points are processed... t-2 f t-1 f t+1 f t-2 ] and the potential features f of the current frame t After passing through a 1x1 convolutional layer, a linear embedding is obtained, which can be expressed by the following formula:
[0052] g t-2 g t-1 g t+1 g t+2 =F Conv1 (f t-2 f t-1 f t+1 f t-2)
[0053] g t =F Conv1 (f y )
[0054] Among them, F Conv (·) represents a 1x1 convolutional layer. The linearly embedded features are then multiplied and passed through a softmax activation function to obtain the mapping features between the current frame and the preceding and following frames [M]. 1 M 2 M 3 M 4 Each mapping feature represents the similarity relationship between the current frame and its neighboring frames, specifically expressed as:
[0055]
[0056] in, The dot product operation is represented by Softmax(·), which represents the normalized exponential function. Then, the four mapped features are averaged and multiplied by the current frame to obtain a hybrid feature M that fuses the similarity relationships between the current frame and the preceding and following frames. t The specific formula can be expressed as:
[0057]
[0058] Where AVG(·) represents the arithmetic mean operation. Finally, the mixed features are passed through a 1x1 convolutional layer and added to the current frame to obtain the final reconstructed frame. The specific formula can be expressed as:
[0059] O t =f t +F Conv1 (M t ).
[0060] Step S3, train the video deblurring model: Set the model hyperparameters, train the network using the ADAM optimizer with an initial learning rate of 0.0001, and employ a cosine annealing decay strategy, reducing the learning rate by half every 200 training epochs. Calculate the L1 loss based on the input blurred video and its corresponding clear video image, and perform backpropagation to update the model parameters.
[0061] Step S4, Reconstruct Clear Video: Input the blurry video into the deblurring model, output and save the generated clear video.
[0062] In summary, the video deblurring network based on spatiotemporal feature fusion and bidirectional recurrent neural networks provided in this embodiment leverages the advantage of recurrent neural networks in capturing dependencies between time steps. It employs forward and backward connections to bidirectionally model the temporal series features between video frames, achieving a deblurring effect. The DAB-based RNN units ensure good extraction of spatial features from frame information, providing more accurate information in the temporal domain. Furthermore, the spatiotemporal fusion reconstruction module calculates the similarity between the current frame and different time points, fusing effective information to reconstruct the current frame. Therefore, this embodiment can generate high-fidelity and high-quality videos.
[0063] The implementation examples described above are merely preferred embodiments of the present invention and should not be considered as limiting the implementation methods and scope of protection of the present invention. Those skilled in the art can make various modifications and adjustments to adapt to changes and needs within the field, aiming to improve or enhance the performance, functionality, and effects of the present invention. Within the scope of the claims, any modifications, equivalent substitutions, and improvements that conform to the spirit and principles of the present invention are included within the scope of protection of the present invention.
Claims
1. A video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural networks, characterized in that, Specifically, the following steps are included: Step S1, Data preprocessing: Divide the input blurred video into a sequence with the current frame and the four frames before and after it as the basic unit, and use this as input into a video deblurring model based on spatiotemporal feature fusion and bidirectional recurrent neural network. Step S2, constructing a video deblurring model: including a forward unit, a backward unit, and a spatiotemporal feature fusion and reconstruction module; the forward unit and the backward unit propagate the feature information of the current frame backward and forward in time, respectively; the spatiotemporal feature fusion and reconstruction module uses the features of the previous and next frames to effectively fuse the features with the current frame in time and reconstruct the model. The forward and backward units are composed of RNN units based on dense attention blocks (DAB), specifically including: In the forward unit, features are first extracted from the input blurred frame, then concatenated with the hidden state output by the previous forward unit, and then multiple cascaded dense attention DABs extract blurred features. The output hidden state is propagated to the next forward unit, and the output forward latent features are input to the backward unit. In the backward unit, the forward latent features are received as input for fuzzy feature extraction, and then concatenated with the hidden state output by the next backward unit to extract features. The output hidden state is propagated to the previous backward unit, and the output latent features are input into the spatiotemporal feature fusion and reconstruction module to reconstruct a clear image. Residual connections are added between the multiple dense attention blocks (DABs). Specifically, each DAB includes a dense feature extraction module, a channel attention mechanism, and residual connections. The dense feature extraction module extracts features using four convolutional groups, with dense connections added between different groups. Each convolutional group includes a 3x3 convolutional layer, a ReLU activation function, a 3x3 dilated convolution, and another ReLU activation function. The channel attention mechanism includes a global average pooling operation, a 1x1 convolutional layer, a ReLU activation function, another 1x1 convolutional layer, and a sigmoid nonlinear activation function to obtain a weighted attention map, which is then multiplied by the extracted features to obtain weighted features. The residual connections refer to adding the input features to the weighted features. Step S3, train the video deblurring model: calculate the L1 loss based on the input blurred video and its corresponding clear video image and perform backpropagation to update the model parameters; Step S4, Reconstruct Clear Video: Input the blurry video into the trained deblurring model, output and save the generated clear video.
2. The video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural network according to claim 1, wherein step S2 specifically includes the following steps: The input basic unit sequence is input into the forward unit to extract image features, thereby obtaining forward latent features at different times; The forward latent features are input into the backward unit to continue extracting fuzzy features, thereby obtaining backward latent features at different times; The spatiotemporal feature fusion and reconstruction module uses the backward latent feature input values from different times to reconstruct the current frame, and repeats this process to obtain a clear video sequence.
3. The video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural network according to claim 1, characterized in that, In step S2, the spatiotemporal feature fusion and reconstruction module specifically includes: The four potential features of the previous and next frames and the potential features of the current frame of the backward unit are linearly embedded through a 1x1 convolutional layer; The embedded features are multiplied together and then passed through the softmax activation function to obtain the mapping features between the current frame and the previous and next frames. The fused feature is obtained by averaging the four mapping features and multiplying them by the current frame features. The fused features are passed through a 1x1 convolutional layer and added to the current frame to obtain the final reconstructed frame.
4. The video deblurring method based on spatiotemporal feature fusion and bidirectional recurrent neural network according to claim 1, characterized in that, In step S3, the L1 loss between the reconstructed, deblurred video and the real, clear video is used as the loss function to train the model and update the model parameters.