Dual-decoder image deblurring method, electronic equipment and storage medium

Abstract

The invention provides a double-decoder image deblurring method, relates to the technical field of image processing, solves the problem of high-quality deblurring recovery in the prior art, and adopts the scheme that an event image obtained by an event camera and a corresponding blurred image are input into a coding network, the characteristics of the event image and the blurred image are extracted, and the characteristics of the event image and the blurred image are extracted; obtaining output characteristics of the coding network; inputting the output features into a decoding network, reconstructing an image from the input features, realizing preliminary deblurring of the image, and obtaining a preliminary deblurred image; inputting the preliminary deblurred image into a feedback mechanism network, and reconstructing a high-quality image from the feature map to realize deblurring of the image; and measuring the difference between the high-quality image and the real clear image by using a loss function to obtain an optimal model parameter. According to the scheme, the feedback information from the high layer and the feature representation of the current layer are fused, and the network is guided to pay more attention to the useful image area, so that the image reconstruction quality is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a dual-decoder image deblurring method, electronic device, and storage medium. Background Technology

[0002] Image blurring is a common problem affecting image quality and can be caused by a variety of factors, including motion blur due to rapid camera or object movement, optical blur due to inaccurate focus or lens smudges, or blurring caused by excessive exposure time in low-light conditions. The goal of image deblurring is to restore a sharp image as much as possible, recovering the details lost due to blurring, thereby improving the image quality and its performance in downstream visual tasks.

[0003] In recent years, with the development of deep learning technology, many deep learning-based image deblurring methods have been proposed. To better solve the deblurring problem, modern deep learning-based methods choose to train convolutional neural networks on large-scale data to learn the implicit correlation between blurred and sharp images. Although these methods perform well on some existing datasets, they often fail when faced with severe motion blur or complex real-world scenes, and their performance is highly dependent on the type and diversity of blur in the training data. To improve the ability to model rapid dynamic changes, recent research has introduced event cameras as an auxiliary information source. Event cameras have the characteristics of high temporal resolution and no motion blur, providing neural networks with more accurate dynamic cues, thereby improving the model's generalization ability and restoration quality under complex dynamic blur conditions. Event cameras output event data asynchronously with low latency, without producing motion blur even under severe motion, and can record texture and motion information continuously.

[0004] Event camera deblurring has been proven to be effective in the field of deblurring, but existing methods usually use convolution or attention mechanisms to fuse features of two modalities directly. They still have limitations in modal co-fusion and reconstruction structure design, making it difficult to achieve high-quality deblurring recovery. Summary of the Invention

[0005] To address the problem of high-quality deblurring and restoration in existing technologies, the present invention aims to provide a dual-decoder image deblurring method, electronic device, and storage medium.

[0006] In a first aspect, according to some embodiments, the present invention provides a dual-decoder image deblurring method, comprising:

[0007] S1. The event image and the corresponding blurred image obtained by the event camera are input into the coding network, the features of the event image and the blurred image are extracted, and the event features are fused with the image features. After cross-modal feature fusion, the output features of the coding network are obtained.

[0008] S2. Input the output features into the decoding network, reconstruct the image from the input features, achieve preliminary deblurring of the image, and obtain a preliminary deblurred image;

[0009] S3. The preliminary deblurred image is input into the feedback mechanism network, and the feature map obtained by the feedback mechanism network is re-input into the decoding network in S2 to reconstruct a high-quality image from the feature map, thereby achieving image deblurring.

[0010] S4. The difference between the high-quality image and the real clear image is measured using the loss function, and the parameters are updated using a parameter update method with faster convergence speed to obtain the optimal model parameters.

[0011] In some embodiments, the coding network includes: two structurally identical coding sub-networks and a feature fusion network;

[0012] The encoding sub-network comprises, in sequence: a convolution with a kernel of 3, a dilated convolution residual module, a downsampling module, a dilated convolution residual module, a downsampling module, and a dilated convolution residual module;

[0013] The feature fusion network comprises, in sequence: a cross-modal feature fusion module, a feature fusion module based on local scoring, a cross-modal feature fusion module, a feature fusion module based on local scoring, a cross-modal feature fusion module, and a feature fusion module based on local scoring.

[0014] The process involves extracting features from the event image and the blurred image, fusing the event features with the image features, and then performing cross-modal feature fusion to obtain the output features of the coding network. Specifically, this includes:

[0015] The event image is input into the first encoder, and the blurred image is input into the second encoder. The outputs of the first dilated convolutional residual modules in the first and second encoders are the inputs of the first cross-modal feature fusion module. The outputs of the second dilated convolutional residual modules in the first and second encoders are the inputs of the second cross-modal feature fusion module. The outputs of the third dilated convolutional residual modules in the first and second encoders are the inputs of the third cross-modal feature fusion module. The outputs of the first dilated convolutional residual module and the first cross-modal feature fusion module in the second encoder are the inputs of the first feature fusion module based on local scoring. The output of the first feature fusion module based on local scoring is a first feature map F1, which, after downsampling, is combined with the output feature map of the second cross-modal feature fusion module. Figure 1 The input is fed into the second feature fusion module based on local scoring to obtain the second feature map F2; the feature map F2 is downsampled and then combined with the output feature map of the third cross-modal feature fusion module. Figure 1 The input is fed into the third feature fusion module based on local scoring to obtain the output features of the encoding network.

[0016] In some embodiments, the decoding network includes two structurally identical encoding sub-networks, namely a first decoder and a second decoder. Each encoding sub-network consists of 3 DCRM modules, 4 upsampling modules, and 1 convolution with a kernel of 3.

[0017] The step of inputting the output features into the decoding network and reconstructing the image from the input features to achieve preliminary deblurring of the image and obtain a preliminary deblurred image specifically includes:

[0018] The output features are processed by:

[0019] The first DCRM module in the first decoder obtains a first feature map T1. After upsampling, feature map T1 is added element-wise with feature map F2. The resulting feature is input to the second DCRM module in the first decoder. The output feature is upsampled and added element-wise with the first feature map F1. The added feature map is then input to the third DCRM module in the first decoder and a convolution with a kernel of 3 to obtain the decoder output. The input of the second decoder is the same as the output of the first decoder. The output feature map of the first DCRM module in the second decoder is the second feature map T2. The outputs of the first decoder and the second decoder are added element-wise to obtain a preliminary deblurred image.

[0020] In some embodiments, the feedback mechanism network includes four feedback mechanism modules, which are used to guide the network to pay more attention to useful image regions and improve image reconstruction quality by fusing feedback information from higher layers with feature representations of the current layer.

[0021] The step of re-inputting the feature map obtained from the feedback mechanism network into the decoding network in S2 to reconstruct a high-quality image from the feature map and achieve image deblurring specifically includes:

[0022] The initially deblurred image is downsampled to obtain the second image 2. The feature map T1 and the second image 2 are input into the first FAM module, and the output feature map is sequentially upsampled, element-wise added to the feature map F2, and then input into the second DCRM module in the first decoder to obtain feature map T3. Feature map T3 and the initially deblurred image obtained in step S2 are input into the second FAM module, and the resulting output feature map is sequentially upsampled, element-wise added to the first feature map F1 to obtain feature map T4. Feature map T4 is then processed by the third DCRM module in the first decoder and a convolution with a kernel of 3 to obtain image 3. After the second feature map T2 and the image 2 in step S2 are input into the third FAM module, the output feature map is sequentially upsampled, element-wise added to the feature map F2, and then input into the second DCRM module in the second decoder 2 to obtain the fifth feature map T5. The fifth feature map T5 and the preliminary deblurred image obtained in step S2 are input into the fourth FAM module, and the resulting output feature map is sequentially upsampled, element-wise added to the first feature map F1 to obtain the sixth feature map T6. The feature map T6 is then processed by the third DCRM module in the second decoder 2 and a convolution with a kernel of 3 to obtain the fourth image 4. The third image 3 and the fourth image are then processed together. Figure 4 The pixels are added together to obtain the final deblurred image.

[0023] In some embodiments, the dilated convolution residual module includes three branches. The input feature map of the dilated convolution residual module is first subjected to a convolution with kernel 1 in the first branch to obtain the A1 feature map. The input feature map of the dilated convolution residual module is first subjected to a convolution with kernel 3 in the second branch to obtain the A2 feature map. The input feature map of the dilated convolution residual module is then subjected to a convolution with kernel 5 and a DRRB module in the third branch to obtain the A3 feature map. The A3 feature map and the A2 feature map are element-wise added together, and then subjected to the DRRB module to obtain the A4 feature map. The A4 feature map is element-wise added together with the A3 and A1 feature maps, and then subjected to the DRRB module to obtain the A5 feature map. The A5 feature map, the A4 feature map, and the A3 feature map are combined. Figure 1 The input is fed into the cross-layer attention fusion module, and finally the output feature map of the dilated convolution residual module is obtained;

[0024] The DARB module is a residual structure module. The main network of the residual structure module includes four hybrid modules, a concatenation operation, a convolution with a kernel of 1, and an attention module. Each hybrid module includes a convolution with a kernel of 3 and a ReLU activation function. The output features of each hybrid module are concatenated and fused with the output features of each of the previous few hybrid modules. The fused features are used as the input features of a hybrid module.

[0025] The cross-layer attention fusion module in the dilated convolutional residual module contains spliced ​​features from three consecutive layers. By splicing the three features , , Remodeling Features of Dimensions A 1×1 convolution is used to aggregate information across the channel dimensions of each pixel location to extract pixel-level cross-channel contextual features; a 3×3 depthwise separable convolution is used to generate queries, keys, and values; the queries and keys are then reshaped into... and The The dimension is The Dimensions A two-dimensional matrix; calculated to obtain The interlayer correlation attention matrix A; the reshaped value matrix Multiply with the interlayer related attention matrix A and with the input features The features are added together to obtain the output features of the cross-layer attention fusion module; the specific processing steps of the cross-layer attention fusion module are shown in Formula 1, Formula 2, and Formula 3 below:

[0026] (1)

[0027] (2)

[0028] (3)

[0029] In the formula, This represents the Softmax function, which normalizes the inter-layer similarity matrix and is used to generate attention weights. It is a learnable scaling factor used to adjust the magnitude of the attention score. For convolution weights, Features after splicing It is the output characteristic of the module.

[0030] In some embodiments, the cross-modal feature fusion module includes two parts: a feature alignment unit and a multi-head attention mechanism.

[0031] In the feature alignment process, the cross-modal feature fusion module guides the features of one modality to participate in the change process of the features of another modality through convolution operations, thereby bridging the gaps in inconsistencies between modalities;

[0032] Image features and event image features are adjusted together using a combination of addition and multiplication to compensate for representational shifts caused by modal differences. The aligned features are then input into an attention module, where a multi-head attention module acts bidirectionally, allowing image features to be used as query vectors to extract key responses from event image features. The weights of the attention module are shared. The cross-modal feature fusion module is described in Equations 4, 5, and 6.

[0033] (4)

[0034] (5)

[0035] (6)

[0036] in and Aligned image features Event characteristics , This indicates a multi-head attention module. This indicates a channel-dimension concatenation operation. This is the final output of the cross-modal feature fusion module.

[0037] In some embodiments, the feature fusion module based on local scoring is used to integrate image features. Blending features after spatial alignment The input is passed to a 3×3 convolutional layer with ReLU activation, resulting in a layer of size [size missing]. Features;

[0038] For each pixel location, neighboring pixels are sampled from both image features and event features within a specified window to obtain... , representing the local region features sampled around each spatial location for image and event features, respectively; through calculation and The similarity is used to generate a matching score, as shown in Formula 7:

[0039] (7)

[0040] In the formula This represents the inner product operation. The scoring map contains similarity scores between sharp representations of both modalities, thereby suppressing redundant event responses in the deblurring task. Using this similarity score and the alignment features of each neighboring pixel, the features are weighted and aggregated, as shown in Equation 8:

[0041] (8)

[0042] In the formula, Indicates the location Nearby Spatially aligned event features obtained by sampling from each neighborhood. A similarity score is given between the image and event features at that neighborhood location. This refers to the Sigmoid function, which maps matching scores to the interval (0, 1). This represents the number of neighboring pixels at each location that participate in the weighted calculation; For the final position Aggregated features obtained by weighted aggregation;

[0043] The output of the feature fusion module based on local scoring is aggregated features. Spatial alignment event characteristics The result is obtained by adding the ingredients together, as shown in Formula Nine:

[0044] (9)

[0045] By comprehensively considering neighborhood information, the feature fusion module based on local scoring can obtain more reliable fused features.

[0046] In some embodiments, the feedback mechanism module includes:

[0047] Let the input feature map of the current layer be... The feedback image from the upper layer is ;

[0048] For the feedback image Apply a 3×3 convolution operation and use the Sigmoid activation function to generate an attention map. , representing the importance weight of the feedback feature at different locations;

[0049] Current input features Pass through a 3×3 convolutional layer and with attention map Perform element-wise multiplication to obtain the attention-modulated feature response and the input feature map. Add them together to get the final output. This completes the feedback enhancement process; the specific formulas are shown in Formulas 10 and 11, as follows:

[0050] (10)

[0051] (11)

[0052] In the formula, This represents the Sigmoid activation function. It is a 3×3 convolution operation.

[0053] In some embodiments, the loss function is as shown in Formula XII, as follows:

[0054] (12)

[0055] In the formula, This represents the importance percentage of the loss function, and its value is... =0.5、 =0.2、 =0.1, =0.1;

[0056] It also includes mean absolute error loss, used to measure the deblurred image. With real images The difference at the pixel level is represented by Formula Thirteen, as follows:

[0057] (13)

[0058] In the formula This represents the total number of pixels in the image;

[0059] The mean absolute error loss assigns equal weight to the error of each pixel;

[0060] The structural similarity loss measures the similarity between the predicted image and the real image in terms of brightness, contrast, and structure. Its definition is shown in Formula XIV, as follows:

[0061] (14)

[0062] In the formula, A larger index value indicates a more similar structure, with a maximum value of 1.

[0063] The frequency reconstruction loss, calculated by Fourier transform to determine the difference in image spectrum at different scales, is defined using Formula 15, as follows:

[0064] (15)

[0065] In the formula, Represents the Discrete Fourier Transform. and Let S represent the predicted image and the real image at the S-th scale, respectively;

[0066] The LHSV loss is publicly disclosed and expressed using Formula Sixteen, as follows:

[0067] (16)

[0068] In the formula, This represents a blurred image in the HSV color space. and These are the mean and standard deviation of the image, respectively. and It's a penalty added to improve network stability. The value is 0.001. The value is 0.009; f(xhsv) represents the true, sharp image in the HSV color space, where f(xhsv) is the deblurred image in the HSV color space.

[0069] In some embodiments, the parameter update method with faster convergence speed is used to update the parameters, specifically including:

[0070] The parameters are updated using formulas 17 and 18, as follows:

[0071] (17)

[0072] (18)

[0073] In the formula, Indicates model parameters, Indicates the first The model parameter values ​​at the next iteration ( ), This represents the gradient of the loss function of the model in this invention. This represents the second derivative of the loss function in the model of this invention. , The initial values ​​of the model parameters are represented by the intermediate values ​​of the parameter values ​​and the predicted parameter values ​​at the previous time step. In each iteration, the intermediate values ​​of the parameter values ​​and the predicted parameter values ​​at the previous time step are used as the input variables of the second derivative of the loss function.

[0074] In a second aspect, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method described in any of the first aspects above.

[0075] Thirdly, according to embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the first aspects above.

[0076] The above-described technical solution of the present invention has at least the following beneficial technical effects:

[0077] This application's dilated convolutional residual module can more effectively utilize high-level semantic feedback information from deep decoders to guide shallow decoders in restoring texture details. Furthermore, a feedback mechanism module is proposed that integrates feedback information from higher layers with the feature representation of the current layer, guiding the network to pay more attention to useful image regions, thereby improving image reconstruction quality. This application also proposes an improved loss function that better measures the difference between the deblurred image and the real image, thus improving the model's image deblurring performance. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the conventional art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a dual-decoder image deblurring method provided in an embodiment of the present invention.

[0080] Figure 2 This is a diagram of a dual-decoder image deblurring network structure based on cross-modal fusion of events and images provided in an embodiment of the present invention.

[0081] Figure 3 This is a dilated convolution residual module provided in an embodiment of the present invention.

[0082] Figure 4 This is a DRAB module provided in an embodiment of the present invention.

[0083] Figure 5 This is a cross-layer attention fusion module provided in an embodiment of the present invention.

[0084] Figure 6 This is a cross-modal feature fusion module provided in an embodiment of the present invention.

[0085] Figure 7 This is a feature fusion module based on local scoring provided in an embodiment of the present invention.

[0086] Figure 8 This is a feedback mechanism module provided in an embodiment of the present invention.

[0087] Figure 9 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the present invention.

[0090] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0091] In event-assisted image deblurring tasks, although the information captured by the event camera within the equivalent exposure time is highly correlated with the frame image, significant differences in their perception principles, dynamic range, and spatiotemporal distribution lead to modal inconsistencies in their feature spaces. In event-guided image restoration tasks, existing methods typically use convolution or attention mechanisms to directly fuse features from both modalities, ignoring the differences in semantic hierarchy and statistical distribution between modalities. This can easily cause information redundancy or guidance bias, affecting the final reconstruction quality. Therefore, this invention designs a cross-modal feature fusion module to enhance feature fusion in a more consistent environment, improving the quality of the fused image.

[0092] Event cameras inherently possess noise characteristics, and their response mechanisms differ from frame cameras due to their different imaging device types. Therefore, the location where an event is triggered may not perfectly correspond to the blurred area in the image. To determine the location where events exist and where blurring may occur, while simultaneously removing unnecessary event regions, this invention designs a feature fusion module based on local scoring. By comprehensively considering neighborhood information, it obtains more reliable fusion features. Even with repetitive structural patterns in the image, this module effectively mitigates uncertainties and ambiguities between modalities, thereby improving the quality of the fused image.

[0093] To address the common problems of insufficient feature extraction and loss of detail in low-resolution images, this invention designs a dilated convolution residual module.

[0094] To more effectively utilize high-level semantic feedback information from deep decoders to guide shallow decoders in restoring texture details, this paper proposes a feedback mechanism module. This module guides the network to pay more attention to useful image regions by fusing feedback information from higher layers with the feature representation of the current layer, thereby improving the image reconstruction quality.

[0095] like Figure 1 and Figure 2 As shown, a dual-decoder image deblurring method includes:

[0096] S1. The event image and the corresponding blurred image obtained by the event camera are input into the coding network, the features of the event image and the blurred image are extracted, and the event features are fused with the image features. After cross-modal feature fusion, the output features of the coding network are obtained.

[0097] S2. Input the output features into the decoding network, reconstruct the image from the input features, achieve preliminary deblurring of the image, and obtain a preliminary deblurred image;

[0098] S3. The preliminary deblurred image is input into the feedback mechanism network, and the feature map obtained by the feedback mechanism network is re-input into the decoding network in S2 to reconstruct a high-quality image from the feature map, thereby achieving image deblurring.

[0099] S4. The difference between the high-quality image and the real clear image is measured using the loss function, and the parameters are updated using a parameter update method with faster convergence speed to obtain the optimal model parameters.

[0100] The following is a detailed explanation.

[0101] Step 1: Input the event image and the corresponding blurred image obtained from the event camera into the encoding network, extract the features of the event image and the blurred image, and fuse the event features with the image features;

[0102] Specifically, in step 1, the encoding network consists of two structurally identical encoding sub-networks and a feature fusion network. The encoding sub-networks include, in sequence, convolutions with 3 kernels, dilated convolutional residual modules (DCRM), and references... Figure 3 As shown, the module consists of a downsampling module, a dilated convolution residual module, and a downsampling module and a dilated convolution residual module.

[0103] The feature fusion network consists of a cross-modal feature fusion module (CFFM), a local scoring-based feature fusion module (LSFF), a cross-modal feature fusion module, a local scoring-based feature fusion module, a cross-modal feature fusion module, and a local scoring-based feature fusion module in sequence.

[0104] The event image is input into editor 1, and the blurred image is input into encoder 2. The output of the first dilated convolutional residual module in encoder 1 and encoder 2 is the input of the first cross-modal feature fusion module. The output of the second dilated convolutional residual module in encoder 1 and encoder 2 is the input of the second cross-modal feature fusion module. The output of the third dilated convolutional residual module in encoder 1 and encoder 2 is the input of the third cross-modal feature fusion module. The output of the first dilated convolutional residual module and the output of the first cross-modal feature fusion module in encoder 2 are the inputs of the first feature fusion module based on local scoring. The output of the first feature fusion module based on local scoring is feature map F1. Feature map F1, after downsampling, is compared with the output feature map of the second cross-modal feature fusion module. Figure 1 The input is fed into the second feature fusion module based on local scoring to obtain feature map F2; feature map F2 is downsampled and then combined with the output feature map of the third cross-modal feature fusion module. Figure 1 The input is fed into the third feature fusion module based on local scoring to obtain the output features of the encoding network;

[0105] Step 2: Input the output features obtained in Step 1 into the decoding network to reconstruct the image from the input features, thus achieving preliminary deblurring of the image; the decoder network contains two structurally identical encoding sub-networks (decoder 1 and decoder 2), each encoding sub-network consisting of 3 DCRM modules, 4 upsampling modules, and 1 convolution with a kernel of 3;

[0106] The input features in step 1 are processed by the first DCRM module in decoder 1 to obtain feature map T1. Feature map T1 is upsampled and then element-wise added to feature map F2 in encoder. The resulting feature is input to the second DCRM module in decoder 1. The output feature is upsampled and then element-wise added to feature map F1 in encoder. The added feature map is then input to the third DCRM module in decoder and a convolution with a kernel of 3 to obtain the decoder output. The input of decoder 2 is the same as the output of decoder 1, and the feature processing process is also the same. The output feature map of the first DCRM module in decoder 2 is T2. The outputs of decoder 1 and decoder 2 are element-wise added to obtain the preliminary deblurred image.

[0107] Step 3: Input the preliminary deblurred image obtained in Step 2 into the feedback mechanism network, and re-input the feature map obtained from the feedback mechanism network into the two decoders in Step 2. Reconstruct the final high-quality image from the feature map to achieve image deblurring.

[0108] The feedback mechanism network consists of four feedback mechanism modules (FAM). By fusing feedback information from higher layers with the feature representation of the current layer, the network is guided to pay more attention to useful image regions, thereby improving the image reconstruction quality.

[0109] The specific process is as follows: The preliminary deblurred image obtained in step 2 is downsampled to obtain image 2; after feature map T1 and image 2 from step 2 are input into the first FAM module, the output feature map is upsampled sequentially, element-wise added to feature map F2, and then input into the second DCRM module in decoder 1 to obtain feature map T3; feature map T3 and the preliminary deblurred image obtained in step 2 are input into the second FAM module, and the output feature map is upsampled sequentially, element-wise added to feature map F1 to obtain feature map T4; feature map T4 is processed by the third DCRM in decoder 1 and a convolution with a kernel of 3 to obtain... Image 3 is obtained; after the feature map T2 and image 2 from step 2 are input into the third FAM module, the output feature map is upsampled sequentially, and then element-wise added to feature map F2 before being input into the second DCRM module in decoder 2 to obtain feature map T5; feature map T5 and the preliminary deblurred image obtained in step 2 are input into the fourth FAM module, and the output feature map is upsampled sequentially and then element-wise added to feature map F1 to obtain feature map T6; feature map T6 is processed by the third DCRM in decoder 2 and convolution with a kernel of 3 to obtain image 4; image 3 and image 4 are pixel-wise added to obtain the final deblurred image.

[0110] Step 4: Measure the difference between the deblurred image and the real clear image using the loss function, and update the parameters using a parameter update method with faster convergence speed to improve the convergence speed during model training and finally obtain the optimal model parameters.

[0111] Specifically, the dilated convolution residual module in step 1 consists of three branches. In the first branch, the input feature map of this module first undergoes a convolution with a kernel of 1 to obtain feature map A1. In the second branch, the input feature map first undergoes a convolution with a kernel of 3 to obtain feature map A2. In the third branch, the input feature map undergoes a convolution with a kernel of 5 and a DRAB module. For details of the DRAB module, please refer to [link to DRAB module documentation]. Figure 4 As shown, feature map A3 is obtained; feature map A3 is element-wise added to feature map A2, and then passed through the DARB module to obtain feature map A4; feature map A4 is element-wise added to feature maps A3 and A1, and the resulting feature map is passed through the DARB module to obtain feature map A5; feature maps A5, A4, and A3 are input together into the cross-layer attention fusion module, which can be found in [reference needed]. Figure 5 As shown, the output feature map of the dilated convolutional residual module is finally obtained; the DARB module is a residual structure module, and the main network of the residual structure module consists of four hybrid modules, a concatenation operation, a convolution with a kernel of 1, and an attention module; each hybrid module consists of a convolution with a kernel of 3 and a ReLU activation function; the output feature of each hybrid module is concatenated and fused with the output features of each of the previous few hybrid modules, and the fused feature is used as the input feature of a hybrid module;

[0112] The cross-layer attention fusion module in the dilated convolutional residual module contains concatenated features from three consecutive layers. Firstly, this invention uses a splicing operation to combine three features. , , Remodeling Features of Dimensions Secondly, this invention employs 1×1 convolutions to aggregate information across the channel dimensions of each pixel location to extract pixel-level cross-channel contextual features. Then, a 3×3 depthwise separable convolution is used to generate the query (Q), key (K), and value (V). Subsequently, the query and key are reshaped into... (dimension is) )and (dimension is) The two-dimensional matrix is ​​calculated to obtain The interlayer correlation attention matrix A is given. The final reshaped value matrix will be... Multiply by the interlayer related attention matrix A (with scaling factor α) and then by the input features. The features are summed to obtain the output features of the cross-layer attention fusion module; the specific processing procedure of the cross-layer attention fusion module is as follows:

[0113] (1)

[0114] (2)

[0115] (3)

[0116] In the formula, This represents the Softmax function, which normalizes the inter-layer similarity matrix and is used to generate attention weights. It is a learnable scaling factor used to adjust the magnitude of the attention score. For convolution weights, Features after splicing It is the output characteristic of the module.

[0117] Specifically, the cross-modal feature fusion module in step 1 consists of two parts: a feature alignment unit and a multi-head attention mechanism. The cross-modal feature fusion module can be referenced from... Figure 6 As shown, during feature alignment, the cross-modal feature fusion module guides features from one modality to participate in the transformation process of features from another modality through convolution operations, thereby bridging the gaps in inconsistencies between modalities. Image features and event image features are mutually adjusted through a combination of addition and multiplication to compensate for the representational shift caused by modal differences. The aligned features are then input into the attention module, where the multi-head attention module plays a bidirectional role. Image features can be used as a query vector to extract key responses (Key, Value) from event image features, and conversely, event image features can perform the same operation on image features, thereby achieving cross-guidance of information between the two modalities and alignment of salient regions. Since the weights of the attention module are shared, motion information can be fused in a more consistent space; the cross-modal feature fusion module can be described as follows:

[0118] (4)

[0119] (5)

[0120] (6)

[0121] in and Aligned image features Event characteristics , This indicates a multi-head attention module. This indicates a channel-dimension concatenation operation. This is the final output of the module.

[0122] For the feature fusion module of local scoring, please refer to... Figure 7 As shown, further, the feature fusion module based on local scoring in step 1 will integrate image features... Blending features after spatial alignment The input is passed to a 3×3 convolutional layer with ReLU activation, resulting in a layer of size [size missing]. The features. Next, for each pixel location, the present invention samples its surrounding neighboring pixels from the image features and event features respectively using a specified window, to obtain... , respectively, represent the local region features obtained by sampling image and event features around each spatial location ( (This refers to the number of neighboring pixels). Then, the present invention calculates... and Similarity-based matching score generation:

[0123] (7)

[0124] In the formula The inner product operation is represented by a score map containing similarity scores between sharp representations of both modalities, effectively suppressing redundant event responses in deblurring tasks. The features are then weighted and aggregated using this similarity score and alignment features of each neighboring pixel, as shown below:

[0125] (8)

[0126] In the formula, Indicates the location Nearby Spatially aligned event features obtained by sampling from each neighborhood. A similarity score is given between the image and event features at that neighborhood location. This refers to the Sigmoid function, which maps matching scores to the interval (0, 1). This indicates the number of neighboring pixels involved in the weighted calculation at each location (set to 9 in this invention). For the final position The aggregation features obtained by weighted aggregation.

[0127] Finally, the output of the feature fusion module based on local scoring is composed of aggregated features. Spatial alignment event characteristics Adding them together, we get:

[0128] (9)

[0129] By comprehensively considering neighborhood information, the feature fusion module based on local scoring can obtain more reliable fused features. Even if there are repeated structural patterns in the image, this module can effectively alleviate the uncertainty and ambiguity between modalities.

[0130] Furthermore, the feedback mechanism module in step 3 can be referred to... Figure 8 As shown, let the input feature map of the current layer be... The feedback image from the upper layer is First, for the feedback image... Apply a 3×3 convolution operation and use the Sigmoid activation function to generate an attention map. This represents the importance weight of the feedback feature at different locations; the attention map can adaptively reflect the importance of different spatial locations, thereby highlighting information regions beneficial to the current input feature and effectively guiding the enhancement and reconstruction of the current layer's features; subsequently, the current input feature... Pass through a 3×3 convolutional layer and with attention map Perform element-wise multiplication to obtain the attention-modulated feature response and the input feature map. Add them together to get the final output. This completes the feedback enhancement process; the specific formula is as follows:

[0131] (10)

[0132] (11)

[0133] In the formula, This represents the Sigmoid activation function. It is a 3×3 convolution operation.

[0134] In step 4, the content of the loss function can be found as follows:

[0135] In order to effectively guide the network to recover clear texture, structural and color information from severely blurred images, this invention designs a composite loss function that jointly considers pixel accuracy, structural similarity and frequency domain consistency.

[0136] The overall loss function is defined as follows:

[0137] (12)

[0138] In the formula, This represents the importance percentage of the loss function, and its value is... =0.5、 =0.2、 =0.1, =0.1. Mean Absolute Error (MAE) loss is used to measure the loss of a deblurred image. With real images The difference at the pixel level is expressed by the following formula:

[0139] (13)

[0140] In the formula This represents the total number of pixels in the image. The loss assigns equal weight to the error of each pixel, providing a stable gradient across the entire image range, and is particularly effective for restoring blurred areas.

[0141] To improve the structural and perceptual quality of images, this invention introduces a structural similarity (SSIM) loss, which measures the similarity between the predicted image and the real image from three aspects: brightness, contrast, and structure. Its definition is as follows:

[0142] (14)

[0143] In the formula, A higher index value indicates a more similar structure, with a maximum value of 1. This is achieved by minimizing... This encourages networks to generate clearer structural details that better match human visual perception.

[0144] To further enhance image detail preservation, especially high-frequency texture information, this paper introduces a frequency reconstruction (MSFR) loss. This loss calculates the difference in image spectrum at different scales using Fourier transform, and is defined as follows:

[0145] (15)

[0146] In the formula, Represents the Discrete Fourier Transform. and Let represent the predicted image and the real image at the S-th scale, respectively. This loss term enhances the modeling ability for high-frequency information, helps to restore fine textures and edges, and mitigates the over-smoothing problem common in traditional methods.

[0147] To more accurately measure the color difference between a real image and a deblurred image, this invention discloses the LHSV loss, expressed as follows:

[0148] (16)

[0149] In the formula, This represents a blurred image in the HSV color space. and These are the mean and standard deviation of the image, respectively. and These are penalty items added to increase network stability. Their values ​​are 0.001 and 0.009, respectively. f(xhsv) represents the true, sharp image in the HSV color space, where f(xhsv) is the deblurred image in the HSV color space.

[0150] Furthermore, in step 4, in order to accelerate the convergence speed during model training, this invention employs a fast optimization method to update the model parameters, the specific process of which is as follows:

[0151] (17)

[0152] (18)

[0153] In the formula, Indicates model parameters, Indicates the first The model parameter values ​​at the next iteration ( ), This represents the gradient of the loss function of the model in this invention. This represents the second derivative of the loss function in the model of this invention. , The initial values ​​of the model parameters are represented by the intermediate value between the parameter values ​​and the predicted parameter values ​​at the previous time step. In each iteration, the intermediate value between the parameter values ​​at the previous time step and the predicted parameter values ​​is used as the input variable of the second derivative of the loss function. This can ensure that the convergence direction can be better adjusted towards the local optimum. Compared with the conventional loss function optimization method in deep learning, this method has better control over the optimization direction and converges faster during model training.

[0154] To further illustrate the effects of the present invention and objectively verify the effectiveness of the method, all paired images in the GoPro test set were used as test images, and the average results of PSNR and SSIM for different methods were calculated, as shown in Table 1.

[0155] Table 1 Quantitative comparisons on the GoPro dataset

[0156]

[0157] All paired images in the ReBlur and RwEvent datasets were used as test images, and the average PSNR and SSIM results for different methods were calculated. The structure is shown in Table 2.

[0158] Table 2 Quantitative comparison on the ReBlur dataset and RwEvent data

[0159]

[0160] According to embodiments of the present invention, an electronic device 900 is also provided, such as... Figure 9 As shown, it includes a memory 901, a processor 902, and a computer program stored in the memory 901 and executable on the processor. When the processor 902 executes the program, it implements the steps of the method described in any of the above embodiments.

[0161] According to embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0162] This invention also provides a computer program product, including a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the methods described in the above embodiments.

[0163] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0164] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of the invention and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of the invention should be included within the protection scope of the invention. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A dual-decoder image deblurring method, characterized in that, include: S1. The event image and the corresponding blurred image obtained by the event camera are input into the coding network, the features of the event image and the blurred image are extracted, and the event features are fused with the image features. After cross-modal feature fusion, the output features of the coding network are obtained. S2. Input the output features into the decoding network, reconstruct the image from the input features, achieve preliminary deblurring of the image, and obtain a preliminary deblurred image; S3. The preliminary deblurred image is input into the feedback mechanism network, and the feature map obtained by the feedback mechanism network is re-input into the decoding network in S2 to reconstruct a high-quality image from the feature map, thereby achieving image deblurring. S4. The difference between the high-quality image and the real clear image is measured using an improved loss function, and the parameters are updated using a parameter update method with faster convergence speed to obtain the optimal model parameters.

2. The method according to claim 1, characterized in that, The coding network includes: two structurally identical coding sub-networks and a feature fusion network; The encoding sub-network comprises, in sequence: a convolution with a kernel of 3, a dilated convolution residual module, a downsampling module, a dilated convolution residual module, a downsampling module, and a dilated convolution residual module; The feature fusion network comprises, in sequence: a cross-modal feature fusion module, a feature fusion module based on local scoring, a cross-modal feature fusion module, a feature fusion module based on local scoring, a cross-modal feature fusion module, and a feature fusion module based on local scoring. The process involves extracting features from the event image and the blurred image, fusing the event features with the image features, and then performing cross-modal feature fusion to obtain the output features of the coding network. Specifically, this includes: The event image is input into the first encoder, and the blurred image is input into the second encoder. The outputs of the first dilated convolutional residual modules in the first and second encoders are the inputs of the first cross-modal feature fusion module. The outputs of the second dilated convolutional residual modules in the first and second encoders are the inputs of the second cross-modal feature fusion module. The outputs of the third dilated convolutional residual modules in the first and second encoders are the inputs of the third cross-modal feature fusion module. The outputs of the first dilated convolutional residual module and the first cross-modal feature fusion module in the second encoder are the inputs of the first feature fusion module based on local scoring. The output of the first feature fusion module based on local scoring is a first feature map F1. After downsampling, the first feature map F1 is input together with the output feature map of the second cross-modal feature fusion module into the second feature fusion module based on local scoring to obtain a second feature map F2. After downsampling, the feature map F2 is input together with the output feature map of the third cross-modal feature fusion module into the third feature fusion module based on local scoring to obtain the output features of the encoding network. The decoding network includes two structurally identical encoding sub-networks, namely the first decoder and the second decoder. Each encoding sub-network consists of 3 DCRM modules, 4 upsampling modules, and 1 convolution with a kernel of 3. The step of inputting the output features into the decoding network and reconstructing the image from the input features to achieve preliminary deblurring of the image and obtain a preliminary deblurred image specifically includes: The output features are processed by the first DCRM module in the first decoder to obtain a first feature map T1. The feature map T1 is upsampled and then element-wise added to the feature map F2. The resulting feature is input to the second DCRM module in the first decoder. The output features are then upsampled and element-wise added to the first feature map F1. The added feature map is then input to the third DCRM module in the first decoder and a convolution with a kernel of 3 to obtain the decoder output. The input of the second decoder is the same as the output of the first decoder. The output feature map of the first DCRM module in the second decoder is the second feature map T2. The outputs of the first decoder and the second decoder are element-wise added to obtain a preliminary deblurred image.

3. The method according to claim 2, characterized in that, The feedback mechanism network includes four feedback mechanism modules, which are used to guide the network to pay more attention to useful image regions and improve image reconstruction quality by fusing feedback information from higher layers with the feature representation of the current layer. The step of re-inputting the feature map obtained from the feedback mechanism network into the decoding network in S2 to reconstruct a high-quality image from the feature map and achieve image deblurring specifically includes: The initially deblurred image is downsampled to obtain the second image 2. The feature map T1 and the second image 2 are input into the first FAM module, and the output feature map is sequentially upsampled, element-wise added to the feature map F2, and then input into the second DCRM module in the first decoder to obtain feature map T3. Feature map T3 and the initially deblurred image obtained in step S2 are input into the second FAM module, and the resulting output feature map is sequentially upsampled, element-wise added to the first feature map F1 to obtain feature map T4. Feature map T4 is then processed by the third DCRM module in the first decoder and a convolution with a kernel of 3 to obtain image 3. The second feature map in step S2... After feature map T2 and image 2 are input into the third FAM module, the output feature map is sequentially upsampled, element-wise added to feature map F2, and then input into the second DCRM module in the second decoder 2 to obtain the fifth feature map T5. The fifth feature map T5 and the preliminary deblurred image obtained in step S2 are input into the fourth FAM module, and the resulting output feature map is sequentially upsampled, element-wise added to the first feature map F1 to obtain the sixth feature map T6. Feature map T6 is processed by the third DCRM in the second decoder 2 and a convolution with a kernel of 3 to obtain the fourth image 4. The third image 3 and the fourth image 4 are pixel-wise added to obtain the final deblurred image. The feedback mechanism module includes: Let the input feature map of the current layer be... The feedback image from the upper layer is ; For the feedback image Apply a 3×3 convolution operation and use the Sigmoid activation function to generate an attention map. , representing the importance weight of the feedback feature at different locations; Current input features Pass through a 3×3 convolutional layer and with attention map Perform element-wise multiplication to obtain the attention-modulated feature response and the input feature map. Add them together to get the final output. This completes the feedback enhancement process; the specific formulas are shown in Formulas 10 and 11, as follows: （10） （11） In the formula, This represents the Sigmoid activation function. It is a 3×3 convolution operation.

4. The method according to claim 1, characterized in that, The improved loss function is shown in Formula XII, as follows: （12） In the formula, This represents the importance percentage of the loss function, with values ​​ranging from 1 to 2. =0.5、 =0.2、 =0.1, =0.2; It also includes mean absolute error loss, used to measure the deblurred image. With real images The difference at the pixel level is represented by Formula Thirteen, as follows: (13) In the formula This represents the total number of pixels in the image; The mean absolute error loss assigns equal weight to the error of each pixel; The structural similarity loss measures the similarity between the predicted image and the real image in terms of brightness, contrast, and structure. Its definition is shown in Formula XIV, as follows: (14) In the formula, A larger index value indicates a more similar structure, with a maximum value of 1. The frequency reconstruction loss, calculated by Fourier transform to determine the difference in image spectrum at different scales, is defined using Formula 15, as follows: (15) In the formula, Represents the Discrete Fourier Transform. and Let S represent the predicted image and the real image at the S-th scale, respectively; Public L HSV The loss is represented by Formula Sixteen, as follows: （16） In the formula, This represents a blurred image in the HSV color space. and These are the mean and standard deviation of the image, respectively. and It's a penalty added to improve network stability. The value is 0.

001. The value is 0.009; f(xhsv) represents the true, sharp image in the HSV color space, where f(xhsv) is the deblurred image in the HSV color space.

5. The method according to claim 1, characterized in that, The dilated convolution residual module specifically includes three branches. In the first branch, the input feature map of the dilated convolution residual module undergoes a convolution with kernel 1 to obtain the A1 feature map. In the second branch, the input feature map undergoes a convolution with kernel 3 to obtain the A2 feature map. In the third branch, the input feature map undergoes a convolution with kernel 5 and a DRAB module to obtain the A3 feature map. The A3 feature map and the A2 feature map are element-wise added, and then the A4 feature map is obtained after passing through the DARB module. The A4 feature map is element-wise added to the A3 and A1 feature maps, and the resulting feature map is then processed by the DARB module to obtain the A5 feature map. The A5 feature map, the A4 feature map, and the A3 feature map are input together into the cross-layer attention fusion module to finally obtain the output feature map of the dilated convolution residual module; The DARB module is a residual structure module. The main network of the residual structure module includes four hybrid modules, a concatenation operation, a convolution with a kernel of 1, and an attention module. Each hybrid module includes a convolution with a kernel of 3 and a ReLU activation function. The output features of each hybrid module are concatenated and fused with the output features of each of the previous few hybrid modules. The fused features are used as the input features of a hybrid module. The cross-layer attention fusion module in the dilated convolutional residual module contains spliced ​​features from three consecutive layers. By splicing the three features , , Remodeling Features of Dimensions A 1×1 convolution is used to aggregate information across the channel dimensions of each pixel location to extract pixel-level cross-channel contextual features; a 3×3 depthwise separable convolution is used to generate queries, keys, and values; the queries and keys are then reshaped into... and The The dimension is The Dimensions A two-dimensional matrix; calculated to obtain The interlayer correlation attention matrix A; the reshaped value matrix Multiply with the interlayer related attention matrix A and with the input features The features are added together to obtain the output features of the cross-layer attention fusion module; the specific processing steps of the cross-layer attention fusion module are shown in Formula 1, Formula 2, and Formula 3 below: （1） （2） （3） In the formula, This represents the Softmax function, which normalizes the inter-layer similarity matrix and is used to generate attention weights. It is a learnable scaling factor used to adjust the magnitude of the attention score. For convolution weights, Features after splicing It is the output characteristic of the module.

6. The method according to claim 1, characterized in that, The cross-modal feature fusion module includes two parts: a feature alignment unit and a multi-head attention mechanism. In the feature alignment process, the cross-modal feature fusion module guides the features of one modality to participate in the change process of the features of another modality through convolution operations, thereby bridging the gaps in inconsistencies between modalities; Image features and event image features are adjusted to each other by combining addition and multiplication to compensate for the representational shift caused by modal differences; The aligned features are then input into an attention module, where a multi-head attention module acts bidirectionally, using image features as query vectors to extract key responses from event image features; the weights of the attention module are shared; the cross-modal feature fusion module is described as shown in Formulas 4, 5, and 6: （4） （5） （6） in and These are the aligned image features. Event characteristics , This indicates a multi-head attention module. This indicates a channel-dimension concatenation operation. This is the final output of the cross-modal feature fusion module.

7. The method according to claim 2, characterized in that, The feature fusion module based on local scoring is used to integrate image features. Blending features after spatial alignment The input is passed to a 3×3 convolutional layer with ReLU activation, resulting in a layer of size [size missing]. Features; For each pixel location, neighboring pixels are sampled from both image features and event features within a specified window to obtain... , respectively, represent the local region features obtained by sampling image and event features around each spatial location; Through calculation and The similarity score is generated as shown in Formula 7: （7） In the formula This represents the inner product operation. The scoring map contains similarity scores between sharp representations of both modalities, thereby suppressing redundant event responses in the deblurring task. Using this similarity score and the alignment features of each neighboring pixel, the features are weighted and aggregated, as shown in Equation 8: （8） In the formula, Indicates the location Nearby Spatially aligned event features obtained by sampling from each neighborhood. A similarity score is given between the image and event features at that neighborhood location. This refers to the Sigmoid function, which maps matching scores to the interval (0, 1). This represents the number of neighboring pixels at each location that participate in the weighted calculation; For the final position Aggregated features obtained by weighted aggregation; The output of the feature fusion module based on local scoring is aggregated features. Spatial alignment event characteristics The result is obtained by adding the ingredients together, as shown in Formula Nine: （9） By comprehensively considering neighborhood information, the feature fusion module based on local scoring can obtain more reliable fused features.

8. The method according to claim 1, characterized in that, The parameter update method, which employs a faster convergence speed, specifically includes: The parameters are updated using formulas 17 and 18, as follows: （17） （18） In the formula, Indicates model parameters, Indicates the first The model parameter values ​​at the next iteration ( ), This represents the gradient of the loss function of the model in this invention. This represents the second derivative of the loss function in the model of this invention. , The initial values ​​of the model parameters are represented by the intermediate values ​​of the parameter values ​​and the predicted parameter values ​​from the previous time step. In each iteration, the intermediate values ​​of the parameter values ​​and the predicted parameter values ​​from the previous time step are used as the input variables for the second derivative of the loss function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images