Moving image deblurring method based on multi-modal bidirectional fusion

Abstract

The invention relates to the technical field of image processing, in particular to a moving image deblurring method based on multi-modal bidirectional fusion, which comprises the following steps of: firstly acquiring multi-modal data information of a moving blurred image, and then acquiring preprocessed event stream information which is compatible with image information and is in a three-dimensional voxel format based on the event stream information; performing multi-modal bidirectional fusion processing on the image information and the preprocessed event flow information to obtain fusion features of the motion blurred image; taking the fusion features as input, carrying out residual connection step-by-step learning through a residual connection module, and generating a residual sequence; and based on the residual sequence, obtaining a deblurred motion clear image through an event-based double integral physical model. According to the method, through a bidirectional fusion strategy from shallow to deep, the development of moving image deblurring is promoted, a new thought and method are provided for multi-modal data fusion, and technical support is provided for efficient application of other tasks such as automatic driving and target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a motion image deblurring method based on multi-modal bidirectional fusion. Background Art

[0002] Blur is a very common distortion phenomenon in image data, mainly including out-of-focus blur and motion blur. Motion blur is particularly common. It is caused by the relative motion between the camera and the object during the exposure time, causing the pixels in the image to be affected by their neighboring pixels, making the image content unclear and details lost. In order to ensure the clarity of moving images, it is necessary to perform motion image deblurring, that is, to restore the lost details and clarity in the image.

[0003] Early deblurring techniques mainly rely on modeling the blurring process and restoring image clarity by optimizing the model. However, this deblurring method is often limited by the size of the solution space and the complexity of the blur kernel, making it difficult to achieve ideal results in practical applications. With the development of deep learning technology, motion deblurring methods based on convolutional neural networks have been widely used, and end-to-end deblurring methods have gradually emerged. Specifically, they directly learn the mapping relationship between blurred images and clear images by training neural networks, which can effectively avoid the error problem of blur kernel estimation in early deblurring methods. However, this motion deblurring method based on convolutional neural networks often only focuses on the information of the image itself, ignoring the influence of camera motion factors that cause blur, resulting in low image quality of the deblurred results of motion images. Summary of the invention

[0004] In order to solve the technical problem of low image quality of the above-mentioned motion image deblurring result, the object of the present invention is to provide a motion image deblurring method based on multimodal bidirectional fusion, and the technical solution adopted is as follows:

[0005] An embodiment of the present invention provides a motion image deblurring method based on multimodal bidirectional fusion, the method comprising the following steps:

[0006] Acquire multimodal data information of a motion blurred image, and input the multimodal data information into an improved motion image deblurring model to obtain a motion clear image; wherein the multimodal data information includes image information and event stream information;

[0007] The implementation process of the improved motion image deblurring model includes:

[0008] Acquire preprocessed event stream information in a three-dimensional voxel format that is compatible with the image information based on the event stream information;

[0009] The image information and the preprocessed event stream information are subjected to multimodal bidirectional fusion processing to obtain the fusion features of the motion blurred image;

[0010] Taking the fused features as input, performing residual connection step by step learning through a residual connection module and generating a residual sequence;

[0011] Based on the residual sequence, the deblurred motion clear image is obtained through an event-based double integration physical model.

[0012] Furthermore, the step of obtaining the preprocessed event stream information includes:

[0013] Firstly, the event stream information within the exposure period of the motion blurred image is accumulated based on the principle of time symmetry to obtain a sequence containing multiple event frames.

[0014] Then, several event frames located in the middle of the sequence are removed, and the remaining event frames are converted into event data in a three-dimensional voxel format as preprocessed event stream information.

[0015] Furthermore, the image information and the preprocessed event stream information are subjected to multimodal bidirectional fusion processing to obtain fusion features, including:

[0016] Inputting the image information and the preprocessed event stream information into the primary fusion module for processing to obtain a first fused image feature and a first fused event feature;

[0017] A deep fusion module is constructed by combining the channel attention mechanism and the spatial attention mechanism, and the first fused image feature and the first fused event feature are bidirectionally fused to obtain a second fused image feature and a second fused event feature;

[0018] The second fused image feature and the second fused event feature are input into the final fusion module, and feature mining is performed in the vertical and horizontal directions respectively to obtain the fused features of the motion blurred image.

[0019] Furthermore, the image information and the pre-processed event stream information are input into the primary fusion module for processing to obtain the first fused image feature and the first fused event feature, including:

[0020] Perform two convolution operations on the image information and the preprocessed event stream information respectively to obtain the local features of the image and the local features of the event;

[0021] Normalizing the local features of the image and the local features of the event, and introducing a nonlinear decision boundary of a nonlinear activation function to obtain an image feature map and an event feature map;

[0022] The image feature map and the event feature map are multiplied element by element to obtain a fused feature map; the image feature map and the fused feature map are added to obtain a first fused image feature; the event feature map and the fused feature map are added to obtain a first fused event feature.

[0023] Furthermore, the deep fusion module is constructed by combining the channel attention mechanism and the spatial attention mechanism, and the first fused image feature and the first fused event feature are subjected to bidirectional fusion processing respectively to obtain the second fused image feature and the second fused event feature, including:

[0024] The importance of each feature channel in the first fused image feature and the first fused event feature is dynamically evaluated by using the channel attention mechanism, and the first fused image feature and the first fused event feature processed by the channel attention are obtained, which are recorded as the third fused image feature and the third fused event feature;

[0025] The spatial attention mechanism is used to focus on analyzing the criticality of each position information in the third fused image feature and the third fused event feature, and the third fused image feature and the third fused event feature processed with spatial attention are obtained, which are recorded as the second fused image feature and the second fused event feature.

[0026] Furthermore, the second fused image feature and the second fused event feature are input into the final fusion module, and feature mining is performed in the vertical and horizontal directions respectively to obtain the fusion feature of the motion blurred image, including:

[0027] The second fused image feature and the second fused event feature are processed by element multiplication and maximum value selection to obtain two composite features, and the two composite features are connected together to obtain a first connection feature;

[0028] Performing average pooling operations on the first connection feature and the first undetermined fusion feature in the vertical and horizontal directions respectively, connecting the obtained features to obtain a second connection feature; wherein the first undetermined fusion feature is a first fusion event feature or a first fusion image feature;

[0029] The second connected feature is processed using four convolutions and normalization operations to obtain four attention weights in the vertical and horizontal directions;

[0030] Performing weighted aggregation processing on the pending fusion event feature and the first connection feature through the attention weight to obtain a second pending fusion feature; wherein the second pending fusion feature is a fusion image mining feature or a fusion event mining feature;

[0031] The fused image mining feature and the fused event mining feature are added to obtain the fused feature of the motion blurred image.

[0032] Furthermore, the weighted aggregation processing of the pending fusion event feature and the first connection feature by using the attention weight to obtain the second pending fusion feature includes:

[0033] Where A2 x represents the second undetermined fusion feature, represents the feature of the pending fusion event, x represents an image or event, Wh(x) represents the attention weight of the pending fusion event feature in the vertical direction, Ww(x) represents the attention weight of the pending fusion event feature in the horizontal direction, represents the first connection feature, Wh(o) represents the attention weight of the first connection feature in the vertical direction, Ww(o) represents the attention weight of the first connection feature in the horizontal direction, Represents element-wise multiplication.

[0034] Furthermore, the fusion feature is used as input, and residual connection is performed through a residual connection module to gradually learn and generate a residual sequence, including:

[0035] First, the fused features are downsampled four times to obtain the features extracted by each downsampling; each sampling block consists of a convolutional layer and an activation function;

[0036] The features extracted by each downsampling pass through two residual modules and then are decoded through four upsampling blocks to obtain a residual sequence.

[0037] Furthermore, the method of obtaining a clear image after deblurring based on the residual sequence through an event-based double integral physical model includes:

[0038] The residual sequence is taken as input, and the expression form of the motion blurred image is derived through an event-based double integral physical model; based on the expression form of the motion blurred image, the difference between the motion blurred image and the average residual sequence is calculated to obtain a clear image after deblurring.

[0039] Furthermore, the calculation formula of the loss function of the improved motion image deblurring model is:

[0040] loss = deblur_loss + res_loss * w; where deblur_loss represents the loss of deblurring, res_loss represents the loss of the predicted residual sequence, and w represents the weight vector;

[0041] deblur_loss = ||D(N)-Gd(N)|| 2 ;res_loss=||E(q)-Gr(q)|| 2; In the formula, D(N) represents the motion clear image after deblurring, Gd(N) represents the real motion clear image, E(q) represents the predicted residual sequence, and Gr(q) represents the real residual sequence.

[0042] The present invention has the following beneficial effects:

[0043] The present invention provides a motion image deblurring method based on multimodal bidirectional fusion. The method first obtains multimodal data information of a motion blurred image, and inputs the multimodal data information into an improved motion image deblurring model to obtain a motion clear image; then, for the improved motion image deblurring model, its implementation process includes: obtaining pre-processed event stream information in a three-dimensional voxel format that is compatible with image information based on event stream information; performing multimodal bidirectional fusion processing on the image information and the pre-processed event stream information to obtain fusion features of the motion blurred image; using the fusion features as input, performing residual connection stepwise learning through a residual connection module and generating a residual sequence; based on the residual sequence, obtaining a deblurred motion clear image through an event-based double integral physical model.

[0044] The present invention uses multimodal fusion technology to fuse the features corresponding to blurred image data and event stream data for analysis. Compared with the existing deblurring method that only analyzes blurred image data, the present invention can effectively enhance the deblurring effect by adding event stream data used to describe camera motion information. When fusing the data of the two data sources, a carefully designed new fusion strategy, i.e. a coarse-to-fine and step-by-step bidirectional attention fusion mechanism, is used to achieve in-depth mining and complementarity of the advantages of both. This not only effectively compensates for the defect that motion information is easily lost in blurred images, but also significantly improves the accuracy of motion deblurring, while also providing technical support for the efficient application of other computer vision tasks such as autonomous driving and target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of the steps of implementing the improved motion image deblurring model in a motion image deblurring method based on multimodal bidirectional fusion according to an embodiment of the present invention;

[0047] Figure 2It is an overall framework diagram of an improved motion image deblurring model in a motion image deblurring method based on multi-modal bidirectional fusion according to an embodiment of the present invention;

[0048] Figure 3 This is a flowchart of step S102 in a motion image deblurring method based on multimodal bidirectional fusion according to an embodiment of the present invention;

[0049] Figure 4 It is a framework diagram of a final fusion module in a motion image deblurring method based on multi-modal bidirectional fusion according to an embodiment of the present invention;

[0050] Figure 5 This is an example diagram of a motion blurred image in a motion image deblurring method based on multimodal bidirectional fusion according to an embodiment of the present invention;

[0051] Figure 6 This is an example diagram of a motion clear image in a motion image deblurring method based on multimodal bidirectional fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the technical solutions proposed by the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0053] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0054] The application scenarios targeted by the present invention may be:

[0055] In order to overcome the problem that existing image deblurring methods only focus on the information of the image itself, ignore the camera motion factors that cause blur, and lead to low quality of deblurring results, the present invention not only collects motion blurred images, but also collects event stream information during the exposure of motion blurred images, which can be used to analyze the camera motion factors that cause blur. Regarding image information and event stream information, on the one hand, event cameras can asynchronously record the logarithmic intensity changes of single pixels with low latency, and event stream data contains detailed motion information, which is often difficult to fully and clearly display in motion blurred images; on the other hand, motion blurred images are known for their rich texture features, while these texture features are relatively scarce in event data streams, so how to use more efficient technical methods to integrate the unique advantages of the two has become a core technical challenge in solving the problem of motion image deblurring.

[0056] Therefore, this embodiment provides a motion image deblurring method based on multimodal bidirectional fusion. The method first obtains multimodal data information of the motion blurred image, and then inputs the obtained multimodal data information into an improved motion image deblurring model to obtain a clear motion image after deblurring.

[0057] Among them, the multimodal data information may include image information and event stream information. The improved motion image deblurring model is a model that has been constructed and trained to improve the traditional motion image deblurring model. The input data of the improved motion image deblurring model is the multimodal data information of the motion blurred image.

[0058] During the training and learning process of the model, the adam optimizer was used, the initial learning rate was set to 0.001, and the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used as evaluation indicators.

[0059] As an example, the calculation formula of the peak signal-to-noise ratio can be:

[0060] In the formula, log represents the logarithmic function, MAX I It represents the maximum possible pixel value in the motion clear image after deblurring, and MSE represents the mean square error function, which is used to measure the difference between the motion blurred image and the motion clear image after deblurring.

[0061] Where x and y represent the two window areas of the motion clear image after blurring, μ x represents the pixel mean of the window area x, μ y represents the pixel mean of the window area y, represents the pixel variance of the window area x, represents the pixel variance of the window area y, σ xy represents the pixel covariance of window area x and window area y, and C1 and C2 represent two constants.

[0062] During the training of the model, the loss function is calculated for the gradient of the model. The calculated gradient can be used to update the parameters of the model in the subsequent optimization step. After the gradient calculation, the optimizer.step() function is used to update the parameters of the model, which aims to minimize the loss function. The calculation formula of the loss function in the improved motion image deblurring model can be:

[0063] loss = deblur_loss + res_loss * w; where deblur_loss represents the loss of deblurring, res_loss represents the loss of the predicted residual sequence, and w represents the weight vector.

[0064] deblur_loss = ||D(N)-Gd(N)|| 2 ;res_loss=||E(q)-Gr(q)|| 2 ; In the formula, D(N) represents the motion clear image after deblurring, Gd(N) represents the real motion clear image, E(q) represents the predicted residual sequence, and Gr(q) represents the real residual sequence.

[0065] In addition, it is worth noting that the implementation process of the improved motion image deblurring model is shown in the flowchart below: Figure 1 As shown, the overall framework diagram is as follows Figure 2 As shown, the specific steps include:

[0066] S101, based on the event stream information, obtaining pre-processed event stream information in a three-dimensional voxel format that is compatible with the image information.

[0067] It should be noted that before fusing the data information of the two modalities, the two modal data need to have the same data format. Therefore, the event stream data corresponding to the acquired motion blurred image needs to be preprocessed and converted into event stream information in a three-dimensional voxel format that is compatible with the image information.

[0068] Specifically, first, for the event flow information within the exposure time period of the motion blurred image, the event flow information is accumulated based on the principle of time symmetry to obtain a sequence containing multiple event frames; then, several event frames located in the middle of the sequence are removed, and the remaining event frames are converted into event data in a three-dimensional voxel format as the preprocessed event flow information.

[0069] In this embodiment, the event stream information is preprocessed. Unlike traditional cameras, event cameras can output a series of asynchronous sparse event data, using symmetrical cumulative event representation, which is divided into 2N equal time intervals within the blurred image exposure time, that is, divided into 2N+1 event potential frames, and N can be 3 in this embodiment; at the same time, the intermediate event potential frames are discarded and processed into a three-dimensional voxel format, that is, the image information m∈C×H×W of the blurred motion image is combined with the event stream information e∈C×H×W of the event voxels during the exposure period as the input of the improved motion image deblurring model, where C represents the number of channels, H represents the height, and W represents the width.

[0070] So far, this embodiment has obtained the preprocessed event stream information corresponding to the motion blurred image.

[0071] S102, performing multimodal bidirectional fusion processing on the image information and the preprocessed event stream information to obtain fusion features of the motion blurred image.

[0072] It should be noted that fusion features refer to the feature performance after the data information of motion images of two different modalities are fused. The multimodal bidirectional fusion processing includes multiple modules, namely the initial fusion module, the deep fusion module and the final fusion module. It is a layer-by-layer bidirectional adaptive fusion from coarse to fine, and bidirectional fusion refers to the application of what is learned in one modality to another modality for operation.

[0073] The above step S102 can be performed by Figure 3 The steps shown achieve:

[0074] S112, inputting the image information and the pre-processed event stream information into a primary fusion module for processing to obtain a first fused image feature and a first fused event feature.

[0075] In this embodiment, the first fused image feature and the first fused event feature are generated by analyzing the local features of the image and the event stream and fusing the complementary information of the two data sources to generate their own preliminary fusion features, which not only enriches the feature dimension but also provides a basis for subsequent deep fusion.

[0076] The above step S112 can be implemented by the following steps:

[0077] In the first step, two convolution operations are performed on the image information and the preprocessed event stream information respectively to obtain the local features of the image and the local features of the event.

[0078] In this embodiment, a convolutional neural network module is designed to act on the image information and the preprocessed event stream information respectively, and two convolution operations with a convolution kernel size of 5×5, a step size of 1, and a padding of 2 are performed consecutively to capture local features.

[0079] In the second step, the local features of the image and the local features of the event are normalized, and the nonlinear decision boundary of the nonlinear activation function is introduced to obtain the image feature map and the event feature map.

[0080] In this embodiment, the normalization layer is processed to ensure data consistency and accelerate the training process, and a nonlinear activation function is used to introduce a nonlinear decision boundary, thereby extracting preliminary shallow features and ensuring that the number of channels matches, laying a solid foundation for subsequent feature fusion, and obtaining an image feature map and an event feature map, the corresponding formula of which is expressed as follows:

[0081] fm =Relu(Batch(Conv 5×5 (Conv 5×5 (m))));f e =Relu(Batch(Conv 5×5 (Conv 5×5 (e))));

[0082] In the formula, f m Represents image feature map, Relu represents nonlinear activation function, Batch represents normalization function, Conv represents convolution processing function, m represents image information of motion blurred image, and e represents event stream information after preprocessing of motion blurred image.

[0083] The third step is to multiply the image feature map and the event feature map element by element to obtain a fused feature map; add the image feature map and the fused feature map to obtain a first fused image feature; add the event feature map and the fused feature map to obtain a first fused event feature.

[0084] In this embodiment, the two feature maps are multiplied element by element, and then the extracted image features and event features are added respectively, in order to fuse the complementary information of the two data sources and generate their respective preliminary fusion features, which not only enriches the feature dimension but also provides a basis for subsequent deep fusion, which can be expressed as follows:

[0085]

[0086] In the formula, represents the first fused image feature, represents the fused feature map, Represents the element-wise multiplication of two feature maps, Represents the first fusion event feature.

[0087] S122, combining the channel attention mechanism and the spatial attention mechanism to construct a deep fusion module, and performing bidirectional fusion processing on the first fused image feature and the first fused event feature respectively to obtain the second fused image feature and the second fused event feature.

[0088] In order to further optimize the feature representation, the deep fusion module is used to perform bidirectional fusion processing on the first fused image feature and the first fused event feature. The specific steps include:

[0089] In the first step, the channel attention mechanism (CA) is used to dynamically evaluate the importance of each feature channel in the first fused image feature and the first fused event feature, and the first fused image feature and the first fused event feature processed by the channel attention are obtained, which are recorded as the third fused image feature and the third fused event feature.

[0090] In this embodiment, the importance of each feature channel can be dynamically evaluated through the channel attention mechanism, and the features that are critical to the task can be enhanced accordingly, while irrelevant or redundant features can be suppressed.

[0091] As an example, the expressions of the third fused image feature and the third fused event feature may be:

[0092] In the formula, f` m Represents the third fused image feature, f` e represents the third fusion event feature, and CA represents the channel attention mechanism.

[0093] In the second step, the spatial attention mechanism (SA) is used to focus on analyzing the criticality of each position information in the third fused image feature and the third fused event feature, and the third fused image feature and the third fused event feature processed with spatial attention are obtained, which are recorded as the second fused image feature and the second fused event feature.

[0094] In this embodiment, by applying the spatial attention mechanism, focusing on the key position information in the image, and identifying and strengthening important areas, the feature representation is further refined, the depth and effectiveness of the features are improved, and the deep interaction and fusion between event data and image data features are promoted.

[0095] As an example, the expressions of the second fused image feature and the second fused event feature may be:

[0096] In the formula, represents the second fused image feature, represents the second fusion event feature, and SA represents the spatial attention mechanism.

[0097] S132, inputting the second fused image features and the second fused event features into a final fusion module, performing feature mining in the vertical and horizontal directions respectively, and obtaining fused features of the motion blurred image.

[0098] In this embodiment, the second fused image feature and the second fused event feature are subjected to deeper feature mining in the vertical and horizontal directions respectively, so as to determine whether the key feature information of the two lattice modalities can be captured in different directions, and finally the final fusion feature of the two modalities is obtained. The framework diagram of the final fusion module is shown in FIG. Figure 4 As shown, the specific implementation steps may include:

[0099] In the first step, the second fused image feature and the second fused event feature are processed by element multiplication and maximum value selection to obtain two composite features, and the two composite features are connected together to obtain a first connected feature.

[0100] As an example, the expression of the first connection feature may be:

[0101] In the formula, f out Represents the first connection feature, Concat represents the connection function, and max represents the maximum value function.

[0102] In the second step, average pooling operations are performed on the first connection feature and the first undetermined fusion feature in the vertical and horizontal directions respectively, and the obtained features are connected to obtain the second connection feature.

[0103] It should be noted that the first undetermined fusion feature is the first fusion event feature or the first fusion image feature. Take this as an example to determine the fusion event mining features. The specific implementation process can be:

[0104] First, determine the second connection feature. Perform average pooling operations on the first connection feature and the first fusion event feature in the vertical and horizontal directions, respectively, and connect the two average pooled features to obtain the second connection feature. The expression of the second connection feature can be:

[0105] Where P represents the second connection feature, AP h Represents the average pooling operation in the vertical direction, AP w Represents an average pooling operation in the horizontal direction.

[0106] In the third step, the second connected feature is processed using four convolutions and normalization operations to obtain four attention weights in the vertical and horizontal directions.

[0107] After obtaining the second connection feature, this embodiment uses four convolution and normalization operations to obtain four attention weights in the vertical and horizontal directions: Wh(e), Ww(e), Wh(o), Ww(o), which can be expressed as:

[0108] Wh(e)=σ(Conv(P)); Ww(e)=σ(Conv(P)); Wh(o)=σ(Conv(P)); Ww(o)=σ(Conv(P)).

[0109] The fourth step is to perform weighted aggregation processing on the pending fusion event feature and the first connection feature through attention weight to obtain the second pending fusion feature.

[0110] As an example, the second undetermined fusion feature is a fusion image mining feature or a fusion event mining feature, and its expression can be:

[0111] Where A2 x represents the second undetermined fusion feature, represents the feature of the pending fusion event, x represents an image or event, Wh(x) represents the attention weight of the pending fusion event feature in the vertical direction, Ww(x) represents the attention weight of the pending fusion event feature in the horizontal direction, represents the first connection feature, Wh(o) represents the attention weight of the first connection feature in the vertical direction, Ww(o) represents the attention weight of the first connection feature in the horizontal direction, Represents element-wise multiplication.

[0112] In this embodiment, Wh€ and Ww€ and f out , Wh(o) and Ww(o) are multiplied to obtain weighted features, and the two weighted features are aggregated by element-by-element addition to obtain the fusion event mining features; then, the feature f is added vertically and horizontally. out And image fusion features Use the same method to obtain the fusion event mining feature A m .

[0113] As an example, the fusion event mining feature expression can be:

[0114] In the formula, A e represents the fusion event mining feature, represents the first fused event feature, Wh(e) represents the attention weight of the event feature in the vertical direction, and Ww(e) represents the attention weight of the event feature in the horizontal direction.

[0115] The fifth step is to add the fused image mining features and the fused event mining features to obtain the fused features of the motion blurred image.

[0116] In this embodiment, the fused image mining feature A is finally m And fusion event mining feature A eThey are connected together to explore the intrinsic connection between the modal data, so that the fused features have stronger robustness and expressiveness, and the fused feature f3 of the motion blurred image is obtained, which not only contains the detailed information of the image data but also highlights the key motion information of the event data.

[0117] As an example, the expression of the fusion feature of the motion blurred image can be: f3 = A e +A m .

[0118] So far, this embodiment has obtained the fusion features of the motion blurred image by deep fusion of two modal data.

[0119] S103, taking the fused features as input, performing residual connection through the residual connection module to gradually learn and generate a residual sequence.

[0120] It should be noted that in order to achieve accurate conversion from blurred images to clear images, an efficient encoder-decoder framework is constructed through residual connections. The framework embeds a carefully designed residual block module, which can enhance the network's ability to extract and reconstruct image features through residual learning, thereby helping to improve the accuracy and quality of the conversion results, and ultimately obtaining a residual sequence rich in details and motion information.

[0121] Specifically, the fused features are first downsampled four times. Each sampling block consists of a convolutional layer and an activation function. Each downsampling doubles the number of channels and halves the input size to extract high-level features in the fused modality. Then the model passes through two residual modules to achieve accurate conversion from blurred images to clear images. After that, it is decoded through four upsampling blocks. Each upsampling block halves the number of channels and doubles the input size. Each downsampling uses a residual connection to solve the problem of information loss during downsampling. The residual learning method is used to enhance the network's ability to extract and reconstruct image features, thereby promoting the improvement of the accuracy and quality of the conversion results, and finally obtaining a residual sequence rich in detail information.

[0122] As an example, the expression of the residual sequence can be:

[0123] E(q)=g(δ[N,q],B;θ), Where E(q) represents the residual of the qth frame, δ[N,q] represents the event flow between the Nth frame and the qth frame, B represents the motion blurred image, θ represents the model parameters, and g represents the sampling process.

[0124] So far, this embodiment has obtained a residual sequence.

[0125] S104, based on the residual sequence, obtaining a deblurred motion clear image through an event-based double integral physical model.

[0126] Specifically, the residual sequence is taken as input, and the expression of the motion blurred image is derived through an event-based double integral physical model; based on the expression of the motion blurred image, the difference between the motion blurred image and the average residual sequence is calculated to obtain a clear image after deblurring.

[0127] In this embodiment, the event camera captures dynamic scenes with its unique mechanism, the core of which is the timely response to changes in light intensity. Each event information can be represented as a tuple (x i ,y i ,t i ,p i ), which contains not only spatial and temporal attributes, but also event polarity, t i Represents a timestamp, and the event is triggered at time t only when the intensity change of the pixel exceeds the set threshold compared with the pixel intensity of the last trigger. Its expression can be:

[0128] In the formula, c represents the set pixel intensity change threshold, log represents the logarithmic function, τ t represents the pixel intensity change at time t, (x i ,y i ) represents the pixel coordinates of the i-th event data, p i ∈{-1,+1} represents the polarity of the i-th event, +1 represents an increase in brightness, and -1 represents a decrease in brightness, and i represents a positive integer, which can be 1, 2, 3…

[0129] The corresponding fuzzy image expression is derived through the event-driven double integration (EDI) physical model:

[0130] Where f represents the middle moment in the exposure time, T represents the exposure time of the motion blurred image, L(f) represents the potential clear frame at time f, p(s) represents the polar component in the event stream, which describes the direction of change of pixel brightness at time s, exp represents an exponential function with a natural constant as the base, and s represents the time variable in the time interval [f, t].

[0131] The motion-blurred image can be subtracted from the average residual sequence to obtain a motion-clear image. The expression of the motion-clear image can be:

[0132] Where D(N) represents the motion clear image.

[0133] At this point, the present embodiment obtains a motion clear image after blurring. An example of a motion blurred image is shown in FIG. Figure 5 As shown in the figure, the example of the motion clear image after deblurring is as follows Figure 6 shown.

[0134] The present invention provides a motion image deblurring method based on multimodal bidirectional fusion. The method adopts the technology of multimodal fusion, interweaves blurred image data with dynamic event stream data, and realizes deep mining and complementarity of the advantages of both through a carefully designed new fusion strategy, that is, a coarse-to-fine, layer-by-layer progressive bidirectional attention fusion mechanism. This strategy not only effectively compensates for the problem of easy loss of motion information in blurred images, but also improves the accuracy of motion deblurring. In addition, the motion image deblurring method provided in this embodiment uses deep learning technology, and through the method of efficiently fusing event stream data with blurred image data, it brings new breakthroughs in the field of image deblurring, and also provides a more reliable solution for social and technological development.

[0135] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A motion image deblurring method based on multimodal bidirectional fusion, characterized in that: The following steps are involved: Acquire multimodal data information of a motion blurred image, and input the multimodal data information into an improved motion image deblurring model to obtain a motion clear image; wherein the multimodal data information includes image information and event stream information; The implementation process of the improved motion image deblurring model includes: Acquire preprocessed event stream information in a three-dimensional voxel format that is compatible with the image information based on the event stream information; The image information and the preprocessed event stream information are subjected to multimodal bidirectional fusion processing to obtain the fusion features of the motion blurred image; Taking the fused features as input, performing residual connection step by step learning through a residual connection module and generating a residual sequence; Based on the residual sequence, the deblurred motion clear image is obtained through an event-based double integration physical model.

2. The motion image deblurring method based on multimodal bidirectional fusion according to claim 1, characterized in that: The step of obtaining the preprocessed event stream information includes: Firstly, the event stream information within the exposure period of the motion blurred image is accumulated based on the principle of time symmetry to obtain a sequence containing multiple event frames. Then, several event frames located in the middle of the sequence are removed, and the remaining event frames are converted into event data in a three-dimensional voxel format as preprocessed event stream information.

3. The motion image deblurring method based on multimodal bidirectional fusion according to claim 1, characterized in that: The image information and the pre-processed event stream information are subjected to multi-modal bidirectional fusion processing to obtain fusion features, including: Inputting the image information and the preprocessed event stream information into the primary fusion module for processing to obtain a first fused image feature and a first fused event feature; A deep fusion module is constructed by combining the channel attention mechanism and the spatial attention mechanism, and the first fused image feature and the first fused event feature are bidirectionally fused to obtain a second fused image feature and a second fused event feature; The second fused image feature and the second fused event feature are input into the final fusion module, and feature mining is performed in the vertical and horizontal directions respectively to obtain the fused features of the motion blurred image.

4. The motion image deblurring method based on multimodal bidirectional fusion according to claim 3, characterized in that: The step of inputting the image information and the pre-processed event stream information into the primary fusion module for processing to obtain the first fused image feature and the first fused event feature includes: Perform two convolution operations on the image information and the preprocessed event stream information respectively to obtain the local features of the image and the local features of the event; Normalizing the local features of the image and the local features of the event, and introducing a nonlinear decision boundary of a nonlinear activation function to obtain an image feature map and an event feature map; The image feature map and the event feature map are multiplied element by element to obtain a fused feature map; the image feature map and the fused feature map are added to obtain a first fused image feature; the event feature map and the fused feature map are added to obtain a first fused event feature.

5. The motion image deblurring method based on multimodal bidirectional fusion according to claim 4, characterized in that: The deep fusion module is constructed by combining the channel attention mechanism and the spatial attention mechanism, and the first fused image feature and the first fused event feature are subjected to bidirectional fusion processing respectively to obtain the second fused image feature and the second fused event feature, including: The importance of each feature channel in the first fused image feature and the first fused event feature is dynamically evaluated by using the channel attention mechanism, and the first fused image feature and the first fused event feature processed by the channel attention are obtained, which are recorded as the third fused image feature and the third fused event feature; The spatial attention mechanism is used to focus on analyzing the criticality of each position information in the third fused image feature and the third fused event feature, and the third fused image feature and the third fused event feature processed with spatial attention are obtained, which are recorded as the second fused image feature and the second fused event feature.

6. The motion image deblurring method based on multimodal bidirectional fusion according to claim 5, characterized in that: The step of inputting the second fused image feature and the second fused event feature into the final fusion module, performing feature mining in the vertical and horizontal directions respectively, and obtaining the fused features of the motion blurred image includes: The second fused image feature and the second fused event feature are processed by element multiplication and maximum value selection to obtain two composite features, and the two composite features are connected together to obtain a first connection feature; Performing average pooling operations on the first connection feature and the first undetermined fusion feature in the vertical and horizontal directions respectively, connecting the obtained features to obtain a second connection feature; wherein the first undetermined fusion feature is a first fusion event feature or a first fusion image feature; The second connected feature is processed using four convolutions and normalization operations to obtain four attention weights in the vertical and horizontal directions; Performing weighted aggregation processing on the pending fusion event feature and the first connection feature through the attention weight to obtain a second pending fusion feature; wherein the second pending fusion feature is a fusion image mining feature or a fusion event mining feature; The fused image mining feature and the fused event mining feature are added to obtain the fused feature of the motion blurred image.

7. The motion image deblurring method based on multimodal bidirectional fusion according to claim 6, characterized in that: The weighted aggregation processing of the pending fusion event feature and the first connection feature by using the attention weight to obtain the second pending fusion feature includes: Where A2 x represents the second undetermined fusion feature, represents the feature of the pending fusion event, x represents an image or event, Wh(x) represents the attention weight of the pending fusion event feature in the vertical direction, Ww(x) represents the attention weight of the pending fusion event feature in the horizontal direction, represents the first connection feature, Wh(o) represents the attention weight of the first connection feature in the vertical direction, Ww(o) represents the attention weight of the first connection feature in the horizontal direction, Represents element-wise multiplication.

8. The motion image deblurring method based on multimodal bidirectional fusion according to claim 1, characterized in that: The method of taking the fusion feature as input, performing residual connection step by step learning through a residual connection module and generating a residual sequence includes: First, the fused features are downsampled four times to obtain the features extracted by each downsampling; each sampling block consists of a convolutional layer and an activation function; The features extracted by each downsampling pass through two residual modules and then are decoded through four upsampling blocks to obtain a residual sequence.

9. The motion image deblurring method based on multimodal bidirectional fusion according to claim 1, characterized in that: The method of obtaining a clear image after deblurring based on a residual sequence through an event-based double integral physical model includes: The residual sequence is taken as input, and the expression form of the motion blurred image is derived through an event-based double integral physical model; based on the expression form of the motion blurred image, the difference between the motion blurred image and the average residual sequence is calculated to obtain a clear image after deblurring.

10. The motion image deblurring method based on multimodal bidirectional fusion according to claim 1, characterized in that: The calculation formula of the loss function of the improved motion image deblurring model is: loss = deblur_loss + res_loss * w; where deblur_loss represents the loss of deblurring, res_loss represents the loss of the predicted residual sequence, and w represents the weight vector; deblur_loss = ||D(N)-Gd(N)|| 2 ;res_loss=||E(q)-Gr(q)|| 2 ; In the formula, D(N) represents the motion clear image after deblurring, Gd(N) represents the real motion clear image, E(q) represents the predicted residual sequence, and Gr(q) represents the real residual sequence.

Images