Motion deblurring method for multimodal fusion images based on event-driven
Through event-driven multimodal fusion defuzzing network, deep space-time event voxel grid and frequency separation technology, combined with high-frequency information fusion module, the problem of insufficient recovery of uneven motion blur image is solved, and clear and sharp images are achieved. Recovery and maintaining scene authenticity exceed the performance of the prior art.
Patent Information
- Application Number
- CN202411372115.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-29
AI Technical Summary
The prior art does not recover enough details when processing uneven motion blur images and ignores the spectral characteristics of event modes and image modes, resulting in the image being oversharpened or losing its authenticity.
Design a multimodal fusion defuzzy network based on event-driven, adopts deep spatiotemporal event voxel grid and frequency separation method, and combines event-image multimodal high-frequency fusion enhancement module and frequency fusion module to effectively fusion event streams and degraded images high-frequency information, restores clear and sharp edges and preserves scene authenticity.
It realizes efficient debuffing in severely blurred areas, restores high frame rate clear videos, and maintains image authenticity and details, which exceeds the performance of existing methods.
Smart Images

Figure CN118887135B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an event-driven multi-modal fusion image motion deblurring method. Background Art
[0002] Motion blur refers to the phenomenon of blurring in images when shooting or observing fast-moving objects because the camera captures the moving trajectory of the object during the exposure time. This blurring can cause the loss of fine details in the scene, especially in the case of uneven blur, which makes the application of existing deblurring methods in real-world scenarios challenging. This degradation causes the information originally contained in the image or video to be damaged or lost, thus affecting the subsequent processing and application of the image.
[0003] Event cameras have unique capabilities that are different from traditional frame cameras. They are particularly good at sensing very fast motion. Their dynamic vision sensors detect changes in the intensity of radiation in the scene and encode useful motion information that is lost in the regular imaging process of traditional cameras with microsecond time resolution, outputting an asynchronous event stream.
[0004] Currently, deblurring methods for event-based cameras include:
[0005] In the paper "Event-Based Fusion for Motion Deblurring with Cross-modal Attention" published in the journal "Proceedings of the IEEE conference on computer vision and pattern recognition", a multi-level event-image cross-modal attention module applied to the network is designed, which allows to focus on the relevant features of the event branch and filter out noise. In addition, they also introduced a new symmetric cumulative event representation specifically for image deblurring. However, in real environments, recovering subtle details from non-uniformly blurred images remains a challenging task.
[0006] The paper "Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera" published in the journal "Proceedings of the IEEE / CVF International Conference on Computer Vision" proposes that event-based cameras can measure intensity changes (called "events") with microsecond accuracy under high-speed motion and challenging lighting conditions. Using active pixel sensors (APS), event cameras allow the simultaneous output of image frames and event streams. However, the output image is captured at a relatively low frame rate and has motion blur. The degraded image can be viewed as the integral of a series of latent images, while events represent the changes between latent images.
[0007] At present, several event-based deblurring methods have been proposed, proving the effectiveness of cross-modal events with rich motion information for image restoration. However, these methods only emphasize event representation and the fusion of cross-modal data, and still have shortcomings such as noise damage, threshold uncertainty, and inability to effectively combine degraded images with high-frequency information of event streams. In short, the current deblurring methods have two drawbacks:
[0008] 1. These methods do not emphasize how to deal with non-uniform blur in degraded images, resulting in insufficient recovery of severely blurred areas.
[0009] 2. These methods ignore the spectral characteristics of the event modality and the image modality. Directly combining the two modalities will result in overly sharp edges while enhancing details and losing the authenticity of the image.
[0010] Therefore, recovering subtle details from non-uniformly blurred degraded images in the real world remains a challenging task. Summary of the invention
[0011] Traditional frame-based cameras are prone to uneven blurring in real scenes due to their "frame" imaging method. In order to solve this problem, the present invention adopts an event camera based on bionics, which can capture the brightness changes of pixels like the human eye and record the motion changes of the scene with extremely high temporal resolution, thereby effectively solving the motion blur problem. On this basis, the present invention designs an event-driven multimodal fusion deblurring network, which uses event streams to remove blur and achieves clear and high-quality image restoration.
[0012] Specifically, in order to achieve effective fusion of two modal data, the present invention first designs a deep spatiotemporal event voxel grid dedicated to event camera deblurring to make full use of event information. In addition, the deblurring network designed by the present invention extracts the high-frequency information of the degraded image by frequency separation, and uses a specially designed event-image multimodal high-frequency fusion enhancement module to perform modal fusion with the event information, thereby restoring the clear and sharp edges of the degraded image. This frequency separation processing method can retain the authenticity of the scene while removing severe blur and restoring the details of the degraded image. Compared with other existing methods, the method proposed by the present invention achieves excellent performance, including excellent visual performance and good weak texture detail retention.
[0013] In view of this, the present invention aims to provide an event-driven multimodal fusion image motion deblurring method to solve the problem of insufficient recovery of severely blurred areas in the prior art. The present invention utilizes the high temporal resolution characteristics of the event stream, and the studied deblurring method can well realize image blur removal under high-speed scene motion to obtain clear and sharp images.
[0014] To achieve the above object, the technical solution created by the present invention is implemented as follows:
[0015] A multimodal fusion image motion deblurring method based on event-driven, specifically comprising the following steps:
[0016] S1: Obtain a degraded image and an event stream corresponding to the degraded image, and construct a deep spatiotemporal event voxel grid based on the number, events, and density of the event stream;
[0017] S2: construct a deblurring network, which includes a UNet network and an improved UNet network;
[0018] S3: Use the loss function to train the deblurring network to obtain a trained deblurring network;
[0019] S4: The degraded image and the deep spatiotemporal event voxel grid are input into the trained deblurring network for processing to obtain a deblurred image.
[0020] Furthermore, step S1 specifically includes the following steps:
[0021] S11: Use the discrete events contained in the event stream to construct a quantity histogram:
[0022] ;
[0023] in, is the quantity histogram, is a two-dimensional pulse function, is the pixel coordinate of the kth discrete event in the degraded image, is the timestamp of the kth discrete event, is the polarity of the kth discrete event, (x, y) is the pixel coordinate of the discrete event corresponding to the current degraded image, and T is the exposure time;
[0024] Construct a time histogram using the discrete events contained in the event stream:
[0025] ;
[0026] in, is the time histogram, is the start time of the current time window;
[0027] Construct a density histogram using the discrete events contained in the event stream:
[0028] ;
[0029] ;
[0030] ;
[0031] in, is the density histogram, D(x,y) is the density mapping function, D(x,y)∈[0,1], is the density at the pixel position (x, y), j is the number of events in the event cluster, is the density function of the event cluster, and the event cluster is all events in the current time window.
[0032] S12: Integrate the quantity histogram, time histogram and density histogram into voxels in sequence to obtain a deep spatiotemporal event voxel grid.
[0033] Further, in step S2, the improved UNet network includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, a first ResBlock module, a second ResBlock module, a third ResBlock module, a first Encoder module, a second Encoder module, a third Encoder module, a first Decoder module, a second Decoder module, a third Decoder module and a SAM module, wherein,
[0034] Input the deep spatiotemporal event voxel grid into the first 3×3 convolutional layer for convolution processing to obtain feature A1, input feature A1 into the first ResBlock module for processing to obtain feature A2, input feature A2 into the second ResBlock module for processing to obtain feature A3, input feature A3 into the third ResBlock module for processing to obtain feature A4;
[0035] The degraded image is input to the second 3×3 convolutional layer for convolution processing to obtain feature B1, feature B1 and feature A2 are input to the first Encoder module for encoding processing to obtain feature B2, feature B2 and feature A3 are input to the second Encoder module for encoding processing to obtain feature B3, feature B3 and feature A4 are input to the third Encoder module for encoding processing to obtain feature B4;
[0036] Input feature B4 to the first Decoder module for decoding to obtain feature C1, perform concatenation operation on feature C1 and feature B3 to obtain feature C2, input feature C2 to the second Decoder module for decoding to obtain feature C3, perform concatenation operation on feature C3 and feature B2 to obtain feature C4, input feature C4 to the third Decoder module for decoding to obtain feature C5, input feature C5 to the SAM module for processing to obtain feature C6;
[0037] Feature C6 is input into the UNet network for processing to obtain feature C7, and feature C7 is concatenated with the degraded image to obtain a deblurred image.
[0038] Furthermore, the network structures of the first ResBlock module, the second ResBlock module and the third ResBlock module are the same, and the first ResBlock module includes a third 3×3 convolutional layer, a fourth 3×3 convolutional layer, a first Leaky ReLU activation layer, a second Leaky ReLU activation layer and a first 1×1 convolutional layer, wherein the features input to the first ResBlock module are sequentially passed through the third 3×3 convolutional layer, the first Leaky ReLU activation layer, the fourth 3×3 convolutional layer and the second Leaky ReLU activation layer, and corresponding 3×3 convolution operations, Leaky ReLU activation operations, 3×3 convolution operations and Leaky ReLU activation operations are performed to obtain feature F1, the input features of the first ResBlock module are input into the first 1×1 convolutional layer for processing to obtain feature F2, and feature F1 is added to feature F2 to obtain the output features of the first ResBlock module.
[0039] Furthermore, the network structures of the first Encoder module, the second Encoder module and the third Encoder module are the same, and the first Encoder module includes a fourth ResBlock module, a frequency separation module, an event-image multimodal high-frequency fusion enhancement module and a frequency fusion module, wherein the feature input to the first Encoder module is subjected to residual processing by the fourth ResBlock module to obtain feature G1, and feature G1 is input to the frequency separation module for processing to obtain high-frequency information of feature G2 and low-frequency information of feature G2, and the high-frequency information of feature G2 and the output feature of the first ResBlock module are input to the event-image multimodal high-frequency fusion enhancement module for feature fusion enhancement to obtain the output feature of the event-image multimodal high-frequency fusion enhancement module, and the output feature of the event-image multimodal high-frequency fusion enhancement module and the low-frequency information of feature G2 are input to the frequency fusion module for feature fusion to obtain the output feature of the first Encoder module.
[0040] Furthermore, the frequency separation module includes an average pooling layer, a UP module, a first module and a second module. The features input to the frequency separation module are pooled through the average pooling layer to obtain feature d1. Feature d1 is input to the UP module for bilinear interpolation upsampling to obtain feature d2. Feature d2 is subtracted from the input features of the frequency separation module to obtain feature d3. Feature d3 is input to the first module for processing to obtain high-frequency information of feature G2. Feature d2 is input to the second module for processing to obtain low-frequency information of feature G2.
[0041] Furthermore, the structures of the first module and the second module are the same. The first module includes a fifth 3×3 convolutional layer, a third Leaky ReLU activation layer and a sixth 3×3 convolutional layer. The features input to the first module are sequentially passed through the fifth 3×3 convolutional layer, the third Leaky ReLU activation layer and the sixth 3×3 convolutional layer, and the corresponding 3×3 convolution operations, Leaky ReLU activation operations and 3×3 convolution operations are performed to obtain the output features of the first module.
[0042] Furthermore, the event-image multimodal high-frequency fusion enhancement module includes a first Norm module, a second Norm module, a first FFN module, a second 1×1 convolution layer, a third 1×1 convolution layer, a fourth 1×1 convolution layer, a seventh 3×3 convolution layer, an eighth 3×3 convolution layer, a first GELU activation module, a second GELU activation module, a third GELU activation module, a first 3×3 deep convolution module, an SG filter module and an MLP module, wherein the high-frequency information of feature G2 is input into the first Norm module for processing to obtain feature E1, feature A2 is input into the second Norm module for processing to obtain feature E2, feature E1 is multiplied by feature E2 to obtain feature E3, feature E3 is input into the first FFN module for processing to obtain feature E4, feature E4 is processed sequentially through the second 1×1 convolution layer, the seventh 3×3 convolution layer, the first GELU activation module, the first 3×3 depth convolution module, the second GELU activation module, the eighth 3×3 convolution layer, the third GELU activation module, the SG filter module and the third 1×1 convolution layer to obtain feature E5, feature E4 is input into the fourth 1×1 convolution layer for convolution processing to obtain feature E6, feature E5 and feature E6 are added to obtain feature E7, feature E7 is input into the MLP module for processing to obtain the output feature of the event-image multimodal high-frequency fusion enhancement module.
[0043] Furthermore, the frequency fusion module includes a ninth 3×3 convolutional layer, a first batch processing normalization module, a second batch processing normalization module, a ReLU activation module, a GAP module, a fifth 1×1 convolutional layer, a tenth 3×3 convolutional layer, an eleventh 3×3 convolutional layer, a Softmax activation function module, a split function module, a CA module and a fifth 1×1 convolutional layer, and performs a concatenation operation on the output features of the event-image multimodal high-frequency fusion enhancement module and the low-frequency information of the feature G2 to obtain feature F1, and inputs feature F1 into the ninth 3×3 convolutional layer, the first batch processing normalization module, the ReLU activation module and the GAP module in sequence for processing to obtain feature F2, and inputs feature F2 into the tenth The 3×3 convolutional layer and the eleventh 3×3 convolutional layer perform convolution processing to obtain features F3 and F4, concatenate features F3 and F4 to obtain feature F5, process feature F5 successively through the Softmax activation function module and the split function module to obtain feature F6, multiply feature F6 and the output feature of the event-image multimodal high-frequency fusion enhancement module to obtain feature F7, multiply feature F6 and the low-frequency information of feature G2 to obtain feature F8, add feature F7 and feature F8 to obtain feature F9, process feature F9 successively through the CA module, the fifth 1×1 convolutional layer and the second batch processing normalization module to obtain the output feature of the frequency fusion module.
[0044] Further, in step S3, the loss function is the LI loss function.
[0045] Compared with the prior art, the invention can achieve the following beneficial effects:
[0046] (1) The present invention creates the event-driven multimodal fusion image motion deblurring method, designs a deep spatiotemporal event voxel grid, and solves the problem of uneven deblurring in image deblurring by constructing a density histogram. The present invention can model the blur generation process by associating event streams with latent images. This paper proposes a deblurring network for reconstructing high frame rate, clear video from a single degraded image frame and its event stream. Video generation is a simple non-convex optimization problem solved in a single variable.
[0047] (2) The event-driven multi-modal fusion image motion deblurring method created by the present invention adopts a strategy of frequency separation of the degraded image and then fusing the high-frequency information of the degraded image with the event modality. This can restore the severely blurred area while retaining the authenticity of the scene, bringing new inspiration to the field of event camera deblurring.
[0048] (3) The event-driven multimodal fusion image motion deblurring method created by the present invention solves the problem that the existing methods only multiply the feature tensors of the two modalities or simply connect the feature tensors of the two modalities in the channel dimension, and cannot achieve deeper feature extraction and fusion of the two modal information. The event-image multimodal high-frequency fusion enhancement module designed by the present invention adopts deep convolution to achieve modal fusion, and deep convolution is simpler than the self-attention mechanism.
[0049] (4) The present invention creates the event-driven multimodal fusion image motion deblurring method and designs a frequency fusion module. The present invention uses global average pooling and softmax operations to obtain the preliminary channel attention weights of the two frequency bands, and then applies channel-level attention weights to the frequency components. The channel attention mechanism is used to further emphasize the information-rich features, thereby achieving more robust and accurate deblurring performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:
[0051] Figure 1 A schematic diagram of a flow chart of a method for deblurring multimodal fusion image motion based on event-driven according to an embodiment of the present invention;
[0052] Figure 2 A schematic diagram of the network structure of the defuzzification network described in the embodiment of the present invention;
[0053] Figure 3 A schematic diagram of the network structure of the frequency separation module according to an embodiment of the present invention;
[0054] Figure 4 A schematic diagram of the network structure of the event-image multimodal high-frequency fusion enhancement module described in an embodiment of the present invention;
[0055] Figure 5 A schematic diagram of the network structure of the frequency fusion module described in the embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.
[0057] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0058] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.
[0059] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.
[0060] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.
[0061] like Figure 1 As shown, the present invention provides an event-driven multimodal fusion image motion deblurring method, which specifically includes the following steps:
[0062] S1: Obtain a degraded image and an event stream corresponding to the degraded image, and construct a deep spatiotemporal event voxel grid based on the number, events, and density of the event stream;
[0063] S2: construct a deblurring network, which includes a UNet network and an improved UNet network;
[0064] S3: Use the loss function to train the deblurring network to obtain a trained deblurring network;
[0065] S4: The degraded image and the deep spatiotemporal event voxel grid are input into the trained deblurring network for processing to obtain a deblurred image.
[0066] In order to bridge the gap between asynchronous events and dense networks, sparse event streams are usually converted to dense event representations. Simple stacking based on time (SBT) or number (SBN) is a common event stream processing method, but it ignores important timestamp information. The spatiotemporal voxel grid method has a wide range of applications in the field of event-based deblurring, but current voxels often only focus on the shallow spatiotemporal information of events, focusing on obtaining high-frequency contour information of the scene in the event frame. However, this method only focuses on obtaining clear and sharp edges, and often leads to oversharpening of defocused images, even non-blurred areas in the scene will become clear and sharp, or severely blurred areas cannot be restored. That is, it is not possible to effectively remove blur for different degrees of blur in different parts of the scene.
[0067] In order to solve this problem, in step S1, the present invention first proposes a novel event representation method - deep spatiotemporal event voxel grid (DSE). It is a 3D histogram with dimensions (6, W, H), where W and H are width and height respectively. This method not only encodes the number and time of event streams, but also introduces a density channel to mine deeper spatiotemporal information. Choosing a suitable measurement function is a key factor in determining the performance of event voxels. The measurement function of the present invention weights the polarity by quantity, time, and density, respectively, and represents it as a quantity histogram, a time histogram, and a density histogram.
[0068] The quantity histogram can count the events of each pixel in the time window. It mainly depicts the contour information of the scene. The quantity histogram is described as:
[0069] ;
[0070] Among them, among them, is the quantity histogram, is a two-dimensional pulse function, is the pixel coordinate of the kth discrete event in the degraded image, is the timestamp of the kth discrete event, is the polarity of the kth discrete event, (x, y) is the pixel coordinate of the discrete event corresponding to the current degraded image, and T is the exposure time. Although the quantity histogram records the event distribution of pixels, they lose the time information. Therefore, we also construct a time histogram to retain the time information of the event stream. The time histogram of the event is described as:
[0071] ;
[0072] in, is the time histogram, is the start time of the current time window, and T is the exposure time.
[0073] Temporal information encoding is achieved by polar time weighting. However, in real scenes, different areas of the degraded image have different degrees of blur. The greater the blur of the degraded image, the faster the objects in the scene move. The event camera will respond to more events because of its high temporal resolution characteristics. There will be different event densities for different areas in the scene. Areas with high event density should have larger weights for the back-end network to learn. Therefore, the present invention also constructs a density histogram to solve the problem of uneven blur in image deblurring. This is not mentioned in the existing deblurring methods. The density histogram is described as:
[0074] ;
[0075] in, is the density histogram, and D(x,y) is the density mapping function.
[0076] For the density mapping function, the present invention first sets the size of the fixed time window to k*k, and regards the events in each time window as an event cluster. And calculates the density of the event cluster , j represents the number of events in the event cluster, is the density function of the event cluster. Finally, define a density mapping function D(x,y) from the density of an event cluster to the weight, so that D(x,y)∈[0,1]. In order to simplify the calculation, the present invention directly uses the number of events in the time window to represent its time density, that is, di=j, and uses the maximum value and minimum value normalization to obtain the density mapping function:
[0077] ;
[0078] The above three histograms fully retain the polarity and fine-grained temporal information of the event and encode different blur levels in the scene. The three event histograms are integrated into voxels in the order of quantity, time, and density, and finally a deep spatiotemporal event voxel grid (DSE) with a shape size of (3K, W, H) is obtained. K represents the number of time windows. If K=2 is selected, the final voxel shape is (6, W, H).
[0079] like Figure 2 As shown in the figure, the deblurring network includes a UNet network (the same as the UNet network used in the paper "U-net: Convolutional net-works for biomedical image segmentation") and an improved UNet network. The operations in the figure are annotated as follows: Element-wise Addition is element-by-element addition, Element-wise Subtraction is element-by-element subtraction, and Element-wise Product refers to the operation of multiplying each element of two vectors or matrices of the same dimension. Concatenation refers to concatenating multiple tensors of the same dimension to form a new tensor. Downsample refers to downsampling, Upsample refers to upsampling, Channel Attention refers to the channel attention mechanism, and Global Average Pooling refers to global average pooling.
[0080] The improved UNet network includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, a first ResBlock module, a second ResBlock module, a third ResBlock module, a first Encoder module, a second Encoder module, a third Encoder module, a first Decoder module, a second Decoder module, a third Decoder module and a SAM module, wherein:
[0081] Input the deep spatiotemporal event voxel grid into the first 3×3 convolutional layer for convolution processing to obtain feature A1, input feature A1 into the first ResBlock module for processing to obtain feature A2, input feature A2 into the second ResBlock module for processing to obtain feature A3, input feature A3 into the third ResBlock module for processing to obtain feature A4;
[0082] The degraded image is input to the second 3×3 convolutional layer for convolution processing to obtain feature B1, feature B1 and feature A2 are input to the first Encoder module for encoding processing to obtain feature B2, feature B2 and feature A3 are input to the second Encoder module for encoding processing to obtain feature B3, feature B3 and feature A4 are input to the third Encoder module for encoding processing to obtain feature B4;
[0083] Input feature B4 to the first Decoder module for decoding to obtain feature C1, perform concatenation operation on feature C1 and feature B3 to obtain feature C2, input feature C2 to the second Decoder module for decoding to obtain feature C3, perform concatenation operation on feature C3 and feature B2 to obtain feature C4, input feature C4 to the third Decoder module for decoding to obtain feature C5, input feature C5 to the SAM module for processing to obtain feature C6;
[0084] The first Decoder module, the second Decoder module and the third Decoder module have the same network structure, and all include a ResBlock module having the same network structure as the first ResBlock module.
[0085] Feature C6 is input into the UNet network for processing to obtain feature C7, and feature C7 is concatenated with the degraded image to obtain a deblurred image.
[0086] Here, the network structures of the first ResBlock module, the second ResBlock module and the third ResBlock module are the same. Taking the first ResBlock module as an example, the first ResBlock module includes a third 3×3 convolutional layer, a fourth 3×3 convolutional layer, a first Leaky ReLU activation layer, a second Leaky ReLU activation layer and a first 1×1 convolutional layer. The features input to the first ResBlock module are sequentially processed by the third 3×3 convolutional layer, the first Leaky ReLU activation layer, the fourth 3×3 convolutional layer and the second Leaky ReLU activation layer, and the corresponding 3×3 convolution operation, Leaky ReLU activation operation, 3×3 convolution operation and Leaky ReLU activation operation are performed to obtain feature F1. The input features of the first ResBlock module are input into the first 1×1 convolutional layer for processing to obtain feature F2. Feature F1 is added to feature F2 to obtain the output features of the first ResBlock module.
[0087] The network structures of the first Encoder module, the second Encoder module and the third Encoder module are the same. Taking the first Encoder module as an example, the first Encoder module includes a fourth ResBlock module, a frequency separation module (FS), an event-image multimodal high-frequency fusion enhancement module (EIHE) and a frequency fusion module (Fusion), wherein the feature input to the first Encoder module is subjected to residual processing by the fourth ResBlock module to obtain feature G1, and feature G1 is input to the frequency separation module for processing to obtain high-frequency information of feature G2 and low-frequency information of feature G2, and the high-frequency information of feature G2 and the output feature of the first ResBlock module are input to the event-image multimodal high-frequency fusion enhancement module for feature fusion enhancement to obtain the output feature of the event-image multimodal high-frequency fusion enhancement module, and the output feature of the event-image multimodal high-frequency fusion enhancement module and the low-frequency information of feature G2 are input to the frequency fusion module for feature fusion to obtain the output feature of the first Encoder module.
[0088] The present invention designs the deblurring network as a two-level network to achieve the purpose of progressively restoring a clear image from degraded images and deep spatiotemporal event voxels. Each subnet (UNet network or improved UNet network) follows a U-shaped architecture, and each level of the subnet consists of an encoder-decoder structure with two downsampling and upsampling layers. Downsampling is performed using convolutions with a kernel size of 4, and upsampling is performed using transposed convolutions with a kernel size of 2. Between the encoder and the decoder, a 3×3 convolutional skip connection is added. For simplicity, we Figure 2Only the skip connections between the encoder and decoder of each layer are shown in . The present invention uses a supervised attention module (SAM) to connect the transfer function between two nodes. Specifically, in each encoder of the improved UNet network, a ResBlock module is first used to extract high-level features from the event and degraded images. Then, a frequency separation module is used to retain the low-frequency information of the degraded image while extracting the high-frequency part of the degraded image. These high-frequency parts are fused with the event information through a high-frequency enhancement block for modal fusion and high-frequency enhancement. Finally, the frequency fusion block aggregates high-frequency and low-frequency information. In the encoder and all decoder components of the second stage, we use ResBlocks to extract high-level features, and the decoder fuses features from the encoder components to compensate for the information loss caused by reordering. The ResBlock modules in the subnets are composed of two 3×3 convolutional layers and a Leaky ReLU activation layer with a negative slope equal to 0.2. Finally, a module with a 1×1 convolution is used for skip connection to obtain the residual output. Here, a supervised attention module (SAM) is used to connect the transfer features between the two subnets. In particular, in each encoder of the first-level network (improved UNet network), the ResBlock module is first used to extract high-level features of events and degraded images. Then, the frequency separation module is used to retain the low-frequency information of the degraded image, while extracting the high-frequency part of the degraded image, and together with the event information, the event-image multimodal high-frequency fusion enhancement module is used to perform modal fusion and high-frequency enhancement. Finally, the high- and low-frequency information is aggregated through the frequency fusion module to generate a clear, sharp, and realistic deblurred image.
[0089] like Figure 3 As shown, the frequency separation module includes an average pooling layer, a UP module, a first module and a second module. The features input to the frequency separation module are pooled through the average pooling layer to obtain feature d1, and feature d1 is input to the UP module for bilinear interpolation upsampling to obtain feature d2. Feature d2 is subtracted from the input features of the frequency separation module to obtain feature d3. Feature d3 is input to the first module for processing to obtain high-frequency information of feature G2, and feature d2 is input to the second module for processing to obtain low-frequency information of feature G2.
[0090] The structures of the first module and the second module are the same. The first module includes the fifth 3×3 convolutional layer, the third LeakyReLU activation layer and the sixth 3×3 convolutional layer. The features input to the first module are sequentially passed through the fifth 3×3 convolutional layer, the third LeakyReLU activation layer and the sixth 3×3 convolutional layer, and the corresponding 3×3 convolution operations, Leaky ReLU activation operations and 3×3 convolution operations are performed to obtain the output features of the first module.
[0091] Compared with the potential degraded image, the degraded image is mainly the difference in high-frequency information in the degraded image, and the motion information of the scene recorded by the event camera at a time resolution of microseconds just contains the high-frequency components used to remove blur. Therefore, the present invention believes that extracting the high-frequency components of the degraded image separately and fusing them with the event voxels for modal fusion can achieve better high-frequency enhancement effects. Based on this idea, firstly, the present invention performs feature separation and feature extraction on the degraded image in the frequency domain, and clearly separates and strengthens the high-frequency information and the low-frequency information through the pooling layer to discriminately determine which features of the low-frequency information should be retained, and the high-frequency information will participate in the modal fusion with the event information in the next stage of the deblurring network, thereby restoring a clear image.
[0092] For the input feature B of the frequency separation module:
[0093] ;
[0094] in, is a set of real numbers, and c×h×w is the dimension of the degraded image.
[0095] The input features of the frequency separation module are downsampled using average pooling to obtain low-resolution low-frequency features:
[0096] ;
[0097] In order to facilitate subsequent processing, the low-frequency information is restored to a feature map of the same size as the degraded image through bilinear interpolation:
[0098] ;
[0099] The high frequency components are obtained by the following formula , the separation and extraction process of high and low frequency information can be described as:
[0100] ;
[0101] ;
[0102] ;
[0103] Among them, Avgpool refers to the average pooling operation. To perform average pooling downsampling on the input feature map to obtain low-resolution low-frequency information, represents the bilinear interpolation upsampling operation, is low frequency information, High frequency information.
[0104] Finally, the present invention encodes the two frequency domain information through 3×3 convolution with activation function to achieve further deep feature extraction. The equation is as follows:
[0105] ;
[0106] ;
[0107] ;
[0108] in, represents the bilinear interpolation upsampling operation, is a 3×3 convolution operation, and are the high-frequency information and low-frequency information of the encoded output features, respectively. For encoding operations, is the corresponding nonlinear Leaky ReLU activation function.
[0109] like Figure 4 As shown, the event-image multimodal high-frequency fusion enhancement module includes a first Norm module, a second Norm module, a first FFN module, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, a seventh 3×3 convolutional layer, an eighth 3×3 convolutional layer, a first GELU activation module, a second GELU activation module, a third GELU activation module, a first 3×3 deep convolutional module, an SG filter module and an MLP module, wherein the high-frequency information of feature G2 is input into the first Norm module for processing to obtain feature E1, feature A2 is input into the second Norm module for processing to obtain feature E2, feature E1 is multiplied by feature E2 to obtain feature E 3. Input feature E3 to the first FFN module for processing to obtain feature E4. Process feature E4 in turn through the second 1×1 convolution layer, the seventh 3×3 convolution layer, the first GELU activation module, the first 3×3 depth convolution module, the second GELU activation module, the eighth 3×3 convolution layer, the third GELU activation module, the SG filter module and the third 1×1 convolution layer to obtain feature E5. Input feature E4 to the fourth 1×1 convolution layer for convolution processing to obtain feature E6. Add feature E5 and feature E6 to obtain feature E7. Input feature E7 to the MLP module for processing to obtain the output feature of the event-image multimodal high-frequency fusion enhancement module.
[0110] Jointly extracting features from event streams and degraded images and fusing complementary information is the key to event-guided image deblurring. Simply multiplying the feature tensors of the two modalities or simply connecting the feature tensors of the two modalities in the channel dimension cannot achieve deeper feature extraction and fusion of the two modal information. The input of the event-image multimodal high-frequency fusion enhancement module is the high-frequency information of the features extracted from the frequency separation module. The features extracted after residual encoding are of the same size. The feature maps of the two modalities are fed to the normalization layer for processing at the same time, and the processing results of the two are multiplied:
[0111] ;
[0112] Here, Norm is the normalization layer. The number of channels of the input tensor is c, and E is the feature extracted after residual encoding. is the multiplication result.
[0113] First, the feedforward neural network, the first FFN module, is used to expand variability (the convolution parameters in all convolution operations are equivalent to variables in a function. As the number of parameters increases, the number of parameters that can be changed during the fitting process increases, i.e., variability) and amplify the high-frequency signals of the features, thereby generating a richer feature representation. That is:
[0114] ;
[0115] ;
[0116] Here, Represents the specific operation of the feedforward neural network, GELU is the GELU activation function, It is an intermediate parameter set during the calculation process and has no actual meaning.
[0117] Then use the 1×1 convolution layer to increase the number of channels of the deblurring network:
[0118] ;
[0119] in, is a 1×1 convolution operation, It is an intermediate parameter set during the calculation process and has no actual meaning.
[0120] A deep convolution module with a kernel size of 3 is used to process information, and the number of deep convolution groups is set to the number of channels. In addition, we set a 3×3 convolution and GELU activation function before and after the deep convolution to increase the network depth:
[0121] ;
[0122] ;
[0123] in, is the convolution enhancement operation, represents a depth-wise separable 3×3 convolution, and the number of depth-wise convolution groups is set to 2c. It is an intermediate parameter set during the calculation process and has no actual meaning.
[0124] Simple gating and a 1×1 convolution are used to restore the number of channels and filter information, and MLP (Multilayer Perceptron) and skip connections are used to learn the complex mapping relationship in the deblurring network. The nonlinear activation after the convolution layer is omitted, and the formula for the whole process is as follows:
[0125] ;
[0126] in, It is a multi-layer perception operation, and SG is a simple gating operation. The SG filter can halve the number of channels and restore them to the original input channels. It is the output result of EIHE.
[0127] like Figure 5 As shown in FIG. 1 , the frequency fusion module includes a ninth 3×3 convolutional layer, a first batch processing normalization module, a second batch processing normalization module, a ReLU activation module, a GAP module, a fifth 1×1 convolutional layer, a tenth 3×3 convolutional layer, an eleventh 3×3 convolutional layer, a Softmax activation function module, a split function module, a CA module, and a fifth 1×1 convolutional layer. The output features of the event-image multimodal high-frequency fusion enhancement module and the low-frequency information of the feature G2 are concatenated to obtain feature F1, and feature F1 is sequentially input into the ninth 3×3 convolutional layer, the first batch processing normalization module, the ReLU activation module, and the GAP module for processing to obtain feature F2, and feature F2 is respectively input into the tenth 3×3 convolutional layer, the first batch processing normalization module, the ReLU activation module, and the GAP module for processing to obtain feature F2. ×3 convolutional layer and the eleventh 3×3 convolutional layer are convolutionally processed to obtain features F3 and F4, features F3 and F4 are concatenated to obtain feature F5, feature F5 is processed by the Softmax activation function module and the split function module in sequence to obtain feature F6, feature F6 is multiplied by the output feature of the event-image multimodal high-frequency fusion enhancement module to obtain feature F7, feature F6 is multiplied by the low-frequency information of feature G2 to obtain feature F8, feature F7 and feature F8 are added to obtain feature F9, feature F9 is processed by the CA module, the fifth 1×1 convolutional layer and the second batch processing normalization module in sequence to obtain the output feature of the frequency fusion module.
[0128] In the frequency fusion module, the present invention fuses the low-frequency features retained in the degraded image and the high-frequency features enhanced after modal fusion to obtain a potentially clear and sharp image. Simply fusing features in different frequency domains through basic convolutional layers may result in information loss and insufficient fusion. The effectiveness of channel attention has been verified in image restoration tasks. In addition, the present invention connects the channels of high and low frequency information and learns the attention weights of different channels to emphasize and merge the beneficial frequency components.
[0129] For high-frequency information and low-frequency information:
[0130] ;
[0131] ;
[0132] in, is the low-frequency information of feature G2, It is the output feature of the event-image multimodal high-frequency fusion enhancement module.
[0133] The present invention connects the channels of high-frequency and low-frequency information and then merges the channels through a 3×3 convolution:
[0134] ;
[0135] Among them, ReLU is the ReLU activation function, BN is the batch normalization operation, and Concatenate is the channel cascade operation. is the output feature of the event-image multimodal high-frequency fusion enhancement module used to restore the clear and sharp edges of the scene. The low-frequency information of feature G2 retained for the original image is used to restore the authenticity of the degraded image scene. is the calculation result.
[0136] Next, the present invention calculates the preliminary channel attention weights of the two frequency bands as follows:
[0137] ;
[0138] in, is the global average pooling operation, is the initial channel attention weight, i=1 represents high-frequency information, i=2 represents low-frequency information, It is a 3×3 convolution operation.
[0139] Then apply the Softmax function to derive the final attention weights:
[0140] ;
[0141] Where [k, k] represents the cascade operation, c is the channel index, and They are the final attention weights of high-frequency information and low-frequency information, respectively.
[0142] After applying channel-level attention weights to the frequency components, the channel attention mechanism is used to further emphasize the information-rich features and achieve more robust and accurate deblurring performance. The final output of the frequency fusion module can be generated in the following way:
[0143] ;
[0144] in, is the output feature of the frequency fusion module, For channel attention.
[0145] The predicted image is obtained using the deblurring network The present invention uses the L1 loss function to constrain the difference between the predicted image and the ideal image without blur. ,Right now:
[0146] ;
[0147] in, For an ideal image without blur, is the predicted image.
[0148] During the training process, the dataset used by the present invention is the same as the dataset used by the EFNEet network. The present invention has been fully trained on the GoPro dataset and fine-tuned on the REBlur dataset. GoPro dataset: A dataset collected by a real event sensor, which can only provide low-resolution and low-quality images and is not suitable as ground truth images for training. Therefore, the event version of the GoPro dataset is widely used to train the corresponding network. The dataset of the present invention includes 3214 pairs of degraded and clear images. The present invention uses 2103 images for training and 1111 images for testing. The event data is simulated by ESIM (an open source event camera simulator). REBlur dataset: This dataset is constructed by capturing events, clear images and degraded images of real-world scenes in a high-precision optical laboratory, and contains 12 linear and nonlinear motion modes. The dataset includes 36 sequences and 1469 pairs of degraded-clear images and their corresponding events, of which 486 pairs are used for training and 983 pairs are used for testing. By using a precisely controlled electronic slider system, the dataset is able to capture degraded images under high-speed motion and provide reliable correspondence through time-aligned event streams.
[0149] The present invention crops the degraded image into a square image of 256×256 size. For data enhancement, horizontal and vertical flipping, random noise, and hot pixels in event voxels are applied. The cosine learning rate strategy adopted in the network training of the present invention has an initial learning rate of 2×10 –4 , the minimum learning rate is 10 –7 The optimizer used is the Adam algorithm, with the following parameters: =0.9 and =0.99. The network input channel is set to C=64.
[0150] The proposed model was implemented in Pytorch version 2.1.1 and python version 3.11.5, with 300,000 iterations on 4 NVIDIA GeForce RTX 3090 GPUs, CUDA and CuDNN versions 11.5 and 8.2.0 respectively. The training batch size was 8 and it took 41 hours.
[0151] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.
[0152] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A multimodal fusion image motion deblurring method based on event-driven, characterized by: The specific steps include: S1: Obtain a degraded image and an event stream corresponding to the degraded image, and construct a deep spatiotemporal event voxel grid based on the number, events, and density of the event stream; S11: Use the discrete events contained in the event stream to construct a quantity histogram: ; in, is the quantity histogram, is a two-dimensional pulse function, is the pixel coordinate of the kth discrete event in the degraded image, is the timestamp of the kth discrete event, is the polarity of the kth discrete event, (x, y) is the pixel coordinate of the discrete event corresponding to the current degraded image, and T is the exposure time; Construct a time histogram using the discrete events contained in the event stream: ; in, is the time histogram, is the start time of the current time window, T is the time range of the event stream within the exposure time; Construct a density histogram using the discrete events contained in the event stream: ; ; ; in, is the density histogram, D(x,y) is the density mapping function, D(x,y)∈[0,1], is the density at the pixel position (x, y), j is the number of events in the event cluster, is the density function of the event cluster, where the event cluster is all events in the current time window; S12: Integrate the quantity histogram, time histogram and density histogram into voxels in sequence to obtain a deep spatiotemporal event voxel grid; S2: constructing a deblurring network, wherein the deblurring network includes a UNet network and an improved UNet network; The improved UNet network includes a first 3×3 convolutional layer, a second 3×3 convolutional layer, a first ResBlock module, a second ResBlock module, a third ResBlock module, a first Encoder module, a second Encoder module, a third Encoder module, a first Decoder module, a second Decoder module, a third Decoder module and a SAM module, wherein: Input the deep spatiotemporal event voxel grid into the first 3×3 convolutional layer for convolution processing to obtain feature A1, input feature A1 into the first ResBlock module for processing to obtain feature A2, input feature A2 into the second ResBlock module for processing to obtain feature A3, input feature A3 into the third ResBlock module for processing to obtain feature A4; The degraded image is input to the second 3×3 convolutional layer for convolution processing to obtain feature B1, feature B1 and feature A2 are input to the first Encoder module for encoding processing to obtain feature B2, feature B2 and feature A3 are input to the second Encoder module for encoding processing to obtain feature B3, feature B3 and feature A4 are input to the third Encoder module for encoding processing to obtain feature B4; Input feature B4 to the first Decoder module for decoding to obtain feature C1, perform concatenation operation on feature C1 and feature B3 to obtain feature C2, input feature C2 to the second Decoder module for decoding to obtain feature C3, perform concatenation operation on feature C3 and feature B2 to obtain feature C4, input feature C4 to the third Decoder module for decoding to obtain feature C5, input feature C5 to the SAM module for processing to obtain feature C6; Input feature C6 into the UNet network for processing to obtain feature C7, and perform concatenation operation on feature C7 and the degraded image to obtain a deblurred image; S3: Performing network training on the deblurring network using a loss function to obtain a trained deblurring network; S4: The degraded image and the deep spatiotemporal event voxel grid are input into the trained deblurring network for processing to obtain a deblurred image.
2. The event-driven multimodal fusion image motion deblurring method according to claim 1, characterized in that: The network structures of the first ResBlock module, the second ResBlock module and the third ResBlock module are the same. The first ResBlock module includes a third 3×3 convolutional layer, a fourth 3×3 convolutional layer, a first Leaky ReLU activation layer, a second Leaky ReLU activation layer and a first 1×1 convolutional layer. The features input to the first ResBlock module are sequentially processed by the third 3×3 convolutional layer, the first Leaky ReLU activation layer, the fourth 3×3 convolutional layer and the second Leaky ReLU activation layer, and the corresponding 3×3 convolution operation, Leaky ReLU activation operation, 3×3 convolution operation and Leaky ReLU activation operation are performed to obtain feature F1. The input features of the first ResBlock module are input into the first 1×1 convolutional layer for processing to obtain feature F2. The features F1 and F2 are added to obtain the output features of the first ResBlock module.
3. The event-driven multi-modal fusion image motion deblurring method according to claim 1, characterized in that: The network structures of the first Encoder module, the second Encoder module and the third Encoder module are the same. The first Encoder module includes a fourth ResBlock module, a frequency separation module, an event-image multimodal high-frequency fusion enhancement module and a frequency fusion module. The feature input to the first Encoder module is subjected to residual processing by the fourth ResBlock module to obtain feature G1, and the feature G1 is input to the frequency separation module for processing to obtain high-frequency information of feature G2 and low-frequency information of feature G2. The high-frequency information of feature G2 and the output feature of the first ResBlock module are input to the event-image multimodal high-frequency fusion enhancement module for feature fusion enhancement to obtain the output feature of the event-image multimodal high-frequency fusion enhancement module. The output feature of the event-image multimodal high-frequency fusion enhancement module and the low-frequency information of feature G2 are input to the frequency fusion module for feature fusion to obtain the output feature of the first Encoder module.
4. The event-driven multi-modal fusion image motion deblurring method according to claim 3, characterized in that: The frequency separation module includes an average pooling layer, a UP module, a first module and a second module. The features input to the frequency separation module are pooled through the average pooling layer to obtain feature d1. Feature d1 is input to the UP module for bilinear interpolation upsampling to obtain feature d2. Feature d2 is subtracted from the input features of the frequency separation module to obtain feature d3. Feature d3 is input to the first module for processing to obtain high-frequency information of feature G2. Feature d2 is input to the second module for processing to obtain low-frequency information of feature G2.
5. The event-driven multimodal fusion image motion deblurring method according to claim 4, characterized in that: The structures of the first module and the second module are the same. The first module includes the fifth 3×3 convolutional layer, the third Leaky ReLU activation layer and the sixth 3×3 convolutional layer. The features input to the first module are sequentially passed through the fifth 3×3 convolutional layer, the third Leaky ReLU activation layer and the sixth 3×3 convolutional layer, and the corresponding 3×3 convolution operations, Leaky ReLU activation operations and 3×3 convolution operations are performed to obtain the output features of the first module.
6. The event-driven multi-modal fusion image motion deblurring method according to claim 3, characterized in that: The event-image multimodal high-frequency fusion enhancement module includes a first Norm module, a second Norm module, a first FFN module, a second 1×1 convolution layer, a third 1×1 convolution layer, a fourth 1×1 convolution layer, a seventh 3×3 convolution layer, an eighth 3×3 convolution layer, a first GELU activation module, a second GELU activation module, a third GELU activation module, a first 3×3 deep convolution module, an SG filter module and an MLP module, wherein the high-frequency information of feature G2 is input into the first Norm module for processing to obtain feature E1, feature A2 is input into the second Norm module for processing to obtain feature E2, and feature E1 is multiplied by feature E2 to obtain feature E3. , feature E3 is input to the first FFN module for processing to obtain feature E4, feature E4 is processed sequentially through the second 1×1 convolution layer, the seventh 3×3 convolution layer, the first GELU activation module, the first 3×3 depth convolution module, the second GELU activation module, the eighth 3×3 convolution layer, the third GELU activation module, the SG filter module and the third 1×1 convolution layer to obtain feature E5, feature E4 is input to the fourth 1×1 convolution layer for convolution processing to obtain feature E6, feature E5 and feature E6 are added to obtain feature E7, feature E7 is input to the MLP module for processing to obtain the output feature of the event-image multimodal high-frequency fusion enhancement module.
7. The event-driven multi-modal fusion image motion deblurring method according to claim 3, characterized in that: The frequency fusion module includes a ninth 3×3 convolutional layer, a first batch processing normalization module, a second batch processing normalization module, a ReLU activation module, a GAP module, a fifth 1×1 convolutional layer, a tenth 3×3 convolutional layer, an eleventh 3×3 convolutional layer, a Softmax activation function module, a split function module, a CA module and a fifth 1×1 convolutional layer, performs a concatenation operation on the output features of the event-image multimodal high-frequency fusion enhancement module and the low-frequency information of the feature G2 to obtain feature F1, and inputs feature F1 into the ninth 3×3 convolutional layer, the first batch processing normalization module, the ReLU activation module and the GAP module in sequence for processing to obtain feature F2, and inputs feature F2 into the tenth 3×3 convolutional layer, the first batch processing normalization module, the ReLU activation module and the GAP module respectively ... ×3 convolutional layer and the eleventh 3×3 convolutional layer are convolutionally processed to obtain features F3 and F4, features F3 and F4 are concatenated to obtain feature F5, feature F5 is processed by the Softmax activation function module and the split function module in sequence to obtain feature F6, feature F6 is multiplied by the output feature of the event-image multimodal high-frequency fusion enhancement module to obtain feature F7, feature F6 is multiplied by the low-frequency information of feature G2 to obtain feature F8, feature F7 and feature F8 are added to obtain feature F9, feature F9 is processed by the CA module, the fifth 1×1 convolutional layer and the second batch processing normalization module in sequence to obtain the output feature of the frequency fusion module.
8. The event-driven multi-modal fusion image motion deblurring method according to claim 1, characterized in that: In step S3, the loss function is the LI loss function.
