Event-driven image motion deblurring method

Abstract

The invention provides an event-driven image motion deblurring method. The event-driven image motion deblurring method comprises the following steps: S1, acquiring a deblurring data set comprising a blurred image, a clear image and an event stream within exposure time; s2, performing preprocessing by adopting a cyclic event representation method, and converting an event stream into an event frame similar to a potential real image; and S3, constructing a deblurring network by adopting linear deformable convolution and a multi-branch feature extraction attention mechanism. And S4, inputting the fuzzy data set into the trained deblurring network to obtain a clear image after deblurring. According to the method, the high time resolution characteristic of the event stream data is utilized, and an advanced convolution and attention mechanism is combined, so that a clear image can be effectively recovered from a motion blurred image.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing technology, and specifically relates to an event-driven image motion deblurring method, which achieves efficient motion blur removal in complex dynamic scenes by combining high temporal resolution data from an event camera with a deep learning network. Background Art

[0002] Traditional image deblurring methods rely on optical flow or physical models between adjacent frames to estimate the blur kernel, but their effectiveness is limited in high-speed motion or complex background scenes. Event cameras can capture microsecond-level dynamic information by asynchronously outputting pixel-level brightness changes (event streams), providing high-precision timing clues for deblurring. However, existing methods fail to fully integrate the features of event data and blurred images, and are prone to losing details in complex backgrounds. The present invention solves the above problems by designing a cyclic event representation, linear deformable convolution, and a multi-branch attention module. Summary of the Invention

[0003] This paper proposes an event-driven image motion deblurring method, the core of which lies in the efficient representation of event data and the optimized design of the deblurring network. Specifically, it includes the following steps:

[0004] S1: Obtain a deblurred dataset including blurred images, clear images, and event streams within the exposure time.

[0005] The step S1 specifically includes the following steps:

[0006] S11: The deblurring network is trained and evaluated using the GoPro dataset, which is widely used in motion deblurring. The dataset consists of 3214 pairs of blurry and clear images with a resolution of 1280*720.

[0007] S12: Use frame interpolation algorithm between adjacent clear images in the dataset to increase the frame rate.

[0008] S13: In order to simulate motion blur in real-world scenes, the continuous clear images in the obtained high frame rate sequence are averaged to obtain blur effects of different degrees.

[0009] S14: Downsampling operations of grayscale and bicubic interpolation are performed on the blurred image and the clear image in sequence, and the resolution is adjusted to 640*360 to keep consistent with the resolution of the event camera.

[0010] S15: Use the event generation simulator to generate a simulated event stream whose noise follows a Gaussian distribution and corresponds to a clear image.

[0011] S2: A cyclic event representation is used for preprocessing to convert the event stream into event frames that are similar to the underlying real image.

[0012] The step S2 specifically includes the following steps:

[0013] S21: Perform bilinear interpolation on the event stream to obtain voxel event representation, which can fully preserve the temporal information of the event.

[0014] S22: Inputting the event voxels into the cyclic encoder / decoder to obtain an event frame similar to the latent clear image, which can more fully preserve the motion information of the event.

[0015] S3: A deblurring network is constructed using linear deformable convolution and multi-branch feature extraction attention mechanism.

[0016] S4: The blurred image and event frame are input into the trained deblurring network for processing to obtain a clear image after deblurring.

[0017] Preferably, the S21 is specifically as follows:

[0018] Events within a period of time are discretized into B channels. Based on experience, B is usually set to 5, so the input of the network is a voxel of size H*W*B, where H and W are the height and width of the sensor respectively. Event voxels are usually interpolated using bilinear interpolation, and the voxel generation model is as follows:

[0019]

[0020] Where E represents the event voxel, (x, y, t) represents the horizontal and vertical coordinates and the time dimension respectively, n represents the time normalized index, p represents the event polarity, and t is the normalized timestamp. n * Represents the time after linear interpolation.

[0021] Preferably, the S22 is specifically as follows:

[0022] The recurrent encoder-decoder consists of a downsampling convolutional layer, an LSTM convolutional layer, a residual block, and an upsampling convolutional layer. Event voxels undergo convolution, downsampling convolution, and the LSTM convolutional layer to encode historical information about the event. The feature map passes through the residual block R in the middle of the network, then undergoes upsampling convolution to restore it to its original resolution and finally is encoded into an image-like representation.

[0023] Preferably, the S3 is as follows:

[0024] The deblurring backbone network includes a 5*5 convolution layer, a residual block with a convolution kernel size of 3, a Relu activation layer, a linear deformable convolution module, and a multi-branch feature extraction attention module. The input blurred image is fused with the event image preprocessed using the cyclic event representation method to generate the initial feature image A1; the feature of A1 is extracted through an encoder consisting of four downsampling stages: each downsampling stage uses a 5×5 convolution kernel with a step size of 2 for spatial compression, and the number of channels is doubled at each stage, outputting a sequence of feature maps B1, B2, B3, and B4, where the size of B4 is 1 / 16 of the original input; B4 is input to the residual block and then processed by the linear deformable convolution module to obtain the feature map B5; B5 is convolutionally upsampled, and the output channels are halved and then passed through B3. The network is connected in the channel dimension by a jump connection and fused with 5×5 convolution to generate B6; B6 is upsampled by convolution, the output channel is halved, and then it is connected to B2 in the channel dimension by a jump connection and fused with 5×5 convolution to generate B7; B7 is upsampled by convolution, the output channel is halved, and then it is connected to B1 in the channel dimension by a jump connection and fused with 5×5 convolution to generate B8; a multi-branch feature extraction attention module is introduced at each jump connection of the decoder to solve the problems of information loss and feature blurring caused by complex background; a 1×1 convolution operation is performed on the final reconstructed feature B8, and a clear image after deblurring is obtained after Sigmoid activation.

[0025] Preferably, the linear deformable convolution includes a standard convolution kernel and a learnable offset generation module. The offset generation module generates an offset for each sampling point from the input feature map through linear transformations such as a fully connected layer and 1x1 convolution. The generated offset is used to adjust the sampling position of the convolution kernel so that it can adapt to the geometric deformation in the input feature map. The parameters of the offset generation module are optimized together with the parameters of the convolution kernel through a backpropagation algorithm. During the training process, the offset generation module learns how to dynamically adjust the sampling position according to the content of the input feature map to maximize the performance of the convolution operation. More accurately capture the boundaries and details of the blurred image and improve the deblurring effect.

[0026] Preferably, the multi-branch feature extraction attention module includes three parallel feature extraction branches. Each branch uses convolution kernels of different sizes (3x3, 5x5, 7x7) to extract features of different scales, and introduces channel attention mechanism and spatial attention mechanism in each branch to enhance the saliency of the target blurred area and suppress background noise. The channel attention mechanism calculates the weight of each channel through global average pooling and full connection layer. The calculated channel weight is multiplied by the original feature map to enhance the feature response of the important channel, which can better highlight the features of the target blurred area; the spatial attention mechanism generates a spatial weight map through convolution operation, and multiplies the spatial weight map with the original feature map to enhance the feature response of the target blurred area. The feature map processed by the spatial attention mechanism can better suppress background noise and highlight the features of the target blurred area. Finally, the features extracted by different branches are fused by weighted summation or splicing to generate the final multi-scale feature map.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) Although traditional event representation preserves event temporal information using interpolation, this form of event representation loses event history and provides less information for image deblurring. To fully utilize the event stream, the present invention proposes a cyclic event representation method. This representation method can preserve high temporal resolution of events while effectively utilizing event history.

[0029] (2) The present invention can more effectively preserve the edges, textures, and details of the image during the deblurring process. By designing a linear deformable convolution, it can better restore the clear details of the blurred image, especially in high-speed motion and complex backgrounds, and the effect is significantly better than existing methods.

[0030] (3) The present invention can better handle different types of complex scenes. The designed multi-branch feature extraction attention module can focus on features of different dimensions respectively, more accurately screening and enhancing useful features while suppressing irrelevant information, thereby improving the model's recognition ability for complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of an event-driven image motion deblurring method in the present invention;

[0032] Figure 2 It is a structural diagram showing a cyclic event of the present invention;

[0033] Figure 3 This is the defuzzified backbone network structure diagram of the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0035] Example 1:

[0036] See Figure 1 , an event-driven image motion deblurring method, comprising the following steps:

[0037] S1: Obtain a deblurred dataset including blurred images, clear images, and event streams within the exposure time.

[0038] The specific steps of step S1 are:

[0039] S11: The deblurring network is trained and evaluated using the GoPro dataset, which is widely used in motion deblurring. The dataset consists of 3214 pairs of blurry and clear images with a resolution of 1280*720.

[0040] S12: Use frame interpolation algorithm between adjacent clear images in the dataset to increase the frame rate.

[0041] S13: In order to simulate motion blur in real-world scenes, the continuous clear images in the obtained high frame rate sequence are averaged to obtain blur effects of different degrees.

[0042] S14: Downsampling operations of grayscale and bicubic interpolation are performed on the blurred image and the clear image in sequence, and the resolution is adjusted to 640*360 to keep consistent with the resolution of the event camera.

[0043] S15: Use the event generation simulator to generate a simulated event stream whose noise follows a Gaussian distribution and corresponds to a clear image.

[0044] S2: A cyclic event representation is used for preprocessing to convert the event stream into event frames that are similar to the underlying real image.

[0045] The step S21 specifically includes the following steps:

[0046] S21: Perform bilinear interpolation on the event stream to obtain voxel event representation, which can fully preserve the temporal information of the event.

[0047] The specific steps of step S21 are:

[0048] Events within a period of time are discretized into B channels. Based on experience, B is usually set to 5, so the input of the network is a voxel of size H*W*B, where H and W are the height and width of the sensor respectively. Event voxels are usually interpolated using bilinear interpolation, and the voxel generation model is as follows:

[0049]

[0050] Where E represents the event voxel, (x, y, t) represents the horizontal and vertical coordinates and the time dimension respectively, n represents the time normalized index, p represents the event polarity, and t is the normalized timestamp. n * Represents the time after linear interpolation.

[0051] S22: Inputting the event voxels into the cyclic encoder / decoder to obtain an event frame similar to the latent clear image, which can more fully preserve the motion information of the event.

[0052] The specific steps of step S22 are:

[0053] like Figure 2 The recurrent encoder-decoder shown in this paper consists of a downsampling convolutional layer, an LSTM convolutional layer, a residual block, and an upsampling convolutional layer. Event voxels undergo convolution, downsampling convolution, and the LSTM convolutional layer to encode historical information about the event. The feature map passes through the residual block R in the middle of the network, then undergoes upsampling convolution to restore it to its original resolution and finally is encoded into an image-like representation.

[0054] S3: A deblurring network is constructed using linear deformable convolution and multi-branch feature extraction attention mechanism.

[0055] S4: The blurred image and event frame are input into the trained deblurring network for processing to obtain a clear image after deblurring.

[0056] like Figure 3The deblurring backbone network shown in the figure includes a 5*5 convolution layer, a residual block with a convolution kernel size of 3, a Relu activation layer, a linear deformable convolution module, and a multi-branch feature extraction attention module. The input blurred image is fused with the event image preprocessed using the cyclic event representation method to generate the initial feature image A1; the feature of A1 is extracted through an encoder consisting of four downsampling stages: each downsampling stage uses a 5×5 convolution kernel with a step size of 2 for spatial compression, and the number of channels is doubled at each stage, outputting a sequence of feature maps B1, B2, B3, and B4, where the size of B4 is 1 / 16 of the original input; B4 is input to the residual block and then processed by the linear deformable convolution module to obtain the feature map B5; B5 is convolutionally upsampled, and the output channels are halved and then passed through B3. The network is connected in the channel dimension by a jump connection and fused with 5×5 convolution to generate B6; B6 is upsampled by convolution, the output channel is halved, and then it is connected to B2 in the channel dimension by a jump connection and fused with 5×5 convolution to generate B7; B7 is upsampled by convolution, the output channel is halved, and then it is connected to B1 in the channel dimension by a jump connection and fused with 5×5 convolution to generate B8; a multi-branch feature extraction attention module is introduced at each jump connection of the decoder to solve the problems of information loss and feature blurring caused by complex background; a 1×1 convolution operation is performed on the final reconstructed feature B8, and a clear image after deblurring is obtained after Sigmoid activation.

[0057] Linear deformable convolution consists of a standard convolution kernel and a learnable offset generation module. The offset generation module generates an offset for each sampling point from the input feature map through linear transformations such as fully connected layers and 1x1 convolutions. The generated offset is used to adjust the sampling position of the convolution kernel so that it can adapt to the geometric deformations in the input feature map. The parameters of the offset generation module are optimized together with the parameters of the convolution kernel through a backpropagation algorithm. During training, the offset generation module learns how to dynamically adjust the sampling position based on the content of the input feature map to maximize the performance of the convolution operation. This allows for more accurate capture of the boundaries and details of blurred images, improving the deblurring effect.

[0058] The multi-branch feature extraction attention module consists of three parallel feature extraction branches. Each branch uses convolution kernels of different sizes (3x3, 5x5, and 7x7) to extract features at different scales. Channel and spatial attention mechanisms are introduced within each branch to enhance the saliency of the blurred target area and suppress background noise. The channel attention mechanism calculates a weight for each channel through global average pooling and a fully connected layer. The calculated channel weights are multiplied with the original feature map to enhance the feature response of important channels, better highlighting the features of the blurred target area. The spatial attention mechanism generates a spatial weight map through convolution operations and multiplies the spatial weight map with the original feature map to enhance the feature response of the blurred target area. The feature map processed by the spatial attention mechanism can better suppress background noise and highlight the features of the blurred target area. Finally, the features extracted by different branches are fused through weighted summation or splicing to generate the final multi-scale feature map.

Claims

1. An event-driven image motion deblurring method, characterized in that: The steps include: S1: Obtain a deblurred dataset including blurred images, clear images, and event streams within the exposure time. The step S1 specifically includes the following steps: S11: The deblurring network is trained and evaluated using the GoPro dataset, which is widely used in motion deblurring. The dataset consists of 3214 pairs of blurry and clear images with a resolution of 1280*720. S12: Use frame interpolation algorithm between adjacent clear images in the dataset to increase the frame rate. S13: In order to simulate motion blur in real-world scenes, the continuous clear images in the obtained high frame rate sequence are averaged to obtain blur effects of different degrees. S14: Downsampling operations of grayscale and bicubic interpolation are performed on the blurred image and the clear image in sequence, and the resolution is adjusted to 640*360 to keep consistent with the resolution of the event camera. S15: Use the event generation simulator to generate a simulated event stream whose noise follows a Gaussian distribution and corresponds to a clear image. S2: A cyclic event representation is used for preprocessing to convert the event stream into event frames that are similar to the underlying real image. The step S2 specifically includes the following steps: S21: Perform bilinear interpolation on the event stream to obtain voxel event representation, which can fully preserve the temporal information of the event. S22: Inputting the event voxels into the cyclic encoder / decoder to obtain an event frame similar to the latent clear image, which can more fully preserve the motion information of the event. S3: A deblurring network is constructed using linear deformable convolution and multi-branch feature extraction attention mechanism. S4: The blurred image and event frame are input into the trained deblurring network for processing to obtain a clear image after deblurring.

2. The event-driven image motion deblurring method according to claim 1, characterized in that: The S21 is specifically as follows: Events within a period of time are discretized into B channels. Based on experience, B is usually set to 5, so the input of the network is a voxel of size H*W*B, where H and W are the height and width of the sensor respectively. Event voxels are usually interpolated using bilinear interpolation, and the voxel generation model is as follows: Where E represents the event voxel, (x, y, t) represents the horizontal and vertical coordinates and the time dimension respectively, n represents the time normalized index, p represents the event polarity, and t is the normalized timestamp. n * Represents the time after linear interpolation.

3. The event-driven image motion deblurring method according to claim 1, characterized in that: The S22 is specifically as follows: The recurrent encoder-decoder consists of a downsampling convolutional layer, an LSTM convolutional layer, a residual block, and an upsampling convolutional layer. Event voxels undergo convolution, downsampling convolution, and the LSTM convolutional layer to encode historical information about the event. The feature map passes through the residual block R in the middle of the network, then undergoes upsampling convolution to restore it to its original resolution and finally is encoded into an image-like representation.

4. The event-driven image motion deblurring method according to claim 2, characterized in that: The S3 is specifically as follows: The deblurring backbone network consists of a 5x5 convolutional layer, a residual block with a convolution kernel size of 3, a ReLU activation layer, a linear deformable convolution module, and a multi-branch feature extraction attention module. The input blurred image is fused with the event image preprocessed using the recurrent event representation method to generate the initial feature image A1. The features of A1 are extracted through an encoder consisting of four downsampling stages: each downsampling stage uses a 5×5 convolution kernel with a stride of 2 for spatial compression, and the number of channels is doubled at each stage, outputting a sequence of feature maps B1, B2, B3, and B4, where the size of B4 is 1 / 16 of the original input; B4 is input into the residual block and then processed by a linear deformable convolution module to obtain feature map B5; B5 is upsampled by convolution, the output channel is halved, and then the channel dimension is skipped with B3, and fused with 5×5 convolution to generate B6; B6 is upsampled by convolution, the output channel is halved, and then the channel dimension is skipped with B2, and fused with 5×5 convolution to generate B7; B7 is upsampled by convolution, the output channel is halved, and then the channel dimension is skipped with B1, and fused with 5×5 convolution to generate B8; a multi-branch feature extraction attention module is introduced at each skip connection of the decoder to solve the problems of information loss and feature blurring caused by complex background; A 1×1 convolution operation is performed on the final reconstructed feature B8, and a clear image is obtained after deblurring after Sigmoid activation.

5. The event-driven image motion deblurring method according to claim 4, characterized in that: The linear deformable convolution includes a standard convolution kernel and a learnable offset generation module. The offset generation module generates an offset for each sampling point from the input feature map through linear transformations such as fully connected layers and 1x1 convolutions. The generated offset is used to adjust the sampling position of the convolution kernel so that it can adapt to the geometric deformation in the input feature map. The parameters of the offset generation module are optimized together with the parameters of the convolution kernel through the backpropagation algorithm. During the training process, the offset generation module learns how to dynamically adjust the sampling position according to the content of the input feature map to maximize the performance of the convolution operation. More accurately capture the boundaries and details of the blurred image and improve the deblurring effect.

6. The event-driven image motion deblurring method according to claim 4, characterized in that: The multi-branch feature extraction attention module includes three parallel feature extraction branches. Each branch uses convolution kernels of different sizes (3x3, 5x5, 7x7) to extract features of different scales, and introduces channel attention mechanism and spatial attention mechanism in each branch to enhance the saliency of the target blurred area and suppress background noise. The channel attention mechanism calculates the weight of each channel through global average pooling and fully connected layers. The calculated channel weight is multiplied with the original feature map to enhance the feature response of important channels and better highlight the features of the blurred area of the target. The spatial attention mechanism generates a spatial weight map through convolution operations. This is then multiplied with the original feature map to enhance the feature response of the blurred target area. This feature map, processed by the spatial attention mechanism, can better suppress background noise and highlight the features of the blurred target area. Finally, features extracted from different branches are fused through weighted summation or concatenation to generate the final multi-scale feature map.

Images