Image deblurring method and device

Abstract

The invention discloses an image deblurring method and device, and belongs to the field of image processing. In order to solve the problem of blurring existing in a frame image, according to an event stream output by an event camera, point cloud representation and voxel representation are applied at the same time, a voxel representation event and a multi-scale frame image are received in a coarse time domain fusion branch, and frame image features and features based on the voxel representation event are extracted; features fused along the time domain dimension are extracted through fine-grained point branches, and diffused point cloud features are further obtained. And finally, in the coarse time domain fusion branch, fusing the extracted features, and reconstructing an image. According to the method, the quality of the deblurred image can be effectively improved by applying the fine-grained features.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image deblurring method and device. Background Art

[0002] Traditional frame-based cameras are prone to blur due to object or camera movement during exposure. Traditional algorithms rely on mathematical models (such as deconvolution), require prior knowledge and are sensitive to light. Although deep learning methods have achieved some results, frame-based methods have limited performance in complex scenes and lighting conditions.

[0003] Event cameras can continuously track visual information and have the advantages of high temporal resolution (milliseconds or even microseconds), high dynamic range and low latency. The brightness changes they detect emphasize high-contrast edges and can compensate for texture loss caused by motion blur, which plays an important role in motion deblurring technology.

[0004] There are already some image deblurring schemes based on event cameras, but there are still challenges in effectively utilizing asynchronous and sparse event streams.

[0005] Prior art 1: Sun, L.; Sakaridis, C.; Liang, J.; Jiang, Q.; Yang, K.; Sun, P.; Ye, Y.; Wang, K.; and Gool, LV 2022. Event-based fusion for motion deblurring with cross-modal attention. In European conference on computer vision, 412–428. Springer.

[0006] Prior Art 2: Yang, W.; Wu, J.; Li, L.; Dong, W.; and Shi, G. 2023. Event-basedMotion Deblurring with Modality-Aware Decomposition and Recomposition. In Proceedings of the 31st ACM International Conference on Multimedia, 8327–8335.

[0007] Prior Art 3: Chen, K.; and Yu, L. 2024. Motion Deblurby learning residual from events. IEEE Transactions on Multimedia.

[0008] Prior Art 4: Yang, W.; Wu, J.; Ma, J.; Li, L.; and Shi, G. 2024. Motion Deblurring via Spatial-Temporal Collaboration of Frames and Events. In ProceedingsoftheAAAI Conference on Artificial Intelligence, vol. 38, 6531–6539.

[0009] Prior Art 5: Zhang,

[0010] EFNet in prior art 1 introduces a multi-head attention mechanism to fuse information from different modalities and develops a specialized event representation tailored for event-based image deblurring tasks. EIFNet in prior art 2 addresses motion deblurring by decomposing and recombining modality-specific and modality-shared features, and uses a dual cross-attention mechanism to separate features and long-range interactions for better fusion. TRMD in prior art 3 utilizes a two-stage residual-based motion deblurring framework that transforms blurry images into clear sequences by leveraging motion features from events, combined with a dual cross-modal fusion module to enhance residual estimation. STCNet in prior art 4 adopts a cross-modal calibration strategy based on differential modalities to reduce redundancy and a common attention mechanism for effective spatial fusion, combined with a frame event spatiotemporal attention scheme to correct cross-temporal errors. GEM in prior art 5 uses a scale-aware network to support variable spatial and temporal scales, and adopts a self-supervised learning strategy to enhance adaptability in different real-world environments.

[0011] Although these event-based image deblurring schemes have produced significant progress, they still have some limitations. For example, previous methods that use voxel grids to represent event data mainly focus on coarse-grained temporal information and ignore fine-grained temporal details, thereby reducing the deblurring performance. Summary of the invention

[0012] In order to alleviate or partially alleviate the above technical problems, the solution of the present invention is as follows: an image deblurring method, which converts an event stream into voxel expression events and point cloud expression events, and converts a frame image into a multi-scale frame image; the point cloud expression event is used as the input of the first branch, and the voxel expression event and the multi-scale frame image are used as the input of the second branch; the second branch is a multi-input and multi-output Unet network, and the frame image features and the features based on the voxel expression events are obtained in the second branch; in the first branch, the point cloud expression events are sampled, grouped and encoded, and the aggregated feature set is obtained after weighting by the attention weight, and the aggregated feature set in the time domain dimension is fused to obtain the fused features along the time domain dimension, and then mapped to a two-dimensional plane to obtain the mapped point cloud features, and then weighted by Gaussian weights to obtain the diffused point cloud features; in the second branch, the diffused point cloud features are aggregated with the frame image features and the features based on the voxel expression events, and the reconstructed frame image is obtained after decoding.

[0013] Furthermore, the first branch receives the frame image features generated by the second branch, and applies a global average pooling operation to the frame image features to obtain the frame image features after global average pooling; a multi-layer perceptron is applied to the frame image features after global average pooling and after sigmoid activation, the multilayer perceptron is multiplied by a preset maximum diffusion range to obtain a diffusion range.

[0014] Furthermore, the Gaussian weight is equal to exp(-dist 2 / 2D 2 ), where D is the diffusion range, dist is the distance from the centroid to the current diffusion point, and the centroid is the centroid of the group obtained during the sampling and grouping process of the point cloud expression event.

[0015] Furthermore, before sampling and grouping the point cloud expression events, the point cloud expression events are first normalized to obtain normalized point cloud expression events; in each time stack, the normalized point cloud expression events are divided into multiple groups through the farthest point sampling method or the KNN method, and the coordinate position of the center of mass of each group is saved.

[0016] Furthermore, in the first branch, the groups are encoded through the convolutional layer and the residual module to obtain a point cloud feature set of the group; and the point cloud feature set of the group is weighted by the attention weight to obtain an aggregated feature set.

[0017] Furthermore, at least the aggregated feature set in the time domain dimension is fused through LSTM to obtain the fused features along the time domain dimension.

[0018] Furthermore, the first branch includes a first diffusion module, a second diffusion module and a third diffusion module, and is used to obtain diffused point cloud features; and the first diffusion module, the second diffusion module and the third diffusion module respectively obtain frame image features from different layers of the second branch, and input diffused point cloud features to different layers in the second branch.

[0019] Furthermore, three downsampling operations and three upsampling operations are performed in the multi-input multi-output Unet network, and the multi-input multi-output Unet network includes two shallow convolution modules and two asymmetric feature fusion modules.

[0020] Further, in the second branch, the result of downsampling the voxel expression event is added to the result of downsampling the frame image with the highest resolution to obtain the first frame image feature, and the first frame image feature is input into the first diffusion module; and the result of adding the first diffused point cloud feature output by the first diffusion module to the first frame image feature is subjected to a second downsampling; the result of the second downsampling is used as the second frame image feature and input into the second diffusion module; and the result of adding the second diffused point cloud feature output by the second diffusion module to the second frame image feature is subjected to a third downsampling; the result of the third downsampling is used as the third frame image feature and input into the third diffusion module; and the result of adding the third diffused point cloud feature output by the third diffusion module to the third frame image feature is subjected to a first upsampling.

[0021] An image deblurring device receives an event stream and a frame image; and the image deblurring device applies any one of the above-mentioned image deblurring methods.

[0022] The technical solution of the present invention has one or more of the following beneficial technical effects:

[0023] (1) For the first time, point cloud-based event representation is used in event-based image deblurring tasks. Combining point cloud and voxel-based representations can preserve temporal details and provide more accurate motion information.

[0024] (2) To deblur high-resolution images with relatively low-resolution events, we spatially align point-based, voxel-based, and frame-based features. We also diffuse point-based features to enrich spatially sparse features.

[0025] (3) A large number of experiments show that, equipped with the proposed module and multi-temporal granular event representation, the optimal embodiment of the present invention achieves the current highest level (SOTA) under different fuzzy conditions in the Ev-REDS and HS-ERGB datasets.

[0026] In addition, other beneficial effects of the present invention will be mentioned in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the architecture of the present invention;

[0028] Figure 2 It is a framework diagram of the image deblurring solution of the present invention;

[0029] Figure 3 It is a schematic diagram of the MIMO-Unet scheme with voxel expression event input added;

[0030] Figure 4 This is a comparison diagram of the image-related deblurring effect of the present invention;

[0031] Figure 5 This is a qualitative comparison chart under the EV-REDS dataset;

[0032] Figure 6 It is a fixed-row comparison chart under the HS-ERGB dataset. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0035] The event stream (or pulse train) ε output by the event camera can be written as:

[0036] ε={e i =(x i ,y i ,t i ,p i )|i=1,...,n},

[0037] Where i is the sequence number, ei is the event with sequence number i, and the event is i ,y i ,p i ,t i ) four-tuple representation, (x i ,y i ,) is the event ei The coordinates of the pixel in the pixel array, t i Is the event e i The timestamp, p i It is the polarity information indicating whether the light intensity becomes weaker or stronger, and n is a positive integer.

[0038] For event streams, voxels represent events. Here is how:

[0039]

[0040] Among them, r k is the kth rasterized time bin, there are b rasterized time bins in total, t 0 and t n is the start timestamp and the end timestamp; v x,y (k) is the voxel value of the kth rasterized time stack at coordinate (x, y). The horizontal and vertical coordinates of coordinate (x, y) range respectively [1, w] and [1, h], where w and h are the resolutions of the event camera.

[0041] For event streams, point clouds represent events. It is expressed as follows:

[0042]

[0043] The polarity information is ignored in the point cloud representation, and the time dimension t is considered as z. In addition, the voxel representation

[0044] According to the inventors' research, the smaller the time domain granularity Δt of the input data is, the higher the quality of the deblurred image is:

[0045]

[0046] in, Yes 0 The frame image at the moment, is the estimated t 0 The frame image at the moment, Δt is the time domain granularity, B(.) is the average light intensity within a period of time, c is the threshold of the light intensity change triggering event in the event camera. Based on the teaching of this discovery, the present invention follows this principle to improve the quality of the image after deblurring.

[0047] For voxel expression events The exposure time T of the blurred image is divided into b time stacks, and events are accumulated in each time stack to become a 2D h×w tensor, that is, Although b is usually set to a small number, such as 6, and the temporal resolution is usually in the millisecond level (which is much higher than the 30FPS temporal resolution of traditional frame cameras), the voxel expression event Compared to the more fine-grained point cloud expression event Still coarse-grained.

[0048] For point cloud expression events The exposure time T is also divided into b time stacks. For point cloud expression, the events in the point cloud are downsampled to a fixed number, such as m events, in each time stack, avoiding stacking of events in the time dimension, that is,

[0049] In the present invention, the point cloud can be normalized according to the resolution of the event camera and the duration of the time stack:

[0050]

[0051] in is the normalized point cloud in the k-th time stack. (x i ,y i ,t i ) are the coordinates and timestamp of the ith event.

[0052] Figure 1 This is a schematic diagram of the architecture of the present invention. Different from using only voxel expression, the present invention combines voxel expression and point cloud expression of event stream to more effectively capture fine-grained temporal dynamic information. According to the analysis above, the smaller the temporal granularity of the input data, the higher the quality of the deblurred image.

[0053] After the event camera outputs the event stream, it is represented by point cloud and voxel, and then input into the fine-grained point branch (first branch) and the coarse time domain fusion branch (second branch). In addition, the second branch also receives multi-scale frame images. The multi-scale frame image is a multiple scale representation of the blurred frame image, such as the multi-scale frame image. Wherein H and W are the resolutions of the blurred frame image output by the ordinary optical camera. In this example, the multi-scale frame image has three different scales or resolutions, where 3 is the number of channels of the frame image, usually RGB channels.

[0054] The multi-scale frame images in the present invention refer to 3 or more frame images expressing the same content, and having different resolutions.

[0055] Figure 2The image deblurring scheme of the present invention is a framework diagram. The image deblurring of the present invention includes a first branch and a second branch. The first branch is a fine-grained point branch, and the second branch is a coarse time domain fusion branch.

[0056] The second branch originates from MIMO-Unet and related derivative solutions in the prior art 6. MIMO-Unet is a multi-input multi-output Unet, which is an improvement on the original Unet network.

[0057] Figure 3 It is a schematic diagram of the MIMO-Unet scheme with added voxel expression event input, which is a modification made on the basis of the MIMO-Unet scheme in the prior art 6. The present invention introduces at least all the technical contents of the prior art 6 into this application by reference and makes it a part of this application. MIMO-Unet includes two processes: downsampling and upsampling (such as deconvolution operation), and includes a shallow convolution module (SCM) and an asymmetric feature fusion (AFF) module.

[0058] Prior art 6: Cho, S.-J.; Ji, S.-W.; Hong, J.-P.; Jung, S.-W.; and Ko, S.-J. 2021. Rethinking coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE / CVF international conference on computer vision, 4641–4650.

[0059] In the second branch, the first layer includes a convolutional layer and a residual connection layer, the second and third layers include a convolutional layer, a feature attention layer, and a residual layer, the fourth layer includes a residual connection layer and a deconvolutional layer, the fifth layer includes a convolutional layer, a residual connection layer, and a deconvolutional layer, and the sixth layer includes a convolutional layer, a residual connection layer, and a convolutional layer. These layers are processed and arranged in sequence, which is consistent with the prior art 6.

[0060] also, Figure 3 The frame image feature F is also shown I1 ~F I3 And the diffuse point cloud feature F D1 ~F D3Schematic diagram of the position of the input and output in MIMO-Unet. Specifically, in the second branch, the result of downsampling the voxel expression event is added to the result of downsampling the frame image with the highest resolution to obtain the first frame image feature, and the first frame image feature is input into the first diffusion module; and the result of adding the first diffused point cloud feature output by the first diffusion module to the first frame image feature is subjected to a second downsampling; the result of the second downsampling is used as the second frame image feature and input into the second diffusion module; and the result of adding the second diffused point cloud feature output by the second diffusion module to the second frame image feature is subjected to a third downsampling; the result of the third downsampling is used as the third frame image feature and input into the third diffusion module; and the result of adding the third diffused point cloud feature output by the third diffusion module to the third frame image feature is subjected to a first upsampling.

[0061] Continue to refer Figure 2 The second branch includes a voxel-based event encoder and a frame image encoder. The frame image encoder receives a multiple-scale image. And through the processing from shallow to deep networks, it can cope with the diversity of ambiguity.

[0062] The voxel expression event encoder in the second branch receives the voxel expression event To extract coarse time domain information and fine spatial domain features. For example, like MIMO-Unet, the second branch of the present invention also includes 3 downsampling (contracting) steps / operations and 3 upsampling (expansive) steps / operations, as well as 2 SCM modules and 2 AFF modules. For the specific implementation details of the second branch, please refer to the prior art 6, and the present invention will not repeat them. The present invention does not limit the number of downsampling and upsampling steps (and the corresponding number of modules in ADFM).

[0063] In the present invention, the frame image features generated by the second branch and coarse-grained voxel-based features of expression events Motion estimation cues as initial estimates.

[0064] The first branch, namely the fine-grained point branch, includes an aggregation and mapping module (AMM) and an adaptive feature fusion module (AFDM).

[0065] In order to better utilize the fine-grained temporal information of events, the first branch receives point cloud expression events According to the inventors' findings, finer-grained time domain information can achieve better deblurring effects.

[0066] The AMM in the first branch includes: sampling and grouping, encoding, aggregation, feature fusion, and mapping. These links will be introduced in turn later.

[0067] In AMM, receive normalized point cloud expression events It is then fed into the sampling and grouping module to capture spatial and temporal information.

[0068] In each time stack, the normalized point cloud represents the event It can be divided into M groups by the farthest point sampling (FPS) method and the KNN method, and the coordinate position information of the centroid of each group is saved or retained, which can be expressed as follows:

[0069]

[0070] in is the i-th group, is the centroid coordinate of the ith group, K is the number of points in each group, and M is the number of groups in each time stack.

[0071] The point encoder includes a convolutional layer and a residual module connected in sequence. For example, the convolutional layer performs a one-dimensional convolution conv1d. Through the convolutional layer and the residual module, the feature set in each group can be refined and improved to enrich the point cloud expression.

[0072] Through attention-driven weights, point features within a group are integrated into the centroid. This aggregation centralizes the grouped features and effectively enhances the temporal and spatial characteristics of the expression. The above process can be expressed by the following formula:

[0073]

[0074] in is a set of point cloud features grouped into groups. is the aggregated feature set, α i is the attention weight of the i-th point in the group, and is generated based on the point cloud feature set input to the attention module. “:” represents all elements in the corresponding dimension. is the feature dimension of each point.

[0075] Furthermore, the information in different time stacks can be integrated through the Long Short-Term Memory (LSTM) network to fuse the features in the time domain dimension. Subsequently, the coordinate position information retained from the grouping process is used to map the aggregated features to the (x, y) two-dimensional plane to facilitate high-resolution frame image features. and coarse-grained voxel-based features of expression events Perform effective alignment and fusion. The above process can be expressed by the following formula:

[0076]

[0077] in It is the feature fused along the time domain dimension. is the point cloud feature after mapping, β i is the feature resolution of the i-th layer in the downsampling step in MIMO-Unet, are the centroid coordinates.

[0078] Preferably, before the LSTM processing, a residual module may be provided, and the LSTM processing may be performed after the residual connection.

[0079] Due to the mapped point cloud features AFDM is used to enrich sparse features and cope with the resolution differences between different visual sensors, i.e., event cameras and ordinary optical cameras, which are too sparse in the spatial domain. AFDM enhances the spatial distribution properties of point features and ensures a robust and detailed representation.

[0080] The AFDM in the first branch includes three diffusion modules: a first diffusion module, a second diffusion module and a third diffusion module. Each diffusion module includes: a multi-layer perceptron (MLP), Sigmoid activation, Gaussian weighting and other links.

[0081] In the present invention, the diffusion module receives the frame image features Specifically, the first diffusion module to the third diffusion module respectively receive the first frame image features generated by the MIMO-Unet in the second branch The second frame image features The third frame image features The diffusion module outputs the diffused point cloud features The diffuse point cloud features It can be obtained by:

[0082]

[0083] in is the Gaussian weight, ⊙ is the element-wise product. However, the Gaussian weight Obtained through:

[0084]

[0085] Where dist is the distance from the centroid to the current diffusion point, is the spread, acting as the standard deviation of the Gaussian function.

[0086] In the diffusion module, the dynamic diffusion range is first estimated based on the blurred image features. Then adaptive dilation determines the influence of the features on neighboring points more accurately by applying a multi-layer perceptron (MLP). The above process can be expressed by the following formula:

[0087]

[0088] in, is the frame image feature after global average pooling, is the frame image feature, α is the preset maximum diffusion range, and σ is the sigmoid activation function, which can ensure that the diffusion range is between 0 and 1.

[0089] At this point, Gaussian weights The diffusion range It can be determined that the Gaussian weight can be completely determined.

[0090] In the present invention, Gaussian weight Used to control the spatial propagation influence of each point feature. Gaussian weight It is highly correlated with the distance from the centroid to ensure that the influence of the point feature decreases sharply with distance, such as exponentially, so that the influence is concentrated near the point, while allowing customizable propagation.

[0091] The first diffusion module to the third diffusion module output the point cloud features of the first diffusion respectively

[0092] Point cloud features of the second diffusion Point cloud features of the third diffusion And serve as the input of the second downsampling module, the third downsampling module and the first upsampling module in the second branch respectively. In other words, the first diffusion module, the second diffusion module and the third diffusion module respectively obtain frame image features from different layers of the second branch, and input the diffused point cloud features to different layers in the second branch.

[0093] In the second branch, the diffuse point cloud features With frame image features and voxel-based features of expression events Aggregation, after decoding, the reconstructed frame image is obtained.

[0094] As an example, in the process of training the network, the structural similarity (SSIM) loss, multi-scale frequency reconstruction (MSFR) loss and MAE loss / L1 loss are used to evaluate the difference between the reconstructed frame image and the real image:

[0095] L total =λ 1 L MAE +λ 2 L SSIM +λ 3 L MSFR ,

[0096] Among them, λ can be set as an example 1 =10,λ 2 =1,λ 3 =0.1.

[0097] Figure 4 It is a comparison chart of the deblurring effect of the image of the present invention. It is not difficult to see that the deblurring effect of the present invention is closer to the actual image than other solutions.

[0098] Figure 5 This is a qualitative comparison chart under the EV-REDS data set. Compared with other solutions, the deblurring effect of the present invention is closer to the actual image and has better deblurring quality.

[0099] Figure 6 This is a fixed-line comparison diagram under the HS-ERGB data set. For a large number of blurred areas in the splashing water image, the present invention can also better eliminate blur and achieve a better deblurring effect.

[0100] Table 1 is a performance comparison of the present invention and some existing technologies on the Ev-REDS and HS-ERGB datasets. The indicators for evaluating the image deblurring effect include: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Image Measurement (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). It is not difficult to see that the present invention is superior to these existing technologies in both types of data and reaches the current best.

[0101] Table 1: Performance comparison on Ev-REDS and HS-ERGB datasets

[0102]

[0103] In order to evaluate the impact of multiple time granularities and the effectiveness of key components in the present invention, the present invention conducts a comprehensive ablation study on the HS-ERGB dataset, as shown in Table 2. Table 2 is the ablation experimental study data of the present invention on the HS-ERGB dataset. Among them, Image represents the image point input, Voxel represents the voxel-based event representation, Cloud is the point cloud-based event representation, and AFDM is the adaptive feature diffusion module. The data results in Table 2 show the effectiveness of multiple time granularities and the adaptive feature diffusion module.

[0104] Table 2: Ablation experimental data of the present invention on the HS-ERGB dataset

[0105]

[0106] In order to better illustrate the present invention, numerous specific details are provided in the above specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0107] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image deblurring method, characterized in that: Convert the event stream into voxel expression events and point cloud expression events, and convert the frame image into a multi-scale frame image; Point cloud expression events are used as input of the first branch, and voxel expression events and multi-scale frame images are used as input of the second branch; The second branch is a multi-input multi-output Unet network, and the frame image features and the features of the voxel-based expression events are obtained in the second branch; In the first branch, the point cloud expression events are sampled, grouped and encoded, and the aggregated feature set is obtained after weighting by attention weights. The aggregated feature set in the time domain dimension is fused to obtain the fused features along the time domain dimension, and then mapped to a two-dimensional plane to obtain the mapped point cloud features, which are then weighted by Gaussian weights to obtain the diffused point cloud features. In the second branch, the diffused point cloud features are aggregated with the frame image features and the features based on voxel expression events, and the reconstructed frame image is obtained after decoding.

2. The image deblurring method according to claim 1, characterized in that: The first branch receives the frame image features generated by the second branch, and applies a global average pooling operation to the frame image features to obtain the frame image features after global average pooling; The multi-layer perceptron is applied to the frame image features after global average pooling and after sigmoid activation, it is multiplied by the preset maximum diffusion range to obtain the diffusion range.

3. The image deblurring method according to claim 2, characterized in that: The Gaussian weight is equal to exp(-dist 2 / 2D 2 ), where D is the diffusion range, dist is the distance from the centroid to the current diffusion point, and the centroid is the centroid of the group obtained during the sampling and grouping process of the point cloud expression event.

4. The image deblurring method according to claim 3, characterized in that: Before sampling and grouping the point cloud expression events, the point cloud expression events are first normalized to obtain the normalized point cloud expression events; In each time stack, the normalized point cloud expression events are divided into multiple groups by the farthest point sampling method or the KNN method, and the coordinate position of the centroid of each group is saved.

5. The image deblurring method according to claim 4, characterized in that: In the first branch, the groups are encoded through convolutional layers and residual modules to obtain a set of point cloud features of the groups; After weighting the grouped point cloud feature sets by the attention weights, the aggregated feature set is obtained.

6. The image deblurring method according to claim 5, characterized in that: At least the aggregated feature set in the time domain dimension is fused through LSTM to obtain the fused features along the time domain dimension.

7. The image deblurring method according to claim 1 or 6, characterized in that: The first branch includes a first diffusion module, a second diffusion module and a third diffusion module, and is used to obtain diffused point cloud features; and, The first diffusion module, the second diffusion module and the third diffusion module respectively obtain frame image features from different layers of the second branch, and input diffused point cloud features to different layers in the second branch.

8. The image deblurring method according to claim 1 or 5, characterized in that: performing 3 downsampling operations and 3 upsampling operations in the MIMO Unet network, and The multi-input multi-output Unet network includes two shallow convolution modules and two asymmetric feature fusion modules.

9. The image deblurring method according to any one of claims 1 to 6, characterized in that: In the second branch, the result of downsampling the voxel expression event is added to the result of downsampling the frame image with the highest resolution to obtain the first frame image feature, and the first frame image feature is input into the first diffusion module; And performing a second downsampling on the result of adding the first diffused point cloud features output by the first diffusion module to the first frame image features; The result of the second downsampling is used as the second frame image feature and input into the second diffusion module; And performing a third downsampling on the result of adding the second diffused point cloud features output by the second diffusion module to the second frame image features; The result of the third downsampling is used as the third frame image feature and input into the third diffusion module; and the first upsampling is performed on the result after the third diffused point cloud feature output by the third diffusion module is added to the third frame image feature.

10. An image deblurring device, characterized in that: The image deblurring device receives an event stream and a frame image; and, The image deblurring device applies the image deblurring method described in any one of claims 1 to 9.

Images