An event-image deblurring method based on LSTM

Abstract

The application discloses an event-image deblurring method based on an LSTM, and comprises the following steps: constructing an event-image deblurring network model for processing a blurred image; training the event-image deblurring network by using training set data in a blurred data set, obtaining a trained event-image deblurring network model; inputting test set data in the blurred data set into the trained event-image deblurring network model, and obtaining a clear frame sequence. The advantages of time sequence information of events are fully used, the deblurring performance is improved, the generalization of the network is improved, one model can be used to complete deblurring output of any time resolution, and the time resolution of the output clear frame is controlled by the time resolution of the input event. Meanwhile, a bidirectional coding-decoding structure is used to enhance the deblurring effect of the network, and the method has the following advantages: one model needs to be trained, and any time resolution output can be achieved, so that the problem of limited generalization of the model is solved.

Description

Technical Field

[0001] This invention belongs to the field of fully automated products and relates to an event-image deblurring method based on LSTM. Background Technology

[0002] Image deblurring: Traditional image deblurring outputs at a fixed temporal resolution (1 frame or 7 frames). To address this task, in 2021, Sung-Jin proposed MIMOUNET, which effectively integrates multi-scale features to improve network deblurring performance. In 2022, Zamir used a transposed attention mechanism, introducing the TransFormer structure into the deblurring task, effectively improving network performance.

[0003] Event deblurring: Event deblurring mostly involves sequence output, but the sequence length is also fixed.

[0004] To address this task, in 2021, Xu Fang proposed REDNET, which used a self-supervised training method to bridge the gap between simulated and real events, improving the network's ability to use real event data. In 2020, Songnan used dynamic convolutional kernels to solve the problem of varying event thresholds, effectively improving the performance of deblurring networks.

[0005] Existing problems:

[0006] Image deblurring: poor results; can only restore output at a fixed temporal resolution, lacks the ability to handle arbitrary temporal resolutions, and exhibits poor model generalization.

[0007] Event deblurring: can only restore output at a fixed time resolution, lacks the ability to handle arbitrary time resolutions, and has poor model generalization.

[0008] Both of the above-mentioned deblurring methods are fixed-time-resolution deblurring methods, and the time resolution of the network output does not have generalization ability. Summary of the Invention

[0009] To address the issue that existing image inpainting methods can only recover clear frame sequences at a fixed temporal resolution, and cannot generalize to other temporal resolutions after the model is trained, the technical solution adopted in this invention is:

[0010] An LSTM-based event-based image deblurring method includes the following steps:

[0011] Construct an event-based image deblurring network model for processing blurred images;

[0012] The event-image deblurring network is trained using the training set data in the fuzzy dataset, resulting in a trained event-image deblurring network model.

[0013] The test set data from the blurred dataset is input into the trained event-image deblurring network model to obtain a sequence of clear frames. 。

[0014] Furthermore, the process of inputting the test set data into the trained event-image deblurring network model to obtain a clear frame sequence is as follows:

[0015] An image encoder is used to extract multi-scale features from the acquired blurred image to obtain the multi-scale features of the blurred image;

[0016] Obtain the event sequence;

[0017] A bidirectional event encoder is used to extract multi-scale features from the acquired event sequence to obtain the multi-scale features of the event sequence;

[0018] For the multi-scale features of blurred images and the multi-scale features of event sequences, a multimodal fusion device is used to fuse features to obtain fused features;

[0019] The fused features are input into a bidirectional latent frame restorer to obtain a clear frame sequence.

[0020] Furthermore: the bidirectional event encoder includes a convolutional layer, a first residual block, and a first bidirectional LSTM;

[0021] The first convolutional layer, the first residual module, and the first bidirectional LSTM are connected sequentially.

[0022] Furthermore: The input to each time step of the first bidirectional LSTM is the event within the exposure time corresponding to the sharp frame to be recovered. Each event segment is represented using a voxel grid. After the input sequence is fed into the bidirectional LSTM in forward order, the input sequence is then reversed and fed into the first bidirectional LSTM. Finally, the forward and reverse outputs are fused to ensure bidirectional information flow. The operation flow of the bidirectional LSTM is as follows:

[0023]

[0024]

[0025]

[0026] Where: (1) represents the forward update process of bidirectional LSTM, It is a hidden state with time step n during positive updates, controlling short-term time memory. (1) In the forward update, the cell state at time step n controls long-term temporal memory; (2) Represents the reverse update process of bidirectional LSTM. (3) represents the hidden state and cell state in the reverse update; (4) represents the final output of the bidirectional LSTM at time step n, which is the convolution result of splicing the forward and reverse hidden states.

[0027] Furthermore: the input of the bidirectional latent frame restorer is the output of the multimodal fusion unit, which is a mixed modal sequence that simultaneously has the texture information of the blurred map and the temporal information of the event sequence, and the output of the bidirectional latent frame restorer is a clear frame output at any temporal resolution;

[0028] The bidirectional latent frame restorer is composed of a second residual module, a first decoder module, a second decoder module, a third decoder module, and an upsampling module connected in series.

[0029] The first decoder module, the second decoder module, and the third decoder module have the same structure;

[0030] The second residual module consists of two convolutional layers and residual jump connections.

[0031] The decoder module consists of a third residual module, a second convolutional layer, and a second bidirectional LSTM connected in series.

[0032] The upsampling module consists of a 2x bilinear interpolation method and a third convolutional layer.

[0033] Furthermore: the decoder recovers the clear frame sequence as follows: first, the input mixed modality sequence is processed through a residual block to extract features, and then it is sequentially processed through the decoder module at each scale to obtain the preliminary output of the clear frame sequence at each scale;

[0034] In addition, each scale's decoder module is followed by a convolutional layer, which adds the output at this point to the downsampling result corresponding to the blurred image to obtain the final output.

[0035] Furthermore: the image encoder comprises a fourth convolutional layer, a third residual module, and a multi-scale fusion module connected in series;

[0036] The input to the image encoder is a single-frame blurred image, and the output is multi-scale information corresponding to the single-frame blurred image;

[0037] Multiscale fusion blocks are composed of convolutions, element-wise multiplications, and element-wise summations.

[0038] Furthermore, the process of extracting multi-scale features from the acquired blurred image using an image encoder to obtain the multi-scale features of the blurred image is as follows:

[0039] The image encoder at each scale passes through a fourth convolutional layer and a third residual block, where the scale of the downsampled image also receives the output of the residual block from the previous scale, which is used as the scaling factor and bias applied to the output of the convolution at the current scale.

[0040] This invention provides an LSTM-based event-image deblurring method, proposing a novel LSTM-based event-grayscale image deblurring network. It fully utilizes the temporal information of events to improve deblurring performance and network generalization, enabling the use of a single model to achieve deblurred output at arbitrary temporal resolutions, with the output temporal resolution controlled by the temporal resolution of the input events. Furthermore, a bidirectional encoder-decoder structure enhances the network's deblurring effect, offering the following advantages: only one model needs to be trained to achieve sequence output at arbitrary temporal resolutions, thus solving the problem of limited model generalization. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the event-image deblurring network structure;

[0043] Figure 2 This is a schematic diagram of an image encoder;

[0044] Figure 3 This is a schematic diagram of a bidirectional event encoder;

[0045] Figure 4 This is a schematic diagram of a multi-scale fusion module;

[0046] Figure 5 This is a schematic diagram of a multimodal fusion device;

[0047] Figure 6 This is a schematic diagram of a bidirectional potential frame restorer. Detailed Implementation

[0048] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0051] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0052] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0053] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0054] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0055] An LSTM-based event-based image deblurring method includes the following steps:

[0056] S1: Construct an event-image deblurring network model for processing blurred images;

[0057] S2: Train the event-image deblurring network using the training set data in the fuzzy dataset to obtain the trained event-image deblurring network model;

[0058] S3: Input the test set data from the fuzzy dataset into the trained event-image deblurring network model to obtain images of a clear frame sequence.

[0059] The steps S1 / S2 / S3 are executed sequentially;

[0060] Figure 1 This is a schematic diagram of the event-image deblurring network model, which consists of an image encoder, a bidirectional event encoder, a multimodal fusion unit, and a bidirectional latent frame restorer for deblurring.

[0061] The process of inputting test set data into a trained event-image deblurring network model to obtain a sequence of clear frame images is as follows:

[0062] An image encoder is used to extract multi-scale features from the acquired blurred image to obtain the multi-scale features of the blurred image;

[0063] Obtain the event sequence;

[0064] A bidirectional event encoder is used to extract multi-scale features from the acquired event sequence to obtain the multi-scale features of the event sequence;

[0065] For the multi-scale features of blurred images and the multi-scale features of event sequences, a multimodal fusion device is used to fuse features to obtain fused features;

[0066] The fused features are input into a bidirectional latent frame restorer to obtain an image of a clear frame sequence.

[0067] The bidirectional event encoder includes a convolutional layer, a first residual block, and a first bidirectional LSTM;

[0068] The first convolutional layer, the first residual module, and the first bidirectional LSTM are connected sequentially.

[0069] The input to each time step of the first bidirectional LSTM is the events within the exposure time of the corresponding sharp frame to be recovered. Each event segment is represented using a voxel grid, and the sequence is input forward into the bidirectional LSTM.

[0070] Then, the input sequence is reversed and fed into a bidirectional LSTM. Finally, the forward and reverse outputs are fused to ensure bidirectional information flow. The operation flow of the bidirectional LSTM is as follows:

[0071]

[0072]

[0073]

[0074] Where: (1) represents the forward update process of bidirectional LSTM, It is a hidden state with time step n during positive updates, controlling short-term time memory. It is a cell state with time step n in the positive update process, which controls long-term time memory. These represent hidden states and cell states in reverse updates;

[0075] (2) Represents the reverse update process of bidirectional LSTM. (3) represents the hidden state and cell state in the reverse update; (4) represents the final output of the bidirectional LSTM at time step n, which is the convolution result of splicing the forward and reverse hidden states.

[0076] Obtain the event sequence;

[0077] A bidirectional event encoder is used to extract multi-scale features from the acquired event sequence to obtain the multi-scale features of the event sequence;

[0078] First, the sequence is fed into the convolution module, and then the event sequence is fed into the bidirectional LSTM step by step to obtain the output of the corresponding scale. The multiple scales are connected in series.

[0079] The multimodal fusion unit fuses the multi-scale features of the blurred image and the multi-scale features of the event sequence to obtain the fused features. Within the multimodal fusion unit, each scale is operated on independently. First, the output of the image encoder (multi-scale blurred image information) and the output of the bidirectional event encoder (event feature sequence) are concatenated and fed into the convolutional module for fusion. After sigmoid activation, a mixed weight of the event and blurred image is obtained. This weight is first multiplied onto the event sequence and then added to the blurred image information to obtain the output of one scale.

[0080] The fused features are then fed into a bidirectional latent frame restorer. The bidirectional latent frame restorer accepts the multi-scale output of the multimodal fusion unit and images of different resolutions from the blurred map to progressively obtain a multi-scale sharp frame sequence.

[0081] Furthermore, the generation of the defuzzified dataset:

[0082] An event camera is a bio-inspired sensor that differs from traditional frame cameras: instead of capturing images at a fixed rate and without the concept of a shutter, it asynchronously measures pixel-by-pixel brightness changes and outputs a sequence of events, where the specific format of the events is an encoding of the time, location, and polarity of the brightness changes.

[0083] Currently, there is no large-scale event-grayscale deblurring dataset. This invention is based on the GoPro dataset and uses a V2E simulator for simulation. Furthermore, the principle of blurred frame synthesis is a temporal averaging of the clear frame sequence. This invention first uses Super-Slomo to interpolate the clear frame sequence to 1920fps, and then performs a temporal averaging on the interpolated sequence, resulting in a more realistic blurred image. In addition, this invention uses a voxel grid representation to convert asynchronous event sequences into synchronous event sequences.

[0084] A novel event-based image deblurring network is proposed, as follows:

[0085] The blurred image is first processed by an image encoder to obtain multi-scale image information. The event sequence is then processed by a bidirectional event encoder to obtain sequence feature output. The event sequence features and blurred image features are fed into a multimodal fusion unit for image-event fusion. A bidirectional latent frame restorer receives the fused features and predicts the deblurred residual output.

[0086] Figure 2This is a schematic diagram of an image encoder. The main steps of the image encoder to obtain multi-scale image information are as follows: First, the blurred image undergoes multi-scale downsampling using nearest-neighbor downsampling. Then, each scale is first processed through a convolutional layer. Finally, a multi-scale fusion module is used to fuse the information between the multiple scales. The multi-scale fusion module uses the features from the previous layer to predict a bias and spatial weight, which is then applied to the features at the current scale. Afterwards, the images are processed by residual blocks.

[0087] The image encoder is a multi-scale encoder structure, which includes a fourth convolutional layer, a third residual module, and a multi-scale fusion module connected in series.

[0088] The input to the image encoder is a single-frame blurred image, and the output is multi-scale information corresponding to the single-frame blurred image;

[0089] Multiscale fusion blocks are composed of convolutions and element-wise product sums.

[0090] The multi-scale fusion module has a two-branch convolutional structure that uses features from the mixed modalities to predict spatial weights and spatial biases respectively.

[0091] The blurred image undergoes two downsampling operations to form the information input for the residual scale structure. At each scale, the input is first convolved to extract features. Then, it receives the input from the residual block of the previous scale, predicts the scaling factor and bias, and applies them to the output of the convolutional module within the current scale. Finally, it passes through the residual block again for further feature extraction.

[0092] The process of extracting multi-scale features from the acquired blurred image using an image encoder to obtain the multi-scale features of the blurred image is as follows:

[0093] The image encoder at each scale passes through a fourth convolutional layer and a third residual block, where the scale at which the downsampling resolution is located also receives the output of the residual block from the previous scale, using it as the output of the convolution applied to the current scale as scaling factors and biases.

[0094] Figure 3 It is a bidirectional event encoder. The main steps of the bidirectional event encoder are: multi-scale processing of the received events (sequences), first convolution processing and then feeding them into bidirectional LSTM processing.

[0095] The bidirectional event encoder is a multi-scale encoder structure that includes convolutional layers, a first residual block, and a first bidirectional LSTM.

[0096] The first convolutional layer, the first residual module, and the first bidirectional LSTM are connected sequentially.

[0097] First, the event sequence is fed into a convolutional module to extract features. Then, a first bidirectional LSTM is used for temporal information modeling. The input to each time step of the first bidirectional LSTM is the event within the exposure time of the corresponding sharp frame to be recovered, and each event segment is represented using a voxel grid. After feeding the forward sequence into the first bidirectional LSTM, the input sequence is reversed and fed back into the first bidirectional LSTM. Finally, the forward and reverse outputs are fused to ensure bidirectional information flow. The three scales are concatenated, with residual blocks used for feature enhancement in between, finally yielding the outputs for the corresponding scales. The operation flow of the bidirectional LSTM at different scales is as follows:

[0098]

[0099] Figure 4 This is a schematic diagram of a multi-scale fusion module;

[0100] The multimodal fusion unit is composed of convolutional modules. It uses the features concatenated from the event-blurred image to predict spatial weights and spatial biases through convolution and sigmoid. These weights and biases are then applied to the event features and the blurred image features respectively, ultimately yielding the event-image hybrid features.

[0101] Figure 5 The diagram shows a multimodal fusion processor. The main steps of the multimodal fusion processor are: using the features of event-image stitching as spatial weights to apply to the event sequence features, and adding this result to the image features to obtain the enhanced event-image fusion features.

[0102] The input of the bidirectional latent frame restorer is the output of the multimodal fusion unit, which is a mixed modal sequence that simultaneously contains texture information of the blurred map and event information of the event sequence. The output of the bidirectional latent frame restorer is a clear frame output at any time resolution.

[0103] The bidirectional latent frame restorer is composed of a second residual module, a first decoder module, a second decoder module, a third decoder module, and an upsampling module connected in series.

[0104] The first decoder module, the second decoder module, and the third decoder module have the same structure;

[0105] The second residual module consists of two convolutional layers and residual jump connections.

[0106] The decoder module consists of a third residual module, a second convolutional layer, and a second bidirectional LSTM connected in series.

[0107] The upsampling module consists of a 2x bilinear interpolation method and a third convolutional layer.

[0108] The decoder process for recovering the clear frame sequence is as follows: First, the input mixed modality sequence is processed through a residual block to extract features, and then it is processed sequentially through the decoder module at each scale to obtain the output of the clear frame sequence at each scale.

[0109] The first decoder module is also a multi-scale structure. The output of each scale is a clear frame sequence with 4x downsampling, a clear frame sequence with 2x downsampling, and a clear frame sequence with the original resolution (all three scales are needed during training, but only the clear frame sequence with the original resolution is needed during testing).

[0110] Furthermore, each scale's decoder module is followed by a convolutional layer, which adds the output at this point to the downsampled result of the blurred image pair to obtain the final output.

[0111] In addition, the decoder also receives jump connections from the encoder to enable residual prediction of the encoded data.

[0112] Figure 6 As a bidirectional latent frame restorer, the decoder has a multi-scale structure, with scales connected using bilinear interpolation and convolution (the combination of which acts as upsampling). Furthermore, each scale has corresponding supervision and skip connections from the input blurred map, allowing the network to produce a residual output. The output sequence length of the second bidirectional LSTM is the same as the input event sequence length; the length of the output sequence is determined by the length of the input event sequence. By adjusting the temporal resolution of the input event sequence, the output sequence length of the bidirectional latent frame restorer can be controlled, thus achieving output at arbitrary temporal resolution.

[0113] The AdamW optimizer was chosen for the network, with an initial learning rate of 0.0004. The training resolution was 128*128, and the testing resolution was 320*180.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An event-image deblurring method based on LSTM, characterized in that: Includes the following steps: Construct an event-based image deblurring network model for processing blurred images; The event-image deblurring network is trained using the training set data in the fuzzy dataset, resulting in a well-trained event-image deblurring network model. The test set data from the fuzzy dataset is input into the trained event-image deblurring network model to obtain a sequence of clear frames; The process of inputting the test set data into the trained event-image deblurring network model to obtain a sequence of clear frames is as follows: An image encoder is used to extract multi-scale features from the acquired blurred image to obtain the multi-scale features of the blurred image; Obtain the event sequence; A bidirectional event encoder is used to extract multi-scale features from the acquired event sequence to obtain the multi-scale features of the event sequence; The multi-scale features of the blurred image and the multi-scale features of the event sequence are fused using a multimodal fusion tool to obtain the fused features. The fused features are input into a bidirectional latent frame restorer to obtain a clear frame sequence; The bidirectional event encoder includes a first convolutional layer, a first residual block, and a first bidirectional LSTM; The first convolutional layer, the first residual module, and the first bidirectional LSTM are connected sequentially. The decoder recovers the clear frame sequence as follows: First, the input mixed modal sequence is processed through a residual block to extract features, and then it is processed sequentially through the decoder module at each scale to obtain the preliminary output of the clear frame sequence at each scale. In addition, each scale's decoder module is followed by a convolutional layer, which adds the output at this point to the downsampling result corresponding to the blurred image to obtain the final output.

2. The LSTM-based event-image deblurring method according to claim 1, characterized in that: The input to each time step of the first bidirectional LSTM is the event within the exposure time corresponding to the sharp frame to be recovered. Each event segment is represented using a voxel grid. After the input sequence is fed into the bidirectional LSTM in forward order, the input sequence is then reversed and fed into the first bidirectional LSTM. Finally, the forward and reverse outputs are fused to ensure bidirectional information flow. The operation flow of the bidirectional LSTM is as follows: Where: (1) represents the forward update process of bidirectional LSTM, It is a hidden state with time step n during positive updates, controlling short-term time memory. (1) In the forward update, the cell state at time step n controls long-term temporal memory; (2) Represents the reverse update process of bidirectional LSTM. (3) represents the hidden state and cell state in the reverse update; (4) represents the final output of the bidirectional LSTM at time step n, which is the convolution result of splicing the forward and reverse hidden states.

3. The LSTM-based event-image deblurring method according to claim 1, characterized in that: The input of the bidirectional latent frame restorer is the output of the multimodal fusion unit, which is a mixed modal sequence containing both the texture information of the blurred map and the temporal information of the event sequence. The output of the bidirectional latent frame restorer is a clear frame sequence with arbitrary temporal resolution. The bidirectional latent frame restorer is composed of a second residual module, a first decoder module, a second decoder module, a third decoder module, and an upsampling module connected in series. The first decoder module, the second decoder module, and the third decoder module have the same structure; The second residual module consists of two convolutional layers and residual jump connections. The decoder module consists of a third residual module, a second convolutional layer, and a second bidirectional LSTM connected in series. The upsampling module consists of a 2x bilinear interpolation method and a third convolutional layer.

4. The LSTM-based event-image deblurring method according to claim 1, characterized in that: The image encoder comprises a fourth convolutional layer, a third residual module, and a multi-scale fusion module connected in series. The input to the image encoder is a single-frame blurred image, and the output is multi-scale information corresponding to the single-frame blurred image; Multiscale fusion blocks are composed of convolutions, element-wise multiplications, and element-wise summations.

5. The LSTM-based event-image deblurring method according to claim 1, characterized in that: The process of extracting multi-scale features from the acquired blurred image using an image encoder is as follows: The image encoder at each scale passes through a fourth convolutional layer and a third residual block, where the scale of the downsampled image also receives the output of the residual block from the previous scale, which is used as the scaling factor and bias applied to the output of the convolution at the current scale.

Images