Event camera based motion deblurring and image sequence reconstruction method and apparatus

Abstract

This application discloses a method and apparatus for motion deblurring and image sequence reconstruction based on an event camera. The method includes: capturing a clear video sequence with a high-speed camera to simulate a blurred image and then obtaining an event stream, wherein the event stream represents a set of time events triggered within the entire time period; preprocessing the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to a convolutional neural network; establishing a convolutional neural network based on a constructed dual-attention Transformer module and a decoder that outputs a target image according to a target timestamp, training the convolutional neural network using a supervised framework to obtain a target convolutional neural network; reconstructing the target image by passing the event tensors and the blurred image through the trained and optimized target convolutional neural network, inputting the target timestamp, and obtaining a deblurred image sequence.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method and apparatus for motion deblurring and image sequence reconstruction based on an event camera. Background Technology

[0002] Due to the limitations of the physical structure of traditional optical camera shutters, their exposure times are relatively long. When there is relative motion between the camera and the subject being photographed, motion blur often occurs. This not only reduces the quality of the captured image but also adversely affects downstream image processing tasks that require clear object information, such as image segmentation and target recognition.

[0003] Because blurred frame images lack information about object motion and texture, traditional frame-based motion deblurring methods require certain prior assumptions about motion patterns. However, these assumptions do not fully reflect the complex motion scenarios in real-world scenes, and the lack of texture details also hinders the reconstruction of image sequences using traditional methods. Therefore, traditional methods often perform poorly in deblurring in real-world scenes.

[0004] An event camera, a visual sensor based on biological neural mechanisms, has emerged as a breakthrough in solving this problem. Unlike traditional optical cameras, event cameras can record changes in brightness within a scene, generate asynchronous events with microsecond-level delays, and record the coordinates, time, and polarity of the changes. They offer high temporal resolution while also being characterized by low power consumption and low bandwidth requirements. Event cameras can record the motion and edge texture information of targets in high-speed moving scenes.

[0005] Compared to traditional frame-based methods, event-based methods achieve better deblurring performance. However, due to imperfections in the network structure, existing methods struggle to achieve efficient feature extraction and cross-modal fusion for both blurred images and events, hindering further improvements in deblurring performance. Summary of the Invention

[0006] To address at least one deficiency or improvement requirement of the prior art, the present invention provides a method and apparatus for motion deblurring and image sequence reconstruction based on an event camera. This method solves the problems of loss of texture detail information caused by motion blur and the inability to fully realize feature compensation between blurred images and events. It has a better deblurring and image reconstruction effect and achieves high efficiency and applicability for deblurring blurred images.

[0007] To achieve the above objectives, according to a first aspect of the present invention, a method for motion deblurring and image sequence reconstruction based on an event camera is provided. The method includes: capturing a clear video sequence using a high-speed camera to simulate a blurred image and then obtaining an event stream, wherein the event stream represents a set of events triggered within a given time period; preprocessing the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to a convolutional neural network; establishing a convolutional neural network based on a constructed dual-attention Transformer module and a decoder that outputs a target image according to a target timestamp; training the convolutional neural network using a supervised framework to obtain a target convolutional neural network; and reconstructing the target image by passing the event tensors and the blurred image through the trained and optimized target convolutional neural network, inputting the target timestamp, and obtaining a deblurred image sequence.

[0008] In an exemplary embodiment, the step of simulating a blurred image by capturing a clear video sequence with a high-speed camera and then obtaining an event stream includes: selecting multiple consecutive frames from the clear video sequence captured by the high-speed camera, summing them, and averaging them to obtain a simulated blurred image; generating the event stream from the clear video sequence captured by the high-speed camera; or, directly capturing the corresponding event stream using an event camera.

[0009] In an exemplary embodiment, the preprocessing of the event stream to obtain a series of event tensors includes: converting the absolute timestamps of the event stream into normalized relative time shifts; dividing the event stream into multiple time intervals on an average basis according to the time span; counting the total number of positive and negative events and the time surface at each pixel, wherein the time surface is a 2D plane mapping that represents storing the timestamp of the last event at the corresponding position at each pixel; each time interval constitutes an event frame of a corresponding scale; and merging the event frames of the corresponding time interval into a multi-channel tensor.

[0010] In one exemplary embodiment, the step of establishing a convolutional neural network based on the constructed dual-attention Transformer module and a decoder that outputs a target image according to a target timestamp, and training the convolutional neural network using a supervised framework to obtain the target convolutional neural network includes: after establishing the convolutional neural network, inputting the event stream and the blurred image into the convolutional neural network, performing deep feature extraction and mutual compensation in a dual-attention Transformer sequence, wherein the dual-attention Transformer sequence is composed of multiple dual-attention Transformer modules connected in series; after feature extraction and mutual compensation of the blurred image and the event stream in the dual-attention Transformer sequence, obtaining deblurred features; inputting the deblurred features into the decoder for decoding, and decoding and outputting the corresponding potentially clear image according to the input timestamp.

[0011] In one exemplary embodiment, after training the convolutional neural network using a supervised framework, the method further includes: outputting a deblurred image with a reference timestamp, supervised using ground truth; and additionally outputting a deblurred image without a reference timestamp, supervised using motion-guided supervision.

[0012] According to a second aspect of the present invention, a motion deblurring and image sequence reconstruction apparatus based on an event camera is also provided, comprising: an acquisition unit, which acquires an event stream by simulating a blurred image after capturing a clear video sequence with a high-speed camera, wherein the event stream represents a set of time events triggered within a whole time period; a preprocessing unit, which preprocesses the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to a convolutional neural network; a building unit, which builds a convolutional neural network based on a constructed dual-attention Transformer module and a decoder that outputs a target image according to a target timestamp, and trains the convolutional neural network using a supervised framework to obtain a target convolutional neural network; and a reconstruction unit, which reconstructs a target image by passing the event tensors and the blurred image through the trained and optimized target convolutional neural network, inputting the target timestamp, and acquires a deblurred image sequence.

[0013] According to a third aspect of the invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the above-described motion deblurring and image sequence reconstruction method based on an event camera when it is run.

[0014] According to a fourth aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described motion deblurring and image sequence reconstruction method based on an event camera through the computer program.

[0015] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0016] This invention provides a motion deblurring and image sequence reconstruction method based on an event camera. The motion image deblurring method based on an event camera can extract and compensate for the feature information of a single blurred image and its corresponding event. Using a decoder, the potentially sharp image of the target time can be recovered from the deblurred features, further reconstructing a continuous sharp sequence. Simultaneously, a supervised framework with two constraints is proposed to train the network, leveraging the low latency advantage of the event camera to solve the problems of motion blur and image detail and texture erasure. The method is trained on a simulation dataset using both constraints and exhibits good deblurring and reconstruction performance in real-world scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart illustrating an optional motion deblurring and image sequence reconstruction method based on an event camera, provided in an embodiment of this application;

[0019] Figure 2 A schematic diagram of an optional dual-attention Transformer module structure for motion deblurring provided in an embodiment of this application;

[0020] Figure 3 A schematic diagram of the structure of an optional decoder for reconstructing a target temporal image provided in an embodiment of this application;

[0021] Figure 4 A schematic diagram of an optional network model DAC-Net provided in an embodiment of this application;

[0022] Figure 5 A schematic diagram of an optional motion deblurring and image sequence reconstruction device based on an event camera provided in this application embodiment;

[0023] Figure 6This is a schematic diagram of an optional electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0026] According to one aspect of the embodiments of this application, a method for motion deblurring and image sequence reconstruction based on an event camera is provided. The following is in conjunction with... Figure 1 This application describes a motion deblurring and image sequence reconstruction method based on an event camera, as provided in the embodiments of this application.

[0027] Figure 1 This is a schematic flowchart of an optional event camera-based motion deblurring and image sequence reconstruction method provided in an embodiment of this application, as shown below. Figure 1 As shown, the process of this method may include the following steps:

[0028] S102, after capturing a clear video sequence with a high-speed camera to simulate a blurred image, an event stream is obtained, where the event stream represents the set of times triggered within the entire time period;

[0029] S104, preprocess the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to the convolutional neural network;

[0030] S106. A convolutional neural network is built based on the constructed dual-attention Transformer module and the decoder that outputs the target image according to the target timestamp. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network.

[0031] S108: The event tensor and the blurred image are trained and optimized into a target convolutional neural network. The target image is reconstructed by inputting the target timestamp, and the deblurred image sequence is obtained.

[0032] The motion deblurring and image sequence reconstruction method based on an event camera provided in this application can be applied to scenarios that achieve blurry image deblurring and image reconstruction.

[0033] Optionally, by constructing a dataset of blurred images and event streams, and preprocessing the event streams, a network model is built that includes an event-based dual-attention Transformer module and a decoder to process blurred images and event streams. A framework with two types of supervision constraints is used for network training to generate deblurred image sequences. Based on feature extraction and mutual compensation of event streams and blurred images, a dual-attention Transformer module is constructed. The decoder recovers the potential sharp image of the target time from the deblurred features, further reconstructing a continuous sequence of sharp images. Utilizing the constraint relationship between reference timestamp ground truth and motion guidance under non-reference timestamps on the reconstructed images, a dual-supervision framework is proposed to train the network model, and the trained network model is used to reconstruct the deblurred image sequence.

[0034] For example, blurred images can be simulated by capturing clear video sequences with a high-speed camera, and event streams can be simulated by capturing clear video sequences, or event streams can be captured directly using an event camera. Event stream preprocessing: For each set of input event data, it is encoded into a specific multi-channel tensor format as input to the network. Based on feature extraction and mutual compensation of events and blurred images, a dual-attention Transformer module is constructed; a decoder that can output target images based on target timestamps is constructed; a convolutional neural network is built based on the above module and decoder. The network is trained using a supervised framework: In the first stage, the deblurred image with a reference timestamp is output, supervised using ground truth (GT); in the second stage, the deblurred image without a reference timestamp is additionally output, supervised using motion-guided supervision. Event streams are captured using an event camera, and blurred images are captured using an optical camera. After processing the event streams, event tensors that can be input into the network are obtained. The event tensors and blurred images are used to train and optimize the network model, and the target image is reconstructed by inputting the target timestamp, and further deblurred image sequences are obtained.

[0035] Through steps S102 to S108, an event stream is obtained by capturing a clear video sequence with a high-speed camera to simulate a blurred image. The event stream represents the set of events triggered within the entire time period. The event stream is preprocessed to obtain a series of event tensors. Preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to a convolutional neural network. A convolutional neural network is built based on the constructed dual-attention Transformer module and a decoder that outputs the target image according to the target timestamp. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network. The event tensors and blurred image are then used to reconstruct the target image by inputting the target timestamp, and a deblurred image sequence is obtained. This solves the problems of texture detail loss caused by motion blur and the inability to fully realize feature compensation between blurred images and events, achieving good deblurred image reconstruction results and demonstrating high efficiency and applicability for deblurring blurred images.

[0036] In one exemplary embodiment, obtaining an event stream after simulating a blurred image by capturing a clear video sequence with a high-speed camera includes:

[0037] S11: Select multiple consecutive frames of images from a clear video sequence captured by a high-speed camera, sum them, and take the average to obtain a simulated blurred image.

[0038] S12, generate an event stream from a clear video sequence captured by a high-speed camera; or, directly capture the corresponding event stream using an event camera.

[0039] In this embodiment, a clear video sequence captured by a high-speed camera is used to simulate the blurred image B; the event flow is also simulated by capturing a clear video sequence. Alternatively, you can directly capture the event stream using an event camera. A blurred image B is generated by capturing clear video using a high-speed camera. The specific steps are as follows: According to the principle of blurred image generation, select J consecutive frames (J can be 49 in actual operation) from the clear video, sum them, and take the average to obtain the simulated blurred image; finally, generate an event stream from the captured video, or directly use an event camera to capture the corresponding event stream.

[0040] For example, the process of simulating the formation of a blurred image is as follows:

[0041]

[0042] Where B is the simulated blurred image, I n For a sequence {I} containing J frames of clear images n The nth frame in}.

[0043] Optionally, the generated event stream can be generated in the following ways, based on the event stream. and clear video The relationship between the two events is as follows: when the brightness change in the logarithmic domain exceeds the event trigger threshold η>0:

[0044]

[0045] From time τ to time t, video When the change in the logarithmic value of brightness at pixel coordinate x exceeds the threshold η, an event point (t,x,p) is generated, where p is the event polarity: 1 when the logarithmic value of brightness increases and -1 when it decreases; event stream This represents the set of times that are triggered within the entire time period.

[0046] Furthermore, the synthesized blurred image B and the event stream can be obtained. Select a blurred image and its corresponding event stream from the data as a set of data, and set the target time t as the time for reconstructing the sharp latent image, as follows:

[0047]

[0048] Where B represents the blurred image, Indicates the exposure time of a blurred image The event stream within, where t is the time of the sharp potential image to be reconstructed.

[0049] In one exemplary embodiment, the event stream is preprocessed to obtain a series of event tensors, including:

[0050] S21 converts the absolute timestamps of the event stream into normalized relative time displacements;

[0051] S22, divide the event stream into multiple time intervals on an average basis according to the time span, and count the total number of positive and negative events and the time surface at each pixel. The time surface is a 2D plane mapping, which means that the timestamp of the last event at the corresponding position is stored at each pixel.

[0052] S23, each time interval constitutes an event frame of the corresponding scale, and the event frames of the corresponding time period are merged into a multi-channel tensor.

[0053] In this embodiment of the application, for example, during the event stream preprocessing, each set of input event data is encoded into a specific multi-channel tensor format as input to the network. First, the absolute timestamps of the event stream are converted into normalized relative time shifts, denoted as... Event Stream The processing method for a single event (t, x, p) is represented as follows:

[0054]

[0055] in and These are the exposure start time and exposure end time, respectively.

[0056] The specific process for converting the event stream after timestamp normalization into a multi-channel tensor format can be as follows: Divide the event stream into T time intervals on an average basis (T can be 6 in practice), count the total number of positive and negative events at each pixel and their time surface, where the time surface is a 2D plane mapping that represents storing the timestamp of the last event at each pixel; finally, each time interval constitutes a 4×H×W event frame (H and W are the height and width of the event frame, respectively), and merge the event frames of these T time intervals into a 4T×H×W multi-channel tensor.

[0057] In one exemplary embodiment, a convolutional neural network is built based on a constructed dual-attention Transformer module and a decoder that outputs a target image according to the target timestamp. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network, which includes:

[0058] S31, After establishing the convolutional neural network, the event stream and blurred image are input into the convolutional neural network. Deep feature extraction and mutual compensation are performed in the dual attention Transformer sequence. The dual attention Transformer sequence is composed of multiple dual attention Transformer modules connected in series.

[0059] S32, after feature extraction and mutual complementation in the dual-attention Transformer sequence, the blurred image and event stream obtain deblurred features;

[0060] S33, input the deblurred features into the decoder for decoding, and output the corresponding potential sharp image based on the input timestamp.

[0061] In this embodiment, combined with Figure 2 , Figure 3 as well as Figure 4As shown, the convolutional neural network model mainly consists of two parts: a dual-attention Transformer sequence and a decoder. After constructing the convolutional neural network, the event stream and the blurred image are first input into the network. Each image undergoes shallow feature extraction through a 3×3 convolutional layer. Then, deep feature extraction and mutual compensation are performed in the dual-attention Transformer sequence. The dual-attention Transformer sequence is composed of N dual-attention Transformer modules cascaded together (for example, N can be 20 in practice).

[0062] The input signal is processed through a linear layer in the Transformer to generate a query vector Q, a key vector K, and a value vector V, and the attention Attn is calculated using the following formula.

[0063]

[0064] Among them, K T Let d be the rank matrix of the key vector K. k Let Q, K, and V be the dimensions of the vectors.

[0065] In the attention calculation of the i-th (1≤i≤N) dual-attention Transformer module, the event weights Blurred image weights The correction is as follows:

[0066]

[0067] After the event weights are corrected by the blurred image weights, noise in the events can be reduced. The weighted weights of the events and the blurred image are calculated using the following formula. and

[0068]

[0069] Among them, V e and V b These are the value vectors for the event and the blurred image, respectively. Finally, their output features are calculated using a multilayer perceptron within each channel. and

[0070]

[0071] By extracting the weighted features of the event and the blurred image, the output features of the blurred image can be enhanced, such as edge texture, thereby achieving a deblurring effect. After feature extraction and mutual complementation between the blurred image and the event in a dual-attention Transformer sequence, the deblurring feature F can be obtained. dbThe input timestamp is fed into the decoder for decoding. The decoder consists of a multilayer perceptron and a convolutional layer with a 3×3 kernel, which can decode and output the corresponding potentially sharp image based on the input timestamp. The input target timestamp t (where t∈[0,1]) is used to generate a high-dimensional vector η(t) of length 2L using Fourier encoding (in practice, L can be 8).

[0072] η(t)=(cos(2 0 πt),sin(2 0 πt),…,cos(2 L-1 πt),sin(2 L-1 πt)),

[0073] After the high-dimensional vector η(t) enters the decoder, it undergoes dimensionality expansion to form a 2L×H×W tensor, which is then combined with the deblurred feature F that has passed through the convolutional layer. db Combined, a latent sharp image I(t) for the final target time is generated through a multilayer perceptron. The overall network model is as follows:

[0074]

[0075] DCA-Net is the name of the network.

[0076] In this embodiment, a dual-attention Transformer module is used for motion image deblurring based on an event camera. It can extract feature information of a single blurred image and the corresponding event and compensate for each other. Through the decoder, the potential clear image of the target time can be recovered from the deblurred features, and a continuous clear sequence can be reconstructed.

[0077] In one exemplary embodiment, after training the convolutional neural network using a supervised framework, the method further includes:

[0078] S41, outputs a deblurred image with reference timestamp, supervised using ground truth;

[0079] S42, additionally outputs a deblurred image with a non-reference timestamp, using motion-guided supervision.

[0080] In this embodiment, optionally, a supervised framework can be used to train the network: in the first stage, a deblurred image with a reference timestamp is output and supervised using ground truth (GT); in the second stage, a deblurred image without a reference timestamp is additionally output and supervised using motion-guided supervision.

[0081] For example, the overall supervised framework is as follows:

[0082]

[0083] in, To guide the supervision of ground truth. The constraints are used for motion guidance and supervision; λ1 and λ2 are weight indicators. In specific implementation, the first stage is set to 1 and 0, and the second stage is set to 0.2 and 1.

[0084] Specifically, the ground truth-guided supervision process is as follows: a blurry image is synthesized from J clear and sharp images, and there are J reference timestamps. Based on this, select K reference timestamps (K≤J). Corresponding sharp and clear image As ground truth images, constraints are applied to the reconstructed image sequence at the reference timestamp. In practice, images 1, 9, 17, 25, 33, 41, and 49 from a sequence of 49 clear images can be selected as ground truth images at the reference timestamp. Ground truth guides the constraints of oversight. Defined as:

[0085]

[0086] Where ||·||1 represents the l1 norm.

[0087] Furthermore, the specific process of motion guidance and supervision is as follows: for each non-reference timestamp exist Each of them has two corresponding most recent reference timestamps t. j and t j+1 , so that t′ m ∈(t j ,t j+1 ), where 1≤j≤J-1. Then, using the sharp, clear image I(t) at the corresponding time. j ) and I(t j+1 ), and each to t′ m event set between and t′ is generated by Timelens transformation using an event-based frame interpolation network. m Potential image of time

[0088]

[0089] During the training phase, M (M≤J) non-reference timestamps are randomly selected. Motion guidance and supervision are implemented. In practice, M can be set to 3. Constraints for motion guidance and supervision. Defined as:

[0090]

[0091] Where ||·||1 represents the l1 norm.

[0092] It should be noted that the optimization method in the training process is the Adaptive Moment Estimation (Adam) optimization algorithm. This algorithm designs independent adaptive learning rates for different parameters by calculating the first and second moment estimates of the gradient. Based on the training data, iteratively updates the parameters of network layers such as convolutional layers in the neural network. The optimized neural network is then used to construct a multi-layer convolutional neural network based on the optimized parameters of each network layer. An event camera captures an event stream, while an optical camera captures blurred images. The event stream is processed to obtain an event tensor that can be input into the network. The event tensor and the blurred images are then used to train the optimized network model, and the target image is reconstructed by inputting the target timestamp. Finally, a deblurred image sequence is obtained.

[0093] In this embodiment, a supervised framework with two constraints is used to train the network. Taking advantage of the low latency of the event camera, the problems of motion blur and image detail and texture erasure are solved. The network is trained on a simulation dataset using two constraints and has good deblurring and reconstruction effect in real scenes.

[0094] According to another aspect of the embodiments of this application, an apparatus for implementing the above-described motion deblurring and image sequence reconstruction method based on an event camera is also provided. Figure 5 This is a schematic diagram of an optional event camera-based motion deblurring and image sequence reconstruction apparatus according to an embodiment of this application, as shown below. Figure 5 As shown, the device may include:

[0095] The acquisition unit 502 acquires an event stream after simulating a blurred image by capturing a clear video sequence with a high-speed camera, wherein the event stream represents a set of time events triggered within the entire time period;

[0096] The preprocessing unit 504 preprocesses the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to a convolutional neural network.

[0097] Unit 506 is established by building a convolutional neural network based on the constructed dual-attention Transformer module and the decoder that outputs the target image according to the target timestamp. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network.

[0098] The reconstruction unit 508 reconstructs the target image by training and optimizing the event tensor and the blurred image through a target convolutional neural network, inputting the target timestamp, and obtaining the deblurred image sequence.

[0099] It should be noted that the acquisition unit 502 in this embodiment can be used to perform the above step S102, the preprocessing unit 504 in this embodiment can be used to perform the above step S104, the establishment unit 506 in this embodiment can be used to perform the above step S106, and the reconstruction unit 508 in this embodiment can be used to perform the above step S108.

[0100] Through the aforementioned modules, a clear video sequence captured by a high-speed camera is used to simulate a blurred image and obtain an event stream, where the event stream represents the set of events triggered within the entire time period. The event stream is preprocessed to obtain a series of event tensors, where each set of input event data is encoded into a specific multi-channel tensor format as input to a convolutional neural network (CNN). A CNN is built based on a constructed dual-attention Transformer module and a decoder that outputs the target image according to the target timestamp. The CNN is trained using a supervised framework to obtain the target CNN. The event tensors and blurred image are then used to reconstruct the target image by inputting the target timestamp, and a deblurred image sequence is obtained. This solves the problems of texture detail loss caused by motion blur and the inability to fully realize feature compensation between blurred images and events, achieving good deblurred image reconstruction results and demonstrating high efficiency and applicability for deblurring blurred images.

[0101] In one exemplary embodiment, the acquisition unit includes:

[0102] The first acquisition module is used to select multiple consecutive frames of images from the clear video sequence captured by the high-speed camera, sum them and take the average to obtain a simulated blurred image.

[0103] The second acquisition module is used to generate the event stream from a clear video sequence captured by the high-speed camera; or, to directly capture the corresponding event stream using the event camera.

[0104] In one exemplary embodiment, the preprocessing unit includes:

[0105] The conversion module is used to convert the absolute timestamps of the event stream into normalized relative time displacements;

[0106] The segmentation module is used to divide the event stream into multiple time intervals on an average basis according to the time span, and to count the total number of positive and negative events and the time surface at each pixel. The time surface is a 2D plane mapping, which represents the timestamp of the last event at the corresponding position stored at each pixel.

[0107] The merging module is used to construct event frames of the corresponding scale for each time interval and merge the event frames of the corresponding time period into a multi-channel tensor.

[0108] In one exemplary embodiment, the establishment unit includes:

[0109] A module is established to input the event stream and blurred image into the convolutional neural network after the convolutional neural network is established, and to perform deep feature extraction and mutual compensation in a dual-attention Transformer sequence, wherein the dual-attention Transformer sequence is composed of multiple dual-attention Transformer modules connected in series;

[0110] An extraction module is used to obtain deblurred features from the blurred image and event stream after feature extraction and mutual complementation in a dual-attention Transformer sequence;

[0111] The decoding module is used to input the deblurred features into the decoder for decoding, and output the corresponding potentially clear image based on the input timestamp.

[0112] In one exemplary embodiment, the apparatus further includes:

[0113] The first supervised unit is used to output a deblurred image with a reference timestamp, and is supervised using ground truth.

[0114] The second supervisory unit, used to additionally output deblurred images without reference timestamps, employs motion-guided supervision.

[0115] It should be noted that the examples and scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run in a hardware environment and can be implemented by software or hardware. The hardware environment includes a network environment.

[0116] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above-described event camera-based motion deblurring and image sequence reconstruction methods in the embodiments of this application.

[0117] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0118] S1, after capturing a clear video sequence with a high-speed camera to simulate a blurred image, obtain an event stream, where the event stream represents the set of times triggered within the entire time period;

[0119] S2, preprocess the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to the convolutional neural network;

[0120] S3. Based on the constructed dual-attention Transformer module and the decoder that outputs the target image according to the target timestamp, a convolutional neural network is built. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network.

[0121] S4 reconstructs the target image by training and optimizing the target convolutional neural network with the event tensor and the blurred image as input, and obtains the deblurred image sequence.

[0122] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0123] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0124] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described motion deblurring and image sequence reconstruction method based on an event camera is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0125] Figure 6 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application, such as... Figure 6 As shown, it includes a processor 602, a communication interface 604, a memory 606, and a communication bus 608. The processor 602, communication interface 604, and memory 606 communicate with each other via the communication bus 608.

[0126] Memory 606 is used to store computer programs;

[0127] When processor 602 executes a computer program stored in memory 606, it performs the following steps:

[0128] S1, after capturing a clear video sequence with a high-speed camera to simulate a blurred image, obtain an event stream, where the event stream represents the set of times triggered within the entire time period;

[0129] S2, preprocess the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to the convolutional neural network;

[0130] S3. Based on the constructed dual-attention Transformer module and the decoder that outputs the target image according to the target timestamp, a convolutional neural network is built. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network.

[0131] S4 reconstructs the target image by training and optimizing the target convolutional neural network with the event tensor and the blurred image as input, and obtains the deblurred image sequence.

[0132] Optionally, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0133] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0134] As an example, the memory 606 described above may include, but is not limited to, the acquisition unit 502, preprocessing unit 504, establishment unit 506, and reconstruction unit 508 of the event-based camera motion deblurring and image sequence reconstruction apparatus described above. Furthermore, it may include, but is not limited to, other module units of the event-based camera motion deblurring and image sequence reconstruction apparatus described above, which will not be elaborated upon in this example.

[0135] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0136] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0137] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0141] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0142] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0143] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0144] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for motion deblurring and image sequence reconstruction based on an event camera, characterized in that, include: After simulating a blurred image by capturing a clear video sequence with a high-speed camera, an event stream is obtained, wherein the event stream represents the set of times triggered within the entire time period; The event stream is preprocessed to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to a convolutional neural network. A convolutional neural network is built based on the constructed dual-attention Transformer module and the decoder that outputs the target image according to the target timestamp. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network. The event tensor and the blurred image are trained and optimized into a target convolutional neural network. The target image is reconstructed by inputting the target timestamp, and the deblurred image sequence is obtained.

2. The motion deblurring and image sequence reconstruction method based on an event camera as described in claim 1, characterized in that, The process of simulating a blurred image by capturing a clear video sequence with a high-speed camera and then obtaining the event stream includes: A simulated blurred image is obtained by selecting multiple consecutive frames from a clear video sequence captured by the high-speed camera, summing them, and averaging them. The event stream is generated from a clear video sequence captured by the high-speed camera; or, the corresponding event stream is captured directly using the event camera.

3. The motion deblurring and image sequence reconstruction method based on an event camera as described in claim 1, characterized in that, The preprocessing of the event stream yields a series of event tensors, including: The absolute timestamps of the event stream are converted into normalized relative time shifts; The event stream is divided into multiple time intervals on an average basis according to the time span. The total number of positive and negative events and the time surface at each pixel are counted. The time surface is a 2D plane mapping, which represents the timestamp of the last event at the corresponding position stored at each pixel. Each time interval constitutes an event frame of the corresponding scale, and the event frames of the corresponding time period are merged into a multi-channel tensor.

4. The motion deblurring and image sequence reconstruction method based on an event camera as described in claim 1, characterized in that, The convolutional neural network is established based on the constructed dual-attention Transformer module and the decoder that outputs the target image according to the target timestamp. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network, which includes: After establishing the convolutional neural network, the event stream and the blurred image are input into the convolutional neural network. Deep feature extraction and mutual compensation are performed in the dual attention Transformer sequence, wherein the dual attention Transformer sequence is composed of multiple dual attention Transformer modules connected in series. The blurred image and event stream are processed and complemented in a dual-attention Transformer sequence to obtain deblurred features; The deblurred features are input into the decoder for decoding, and the corresponding potentially clear image is output based on the input timestamp.

5. The motion deblurring and image sequence reconstruction method based on an event camera as described in claim 1, characterized in that, After training the convolutional neural network using a supervised framework, the method further includes: Output a deblurred image with reference timestamp, supervised using ground truth; Additional output is a deblurred image with a non-reference timestamp, using motion-guided supervision.

6. A device for motion deblurring and image sequence reconstruction based on an event camera, characterized in that, include: The acquisition unit acquires an event stream by simulating a blurred image after capturing a clear video sequence with a high-speed camera, wherein the event stream represents a set of time events triggered within the entire time period; The preprocessing unit preprocesses the event stream to obtain a series of event tensors, wherein the preprocessing involves encoding each set of input event data into a specific multi-channel tensor format as input to a convolutional neural network. The unit is established by building a convolutional neural network based on the constructed dual-attention Transformer module and the decoder that outputs the target image according to the target timestamp. The convolutional neural network is trained using a supervised framework to obtain the target convolutional neural network. The reconstruction unit reconstructs the target image by training and optimizing the event tensor and the blurred image through a target convolutional neural network, inputting the target timestamp, and obtains the deblurred image sequence.

7. The motion deblurring and image sequence reconstruction apparatus based on an event camera as described in claim 6, characterized in that, The acquisition unit includes: The first acquisition module is used to select multiple consecutive frames of images from the clear video sequence captured by the high-speed camera, sum them and take the average to obtain a simulated blurred image. The second acquisition module is used to generate the event stream from a clear video sequence captured by the high-speed camera; or, to directly capture the corresponding event stream using the event camera.

8. The motion deblurring and image sequence reconstruction apparatus based on an event camera as described in claim 6, characterized in that, The preprocessing unit includes: The conversion module is used to convert the absolute timestamps of the event stream into normalized relative time displacements; The segmentation module is used to divide the event stream into multiple time intervals on an average basis according to the time span, and to count the total number of positive and negative events and the time surface at each pixel. The time surface is a 2D plane mapping, which represents the timestamp of the last event at the corresponding position stored at each pixel. The merging module is used to construct event frames of the corresponding scale for each time interval and merge the event frames of the corresponding time period into a multi-channel tensor.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 6.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 through the computer program.

Images