Deblurring method and system, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a deblurring method and system, electronic equipment and a computer readable storage medium. The deblurring method includes: generating a motion stream based on an event camera signal corresponding to a blurred image, wherein the motion stream characterizes an explicit relationship between the event camera signal and an action causing blurring of the blurred image; performing feature extraction on the event camera signal and the blurred image to obtain event features and image features; and restoring the blurred image into a clear result image according to the motion flow, the event features and the image features.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a deblurring method and system. Furthermore, this application also relates to an electronic device and a computer-readable storage medium. Background Art

[0002] Deblurring methods mainly include blind deblurring and non-blind deblurring. Traditional deblurring methods typically utilize multiple priors, such as total variation priors and heavy-tailed gradient priors, for fuzzing.

[0003] Furthermore, with the rapid development of deep learning technology, deblurring methods based on CNN (Convolutional Neural Network), GAN (Generative Adversarial Networks), and RNN (Recurrent Neural Network) have been widely used, such as DeblurGAN (Deblur Generative Adversarial Networks) and DeblurGAN-v2 (Deblur Generative Adversarial Networks version 2). However, these deblurring methods mostly focus on recovering the clear image itself from the blurred image, while ignoring the source of the image blur.

[0004] Due to the lack of modeling of the fuzzing process and the scarcity of training data, most of the aforementioned defuzzification methods rely on data augmentation to increase the number of training data samples. However, in most cases, this leads to the restored result deviating significantly from the true value. Summary of the Invention

[0005] This application provides a deblurring method, system, electronic device, and computer-readable storage medium that can at least partially solve the aforementioned problems existing in the related art.

[0006] This application provides a deblurring method, comprising: generating a motion stream based on an event camera signal corresponding to a blurred image, wherein the motion stream characterizes an explicit relationship between the event camera signal and the action that causes the blurred image to blur; performing feature extraction on the event camera signal and the blurred image to obtain event features and image features; and restoring the blurred image to a clear result image based on the motion stream, the event features, and the image features.

[0007] In one embodiment of this application, generating a motion stream based on an event camera signal corresponding to a blurred image and the blurred image includes: dividing the exposure time of the blurred image into multiple intervals; segmenting and preprocessing the event camera signal into an event voxel grid corresponding to each of the multiple intervals based on the multiple intervals; performing feature extraction on the blurred image and the event voxel grid to obtain the motion stream; and training the motion stream, wherein the motion stream includes a feature map corresponding to each of the multiple intervals; and the feature map includes the displacement between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0008] In one embodiment of this application, generating a motion stream based on an event camera signal corresponding to a blurred image includes: dividing the exposure time of the blurred image into multiple intervals; segmenting and preprocessing the event camera signal into an event voxel grid corresponding to each of the multiple intervals based on the multiple intervals; performing feature extraction on the blurred image and the event voxel grid to obtain the motion stream; and training the motion stream, wherein the motion stream includes a feature map corresponding to each of the multiple intervals; and the feature map includes the displacement between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0009] In one embodiment of this application, the motion flow is trained by a self-supervised process of the sharp image going from sharp to blurry and then back to blurry; or the motion flow is trained based on a dataset of sharp images within the exposure time corresponding to the blurred image.

[0010] In one embodiment of this application, training the motion flow through a self-supervised process of blurring the clear image corresponding to the blurred image from clear to blurred and then blurring again includes: distorting the clear image based on the pixel displacements of multiple feature maps to generate multiple distorted images; generating a constructed blurred image based on the average value of the multiple distorted images and the motion flow; adjusting the motion flow based on the difference between the blurred image and the constructed blurred image; and repeating the above steps until the difference is less than a predetermined value.

[0011] In one embodiment of this application, the re-blurring process is performed using a re-blurring network, and the method further includes training the re-blurring network, wherein training the re-blurring network includes optimizing the re-blurring network based on a loss function, wherein the loss function includes the L1 norm between the blurred image and the constructed blurred image.

[0012] In one embodiment of this application, the motion stream includes a cross-frame motion stream. Generating the motion stream based on an event camera signal corresponding to a blurred image and the blurred image includes: generating the cross-frame motion stream based on an event camera signal corresponding to multiple frames of the blurred image and the multiple frames of the blurred image, wherein the cross-frame motion stream represents an explicit relationship between the event camera signal corresponding to the multiple frames of the blurred image and the action that causes the multiple frames of the blurred image to be blurred.

[0013] In one embodiment of this application, deblurring is performed using a deblurring network, which includes an event encoder, an image encoder, and a decoder. The image encoder includes a dual-path deblurring module, which includes a pointwise deblurring path and a spatial deblurring path.

[0014] In one embodiment of this application, restoring a blurred image to a clear result image based on the motion flow, the event features, and the image features includes: fusing cascaded image features and event features through the point-by-point deblurring path to obtain multi-scale point-by-point bimodal features; and spatially aggregating the image features and the multi-scale point-by-point bimodal features using the motion flow through the spatial deblurring path to restore the blurred image to the result image.

[0015] In one embodiment of this application, the spatial deblurring path includes a motion flow-guided deblurring module, which includes a deformable convolution. Spatial aggregation of the image features and the multi-scale point-by-point bimodal features using the motion flow includes: determining a first spatial offset position of the dynamic kernel of the deformable convolution based on the motion flow; inputting the cascaded image features and the multi-scale point-by-point bimodal features into the deformable convolution to determine a second spatial offset position of the dynamic kernel; offsetting the dynamic kernel based on the first and second spatial offset positions; and sampling and spatially aggregating the image features and the multi-scale point-by-point bimodal features using the offset dynamic kernel.

[0016] In one embodiment of this application, the motion flow includes multiple feature maps, each feature map including the pixels of the blurred image and the displacement between the pixels of the clear image corresponding to the blurred image. Determining the first spatial offset position of the dynamic kernel of the deformation convolution based on the motion flow includes: determining the first spatial offset position based on the pixel displacement of the feature map.

[0017] In one embodiment of this application, the method further includes training the deblurring network, wherein training the deblurring network includes: training the motion flow through a self-supervised re-blurring process of a clear image corresponding to the blurred image from clear to blurred; and training the deblurring network based on the trained motion flow.

[0018] In one embodiment of this application, the blurred image includes at least one of a photograph image, a portion of a photograph image, a video frame image, and a portion of a video frame image.

[0019] This application also provides a deblurring system, the system comprising: a reblurring network configured to generate a motion stream based on an event camera signal corresponding to a blurred image, wherein the motion stream characterizes an explicit relationship between the event camera signal and an action that causes the blurred image to blur; a deblurring network configured to perform feature extraction on the event camera signal and the blurred image to obtain event features and image features; and to restore the blurred image to a clear result image based on the motion stream, the event features, and the image features.

[0020] In one embodiment of this application, the reblurring network is further configured to: divide the exposure time of the blurred image into multiple intervals; segment and preprocess the event camera signal into event voxel grids corresponding to each of the multiple intervals based on the multiple intervals; extract features from the event voxel grids to obtain the motion flow; and train the motion flow, wherein the motion flow includes feature maps corresponding to each of the multiple intervals; and the feature maps include displacements between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0021] In one embodiment of this application, the reblurring network is further configured to: divide the exposure time of the blurred image into multiple intervals; segment and preprocess the event camera signal into event voxel grids corresponding to each of the multiple intervals based on the multiple intervals; extract features from the blurred image and the event voxel grids to obtain the motion flow; and train the motion flow, wherein the motion flow includes feature maps corresponding to each of the multiple intervals; and the feature maps include displacements between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0022] In one embodiment of this application, the re-blurring network is further configured to: distort the sharp image based on pixel displacements of the plurality of feature maps to generate a plurality of distorted images; generate a constructed blurred image based on the average value of the plurality of distorted images and the motion flow; adjust the motion flow based on the difference between the blurred image and the constructed blurred image; and repeat the above steps until the difference is less than a predetermined value.

[0023] In one embodiment of this application, the re-blurring network is further configured to: optimize the re-blurring network based on a loss function, wherein the loss function includes the L1 norm between the blurred image and the constructed blurred image.

[0024] In one embodiment of this application, the motion stream includes a cross-frame motion stream, and the re-blurring network is further configured to generate the cross-frame motion stream based on event camera signals corresponding to multiple frames of the blurred images and the multiple frames of the blurred images, wherein the cross-frame motion stream represents an explicit relationship between the event camera signals corresponding to the multiple frames of the blurred images and the actions that cause the multiple frames of the blurred images to be blurred.

[0025] In one embodiment of this application, the deblurring network includes an event encoder, an image encoder, and a decoder, wherein the image encoder includes a dual-path deblurring module, which includes a pointwise deblurring path and a spatial deblurring path.

[0026] In one embodiment of this application, the point-by-point deblurring path is configured to fuse the cascaded image features and the event features to obtain multi-scale point-by-point bimodal features; and the spatial deblurring path is configured to use the motion flow to spatially aggregate the image features and the multi-scale point-by-point bimodal features to restore the blurred image to the resulting image.

[0027] In one embodiment of this application, the spatial deblurring path includes a motion flow-guided deblurring module, wherein the motion flow-guided deblurring module includes a deformable convolution, and the spatial deblurring path is further configured to: determine a first spatial offset position of the dynamic kernel of the deformable convolution based on the motion flow; input the cascaded image features and the multi-scale point-by-point bimodal features into the deformable convolution to determine a second spatial offset position of the dynamic kernel; offset the dynamic kernel based on the first spatial offset position and the second spatial offset position; and use the offset dynamic kernel to sample the image features and the multi-scale point-by-point bimodal features and perform spatial aggregation.

[0028] In one embodiment of this application, the motion flow includes multiple feature maps, each feature map including pixels of a clear image corresponding to the blurred image and displacements between corresponding pixels of the blurred image, and the spatial deblurring path is further configured to: determine the first spatial offset position based on the pixel displacements of the feature maps.

[0029] In one embodiment of this application, the reblurring network is further configured to train the motion stream through a self-supervised reblurring process from sharp to blurry with a sharp image corresponding to the blurred image; and the deblurring network is further configured to be trained based on the trained motion stream.

[0030] In one embodiment of this application, the blurred image includes at least one of a photograph image, a portion of a photograph image, a video frame image, and a portion of a video frame image.

[0031] Another aspect of this application provides an electronic device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the deblurring methods provided in this application.

[0032] In another aspect, this application provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform any of the deblurring methods provided in this application.

[0033] According to the deblurring method, system, electronic device, and computer-readable storage medium provided by the exemplary embodiments of this application, by modeling the explicit relationship between the event camera signal of the blurred image and the action that causes the blurring, valuable guidance can be provided for deblurring processing, and based on this, clearer result images can be obtained, and the deblurring method and system can be applied to more extreme blurred scenarios. Attached Figure Description

[0034] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. Wherein:

[0035] Figure 1 This is a flowchart of a deblurring method according to one embodiment of this application;

[0036] Figure 2 This is a process diagram of a deblurring method according to one embodiment of this application;

[0037] Figure 3A The qualitative comparison results of deblurring methods based on the GoPro dataset are shown;

[0038] Figure 3B The qualitative comparison results of deblurring methods based on the GoPro dataset are shown;

[0039] Figure 4 The qualitative comparison results of deblurring methods based on the REBlur dataset are shown;

[0040] Figure 5 This is a deblurring system according to one embodiment of the present application; and

[0041] Figure 6 This is a schematic diagram of an electronic device according to one embodiment of the present application. DETAILED DESCRIPTION

[0042] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this application and are not intended to limit the scope of this application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0043] It should be noted that in this specification, the terms "first," "second," "third," etc., are used only to distinguish one feature from another and do not imply any limitation on the features, especially not any order of precedence.

[0044] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or steps, but not excluding the presence of one or more other features, elements, steps, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to examples or illustrations.

[0045] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or overly formalized meaning.

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. Furthermore, unless explicitly limited or contradicted by the context, the specific steps included in the methods described in this application are not limited to the order in which they are described, but can be performed in any order or in parallel. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] Figure 1 This is a flowchart of a deblurring method 1000 according to one embodiment of this application. For example... Figure 1 As shown, this application provides a deblurring method 1000 comprising:

[0048] S1, generate a motion stream based on the event camera signal corresponding to the blurred image, wherein the motion stream represents the explicit relationship between the event camera signal and the action that causes the blurred image to become blurred.

[0049] S2 extracts features from the event camera signal and the blurred image to obtain event features and image features.

[0050] S3, based on motion flow, event features, and image features, restores the blurred image to a clear result image.

[0051] The following will combine Figures 2 to 4 The above deblurring method 1000 is explained in detail.

[0052] Images can be acquired using standard cameras or event cameras. Standard cameras (frame-based cameras) can be understood as frame-based vision sensors, which suffer from problems such as undersampling, data redundancy, and data processing latency. For example, standard cameras typically sample and quantize the scene at a set frame rate to acquire visual information, resulting in undersampling between image frames. For machine vision systems requiring real-time processing, undersampling can lead to failures. Furthermore, each image frame of a standard camera records the intensity information of all pixels, regardless of whether this information has changed relative to the previous moment. Therefore, the acquired image data is highly redundant, and acquiring and processing such a large amount of data results in high channel bandwidth requirements, high transmission power consumption, and high computer memory usage. Additionally, standard cameras suffer from data processing latency, which inevitably limits the control frequency of high-speed moving robot systems. Although high-performance standard cameras designed specifically for high-speed, high dynamic range applications exist, these cameras are typically very expensive, bulky, and require additional power supplies, hindering their application in practical systems.

[0053] In comparison, the processing mechanism of biological visual systems is completely different from that of standard cameras. In nature, biological visual systems, including those of humans, employ asynchronous event-driven processing and process information hierarchically and in parallel within large-scale neural networks.

[0054] An event camera is a novel type of bio-inspired visual sensor, and therefore its processing mechanism is completely different from that of a standard camera. In other words, an event camera can capture scene changes efficiently in real time. Unlike frame-based standard cameras, event cameras only report triggered pixel-level light intensity changes and can output asynchronous event streams with microsecond-level resolution. Typical event cameras include, for example, DVS (Dynamic Vision Sensor), ATIS (Asynchronous Time based Image Sensor), and DAVIS (Dynamic and Active Pixel Vision Sensor), etc. Instead of outputting intensity image frames at a constant rate, event cameras only output information related to local pixel-level light intensity changes. When these pixel-level light intensity changes exceed a user-defined threshold, the event camera can timestamp them with microsecond-level resolution and output an asynchronous event stream.

[0055] Both standard and event cameras can be affected by motion blur caused by factors such as camera shake or object movement during exposure. However, as a bio-inspired asynchronous sensor, event cameras record light intensity changes for each pixel at high temporal resolution and show significant advantages in enhancing deblurring.

[0056] However, most traditional deblurring methods focus on designing fusion strategies for events and images, lacking in-depth research into modeling the explicit relationship between event camera signals and the actions causing blurring. In the exemplary embodiments of this application, characterizing the explicit relationship between event camera signals and the actions causing blurring of the image through motion flow provides valuable guidance for deblurring processing, resulting in clearer image outcomes and enabling the deblurring methods and systems to be applied to more extreme blurry scenarios.

[0057] Step S1

[0058] Figure 2 This is a process diagram of a deblurring method 1000 according to one embodiment of this application.

[0059] like Figure 1 and Figure 2As shown, as an alternative, step S1 generates a motion stream based on the event camera signal corresponding to the blurred image, wherein the motion stream characterizes the explicit relationship between the event camera signal and the action that causes the blurred image to blur, and may include, for example,: dividing the exposure time of the blurred image into multiple intervals; segmenting and preprocessing the event camera signal into event voxel grids corresponding to each of the multiple intervals based on the multiple intervals; performing feature extraction on the event voxel grids to obtain the motion stream; and training the motion stream, wherein the motion stream includes feature maps corresponding to each of the multiple intervals; and the feature maps include displacements between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0060] Alternatively, motion flow can be understood as a collection of multiple motion matrices, each of which can characterize an explicit relationship between an event camera signal and the action that causes the blurred image to become blurred.

[0061] Continue to refer Figure 1 and Figure 2 Alternatively, step S1 generates a motion stream based on the event camera signal corresponding to the blurred image, wherein the motion stream characterizes the explicit relationship between the event camera signal and the action that causes the blurred image to blur, and may include, for example,: dividing the exposure time of the blurred image into multiple intervals; segmenting and preprocessing the event camera signal into an event voxel grid corresponding to each of the multiple intervals based on the multiple intervals; performing feature extraction on the blurred image and the event voxel grid to obtain the motion stream; and training the motion stream, wherein the motion stream includes a feature map corresponding to each of the multiple intervals; and the feature map includes displacements between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0062] Specifically, the event camera can capture only asynchronous logarithmic changes in light intensity. For example, at an exposure time [0, T], for each pixel, when the logarithmic change in light intensity reaches a user-defined light intensity threshold c, the event camera can output an asynchronous event. That is, when the logarithmic change in light intensity of a pixel reaches the user-defined light intensity threshold c, the event camera emits an event camera signal e = (x, y, t, p). The event camera signal e can be understood as the format of the event, which includes the pixel coordinates (x, y), the timestamp t ∈ [0, T] when the logarithmic polarity of the light intensity changes, and the polarity of the logarithmic change in light intensity p ∈ {-1, +1}.

[0063] Alternatively, the polarity change p of the logarithm of light intensity can be expressed as formula (1):

[0064]

[0065] Where c is the light intensity threshold; (x,y) are the pixel coordinates, I tLet be the light intensity at pixel coordinates (x, y); and t be the timestamp when the light intensity changes logarithmically.

[0066] As an option, the threshold c can be selected based on the type of event camera.

[0067] At an exposure time [0, T], the potentially sharp image at timestamp t can be labeled L(t), and the light intensity of the sharp image Y is assumed to be L(T / 2), where the sharp image Y can be understood as the ground truth. In deep learning, the ground truth (GT) can be understood as a distortion-free image, representing the accuracy of the training set in classifying supervised learning techniques. The ground truth is used in statistical models to prove or disprove research hypotheses, and its determination can be understood as testing the collection of appropriate target data.

[0068] Alternatively, the corresponding blurred image X can be obtained by formula (2):

[0069]

[0070] Among them, c r is the time-varying threshold; [0, T] is the exposure time; L(t) is the potential sharp image at timestamp t; L(T / 2) is the light intensity of the sharp image Y; and p(s) is the polarity change of the logarithm of the light intensity.

[0071] As an option, the time-varying threshold c r It can approximately follow a normal distribution.

[0072] Furthermore, asynchronous events triggered within the exposure time [0, T] can be represented as Among them, N e This represents the number of asynchronous events; e is the event camera signal; and l is the asynchronous event number.

[0073] In one embodiment of this application, the exposure time [0, T] can be divided into multiple intervals. For example, the exposure time [0, T] is divided into 2N intervals on average, and accordingly, based on the multiple intervals, the event camera signal can be segmented and preprocessed into an event voxel grid corresponding to each of the multiple intervals.

[0074] For blurred image restoration, most deblurring methods use 2D event images or 3D event voxel grids (e.g., stacked by time or event number) as event representations. The following section will use an event voxel grid as an example to describe the process of obtaining motion flow.

[0075] However, those skilled in the art should understand that in the deblurring method proposed in the exemplary embodiments of this application, the event representation may be a 2D event image or a 3D event voxel grid, and this application does not limit it in this regard.

[0076] For example, event camera signals can also be divided into 2N groups. By accumulating the polarity of events in each group, a 3D spatiotemporal histogram of the events can be constructed. This 3D spatiotemporal histogram, also referred to as an event voxel grid below, can be used... express.

[0077] In one embodiment of this application, the above-described re-blurring process is performed using a re-blurring network Gr. Optionally, feature extraction can be performed on the event voxel grid to obtain the motion flow; alternatively, feature extraction can also be performed on both the blurred image and the event voxel grid to obtain the motion flow. Taking feature extraction on both the blurred image and the event voxel grid to obtain the motion flow as an example, obtaining the motion flow may include: using the blurred image X and the event voxel grid E corresponding to the blurred image X as input to the re-blurring network Gr. Optionally, the re-blurring network Gr performs the main computation at a downsampled resolution to improve efficiency. For example, two consecutive strided convolutions are applied to extract the reduced spatial resolution features H / 4×W / 4, and the downsampled features are used to predict the downsampled motion flow graph.

[0078] Alternatively, motion flow can be trained based on a dataset of sharp images within the exposure time corresponding to the blurred image.

[0079] As an alternative, the refuting network Gr can also use, for example, bilinear upsampling to transform the predicted downsampled motion graph. Zoom in to obtain the predicted motion flow at the original resolution. Furthermore, in generating predicted motion flow Subsequently, the motion flow can be trained through a self-supervised process of blurring the sharp image corresponding to the blurred image and then blurring it again. The motion flow includes a feature map corresponding to each of multiple intervals; and the feature map includes the displacement between the pixels of the sharp image and the corresponding pixels of the blurred image.

[0080] Specifically, as can be seen from formula (2), the event camera signal cannot directly capture spatial motion information, but it can encode the change in light intensity of each pixel over time. Therefore, in one embodiment of this application, the motion of pixels causes the image to blur during exposure. In order to establish the relationship between the event camera signal and the motion of each pixel, the motion flow can be trained through a self-supervised process of blurring from sharp to blurred.

[0081] Optionally, the exposure time [0, T] is divided into 2N intervals, and the event camera signal is divided into 2N groups, then the motion flow... It includes 2N feature maps F[k] corresponding to 2N time intervals. Feature maps F[k] may include pixel-level 2D displacements of multiple pixels in the sharp image Y.

[0082] As an option, a blurred image is constructed. It can be constructed by distorting the sharp image Y. For example, according to formula (3), a blurred image can be constructed based on the feature map F[k] and the average value of the image obtained after distorting the sharp image Y.

[0083]

[0084] Where 2N represents the evenly divided interval of exposure time [0, T]; F[k] is the feature map corresponding to one of the 2N time intervals; Y is the sharp image; and The blurred image is constructed based on the sharp image Y.

[0085] Optionally, the re-blurring network Gr takes the blurred image X and its corresponding event voxel grid E as input and performs re-blurring. The predicted motion flow can be obtained through formula (4).

[0086]

[0087] in, X represents the predicted motion flow; X is the blurred image; and E is the event voxel grid corresponding to the blurred image X.

[0088] Furthermore, by learning the process of blurring from sharp to blurry again through self-supervised learning, the training of motion flow can be carried out without manually prepared priors or additional labeled optical flow datasets.

[0089] In other words, in the exemplary embodiments of this application, by modeling the image blurring process, a synthetic image (constructed blurred image) that more closely resembles a naturally blurred image can be constructed by learning the image blurring process itself, even when training data is scarce. Optionally, by using the synthetic blurred image and the corresponding clear image as part of the training data, a trained motion flow can be obtained through self-supervised learning.

[0090] In one embodiment of this application, training the motion flow through a self-supervised process of blurring the clear image corresponding to the blurred image from clear to blurred and then blurring again includes: distorting the clear image based on the pixel displacement of multiple feature maps to generate multiple distorted images; generating a constructed blurred image based on the average value of the multiple distorted images and the motion flow; adjusting the motion flow based on the difference between the blurred image and the constructed blurred image; and repeating the above steps until the difference is less than a predetermined value.

[0091] Specifically, in order to supervise the predicted motion flow encoded with motion information Based on formula (3) and 2N feature maps The average value of the image obtained after distorting the sharp image Y forms the constructed blurred image.

[0092] Optionally, training the reblurring network Gr may include: optimizing the reblurring network Gr based on a loss function, wherein the loss function includes the L1 norm between the blurred image and the constructed blurred image.

[0093] Based on the blurred image X and the constructed blurred image Supervised refuting network Gr, where the loss L of refuting network Gr is... reblur This can be expressed by formula (5):

[0094]

[0095] Among them, L reblur The loss of the re-blurring network Gr is X; X is the blurred image. is the constructed blurred image; 2N is the evenly divided interval of the exposure time [0, T]; Y is a single predicted feature map corresponding to one of 2N time intervals; and Y is the sharp image corresponding to the blurred image X.

[0096] By employing a process of blurring from sharp to blurred and then back to blurred, a network can be trained to infer, in an explicit form of motion flow with pixel-by-pixel motion offset, the motion that causes the blur during exposure.

[0097] Furthermore, in one embodiment of this application, the re-blurring network Gr may employ, for example, a multi-scale U-Net structure to predict motion flow using a U-Net-like network. The re-blurring network Gr predicts motion flow using either a blurred image X and the event voxel grid E corresponding to the blurred image X as input, or only the event voxel grid E corresponding to the blurred image X as input.

[0098] Alternatively, assembling motion flow graphs can enable modeling of complex or nonlinear motion. This motion information can serve as valuable priors, guiding deblurring networks in effectively aggregating blurred image features and resolving various blurring problems.

[0099] Additionally, motion streams include cross-frame motion streams. Generating motion streams based on event camera signals corresponding to blurred images and blurred images may include: generating cross-frame motion streams based on event camera signals corresponding to multiple blurred images and multiple blurred images, wherein the cross-frame motion streams characterize the explicit relationship between event camera signals corresponding to multiple blurred images and actions that cause the multiple blurred images to blur.

[0100] In other words, the motion stream, including the cross-frame motion stream, can be extended to capture temporal motion across multiple frames from the corresponding event camera signals. In one embodiment of this application, the cross-frame motion stream can be trained by re-blurring multiple blurred frames. The deblurring method 1000 proposed in the exemplary embodiments of this application is thus extended to effectively aggregate temporal features across frames to improve the processing effect of event-based video deblurring.

[0101] Step S2

[0102] Reference again Figure 1 and Figure 2 In one embodiment of this application, the steps of obtaining event features and image features, and the steps of restoring a blurred image to a clear result image, can be performed using a deblurring network Gd, wherein the deblurring network Gd may include an event encoder V, an image encoder U, and a decoder W. The image encoder U may include a dual-path deblurring module, which includes a pointwise deblurring path and a spatial deblurring path.

[0103] Figure 3A and Figure 3B Qualitative comparison results of deblurring methods based on the GoPro dataset are shown.

[0104] Taking the existing deblurring method EFNet as an example, it utilizes a channel-level attention mechanism to fuse image features and event features. However, from Figure 3A and Figure 3B As can be seen, EFNet performs worse in some obviously blurred areas compared to single-image deblurring methods that do not use event camera signals. This may be because channel-level attention lacks direct interaction between spatial labels and fails to properly encode long-range context.

[0105] Therefore, to solve the above problems, the exemplary embodiments of this application introduce a dual-path deblurring module. Optionally, the dual-path deblurring module may include a point-by-point deblurring path and a spatial deblurring path.

[0106] In one embodiment of this application, step S2, which extracts features from the event camera signal and the blurred image to obtain event features and image features, may include: fusing cascaded image features and event features through a point-by-point deblurring path to obtain multi-scale point-by-point bimodal features; and spatially aggregating the image features and multi-scale point-by-point bimodal features using motion flow through a spatial deblurring path to restore the blurred image to the resulting image.

[0107] like Figure 2As shown, optionally, the point-by-point deblurring path utilizes point-by-point two-layer MLPs (Multilayer Perceptrons) to fuse cascaded image features h. i and event characteristics v i To obtain multi-scale point-by-point bimodal features H i For example, the three-scale point-by-point bimodal feature H i .

[0108] Optionally, the spatial deblurring path can employ a motion flow guided deblurring module (BGDM) to deblurr image features h. i and multi-scale point-by-point bimodal features H i Spatially aggregating is performed for spatial deblurring. The results from the two paths described above can be fused to generate the final features. The event encoder V generates multi-scale event feature maps from the event voxel grid E corresponding to the blurred image X.

[0109] Optionally, "SConv" may include a convolution with a stride of 2; and "Tconv" may include a transposed convolution with a stride of 2.

[0110] Furthermore, in order to utilize the aforementioned motion flow more effectively in the deblurring process, a novel motion flow guided deblurring module (BGDM) is introduced in the exemplary embodiments of this application to effectively achieve spatial adaptive deblurring.

[0111] Specifically, in one embodiment of this application, the spatial deblurring path may include a motion flow-guided deblurring module, which may include a deformable convolution. Spatial aggregation of image features and multi-scale point-by-point bimodal features using motion flow may include: determining a first spatial offset position of the dynamic kernel of the deformable convolution based on the motion flow; inputting the cascaded image features and multi-scale point-by-point bimodal features into the deformable convolution to determine a second spatial offset position of the dynamic kernel; offsetting the dynamic kernel based on the first and second spatial offset positions; and sampling and spatially aggregating the image features and multi-scale point-by-point bimodal features using the offset dynamic kernel.

[0112] Furthermore, in one embodiment of this application, the motion flow includes multiple feature maps, each feature map including the displacement between pixels of the clear image corresponding to the blurred image and the corresponding pixels of the blurred image. Determining the first spatial offset position of the dynamic kernel of the deformation convolution based on the motion flow may include: determining the first spatial offset position based on the pixel displacement of the feature map.

[0113] Specifically, for example, 2N deformable convolutional kernels grouped into 3×3 groups can be used, where the cascaded image features h i and multi-scale point-by-point bimodal features H i As input to the dynamic kernel.

[0114] For each pixel used for convolution, the motion flow graph The feature map of the k-th prediction The k-th dynamic kernel can be guided, moving it based on a 2D flow offset space. In other words, the 2N dynamic kernels are based on 2N predicted motion flow graphs at all pixels. A first spatial offset (which can be understood as the predicted displacement of the kinetic flow) is performed. Additionally, a second spatial offset (which can be understood as the predicted kernel weight displacement) is performed. Prediction can be made from input features, for example, using four pointwise convolutional blocks. Here, the "18" at the second spatial offset position O represents the bidirectional displacement of the weights of the 3×3 kernel in each kernel. Therefore, the motion-flow-guided dynamic convolution can be formulated as DCN(Concat(h)). i H i ); ), where Concat is the operation to increase the number of channels; The sum of the predicted motion flow and the predicted kernel weight displacement; and w is the weighting coefficient. Optionally, the weighting coefficient w can be adaptively predicted. The final deblurred features, after being mixed with the output of the point-by-point deblurred path, can be fed into the next module for further processing.

[0115] Optionally, in one embodiment of this application, a motion flow can be trained by using a reblurring network Gr, which utilizes a self-supervised reblurring process of a clear image corresponding to a blurred image from clear to blurred; and a deblurring network Gd is trained based on the trained motion flow.

[0116] The decoder W of the deblurring network Gd may include convolutional layers to output the final result: a clear image after deblurring. The deblurring network Gd can be trained end-to-end. Optionally, the loss L of the reblurring network Gr can be adjusted. reblur It can be obtained through formula (5), while the total loss L can be obtained through formula (6):

[0117]

[0118] Where X is a blurred image; Let X be a blurred image constructed based on the blurred image; Y be a sharp image; and This is the clear image after deblurring.

[0119] Step S3

[0120] Reference again Figure 1 and Figure 2 Step S3, which restores the blurred image to a clear result image based on motion flow, event features, and image features, may include, for example, fusing the image features and event features of the cascaded image to be deblurred through a point-by-point deblurring path to obtain multi-scale point-by-point bimodal features; and spatially aggregating the image features and multi-scale point-by-point bimodal features of the image to be deblurred using a spatial deblurring path to restore the image to be deblurred to the result image.

[0121] Furthermore, according to exemplary embodiments of this application, the deblurring method provided by this application can deblur images, videos, etc. In other words, the blurred image X may include at least one of a photograph image, a portion of a photograph image, a video frame image, and a portion of a video frame image.

[0122] Therefore, according to an exemplary embodiment of this application, before restoring the blurred image to a clear result image based on motion flow, event features and image features in step S3, the deblurring method 1000 further includes training the deblurring system using standard video datasets and image datasets.

[0123] Optionally, the deblurring system can be trained using the GoPro dataset and fine-tuned based on the REBlur dataset. Furthermore, in one embodiment of this application, training and testing can also be performed using the UEVD dataset.

[0124] Specifically, the GoPro dataset consists of a training set of 2103 pairs of blurry and sharp images (resolution 1280×720) and a test set of 1111 pairs of blurry and sharp images.

[0125] The REBlur dataset includes a training set consisting of 486 pairs of blurred and sharp images and their corresponding events, and a test set consisting of 983 pairs of blurred and sharp images and their corresponding events.

[0126] The UEVD dataset is an event-based video dataset with unknown exposures, collected by a DAVIS 346 color event camera. The UEVD data includes 53,601 sharp images and corresponding event camera signals for 59 different scenes. The number of frames in the exposure phase is denoted as m, and the number of frames in the readout phase is denoted as n. Different exposure-blurred frames can be synthesized by averaging m different adjacent sharp frames. Therefore, the UEVD dataset includes 43 scenes for training and 16 scenes for testing. In the test set, m+n=14 and m={9, 11, 13} can achieve different exposure-blurred frames.

[0127] Optionally, the exposure time [0, T] can be divided into 2N intervals, with N set to 8, and the event voxel grid can be used. As input to the deblurring model. For event-based deblurring on the GoPro dataset, for example, Adam (Adaptive Momentum, a stochastic optimization method for adaptive momentum) can be used, with an initial learning rate set to 2×10. -4 The cosine learning rate strategy was set to a minimum learning rate of 10. -7 Random noise in the event voxel grid can be augmented by horizontal and vertical flipping. Optionally, the deblurring system was trained on eight NVIDIA V100 GPUs (General Processing Units) with a batch size of 16 for 200k iterations.

[0128] After training the deblurring model on the GoPro dataset, it can be fine-tuned based on the REBlur dataset. For example, the deblurring model can be fine-tuned for 600 iterations on a single V100 GPU with a batch size of 4 and an initial learning rate of 2×10⁻⁶. -5 Furthermore, the other configurations of the deblurring system are the same as when it was trained on the GoPro dataset.

[0129] Additionally, for the UEVD dataset, the deblurring system can use an initial learning rate of 10. -4 The Adam algorithm and a multi-step scheduler are used. The training process can include 21 epochs with a batch size of 2. For video deblurring, two adjacent blurred frames and their corresponding event camera signals are used as input to the deblurring model to explore temporal information.

[0130] Table 1 shows a comparison of deblurring methods based on the GoPro dataset. An open-source event camera simulator was used to simulate event camera signals for the GoPro dataset. Optionally, the contrast intensity threshold c can be based on a Gaussian distribution. (0.2, 9×10)-4 To achieve more realistic results.

[0131] Table 1

[0132]

[0133] As shown in Table 1, deblurring methods can include single-image deblurring methods and event-based deblurring methods. Single-image deblurring methods include Nah et al. (the image deblurring method proposed by Nah et al.), MIMO-UNet (Multi-input multi-output U-network), MPRNet (Multi-Stage Progressive Image Restoration Network), DMPHN (Deep Stacked Hierarchical Multi-patch Network), SRN (Scale-recurrent Network), HINet (Half Instance Normalization Network), Restormer, MSDI, and NAFNet (Nonlinear Activation Free Network). Event-based deblurring methods include LMED, BHA, and D... 2 Nets, EFNet, SRN+ (Scale-recurrent Network+), HINnet+ (Half-Instance Normalization Network+), and ours (the deblurring method of this application). Among them, SRN+ and HINnet+ are non-event-based versions developed by Sun et al., which use the same representation as EFnet. Furthermore, The event representation has been changed from Symmetric Cumulative Event Representations (SCER) to the event voxel grid E of this application. This indicates that the event voxel grid of this application was used. "ours" indicates that the deblurring method of this application was used.

[0134] Furthermore, the above methods are all compared using peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) scores. Compared to conventional methods, the deblurring method provided in the exemplary embodiments of this application improves PSNR by 2.94 dB compared to the best-performing single-image deblurring method HINet+, and by 1.17 dB compared to the best-performing event-based deblurring method EFNet. Moreover, after using the same event voxel grid E, the deblurring method provided in at least one embodiment of this application... In comparison, the PSNR increased by 0.73 dB, thus proving the effectiveness of the deblurring method provided in this application.

[0135] like Figure 3A and Figure 3B As shown, a comparison reveals that the deblurring method provided in the exemplary embodiment of this application restores image details more clearly compared to other deblurring methods. Specifically, given the input as frame 133 of blurred video 410 and frame 216 of blurred video 881, the single-image deblurring method cannot restore the details of a fast-moving car. However, by utilizing event camera signals, EFNet and In the restored clear image, such as a moving car, it achieved good deblurring results; however, EFNet and Blurry lines in the input blurry image could not be recovered. By comparison, the clear image recovered based on the deblurring method provided in the exemplary embodiment of this application clearly distinguishes the moving car and lines.

[0136] Table 2 shows a comparison of deblurring methods based on the REBlur dataset.

[0137] Table 2

[0138]

[0139] As shown in Table 2, deblurring methods may include: SRN (Scale-recurrent Network), NAFNet (Nonlinear Activation-Free Network), Restormer, HINet (Half Instance Normalization Network), BHA, EFNet And ours. Among them, This indicates that the event voxel grid of this application was used. "ours" indicates that the deblurring method of this application was used, as an event representation. The first four methods shown in Table 2 (SRN, NAFNet, Restormer, and HINet) are single-image deblurring methods that do not utilize event camera signals, while the other methods (BHA, SRN+, HINNet+, EFNet, EFNet+, and ours) are event-based deblurring methods. All of the above methods are compared using the scores of the parameters PSNR and SSIM.

[0140] Specifically, based on the REBlur dataset, the deblurring method provided by the exemplary embodiments of this application is compared with multiple single-image deblurring methods and multiple event-based deblurring methods. As shown in Table 2, the deblurring method provided by at least one embodiment of this application improves the PSNR by 0.64 dB compared with the best-performing event-based deblurring method EFNet.

[0141] Figure 4 The qualitative comparison results of deblurring methods based on the REBlur dataset are shown.

[0142] In addition, if Figure 4 As shown, the deblurring method provided in the exemplary embodiments of this application restores clearer image details compared to other deblurring methods, where the input is a blurred image. For example, compared to EFNet, the clear image restored by the deblurring method provided in the exemplary embodiments of this application clearly distinguishes moving checkboxes, text, and circles, achieving a better deblurring effect.

[0143] Table 3

[0144]

[0145] Table 3 compares deblurring methods for real-world event-based video datasets. As shown in Table 3, deblurring methods include: Nah et al. (the image deblurring method proposed by Nah et al.), DMPHN (Deep Stacked Hierarchical Multi-patch Network), CDVD-TSP (Cascaded Deep Video Deblurring-using Temporal SharpnessPrior), MPRNet (Multi-Stage Progressive Image Restoration Net), MIMO-UNet (Multi-input multi-output U-net), LEDVDI, UEVD, and D... 2 Nets. Among them, This indicates that the event voxel grid of this application was used. As an event representation; "ours" indicates that the deblurring method of this application was used; This indicates that the deblurring method of this application was used, and cross-frame motion streams were employed to deblur event-based videos; This indicates that the deblurring method of this application was used, and cross-frame motion flow and BGDM were not employed. All the above methods were compared using the scores of the parameters PSNR and SSIM.

[0146] The deblurring method provided in the exemplary embodiments of this application achieves optimal performance compared to other deblurring methods at different exposure times. For example, the deblurring method provided in the exemplary embodiments of this application shows a significant difference of 5.16 dB in average PSNR (Avg.) compared to single-image deblurring methods. Furthermore, it also outperforms the best-performing event-based deblurring method. Compared to the previous version, PSNR improved by 0.59 dB. Additionally, without cross-frame motion stream and BGDM... This application demonstrates the effectiveness of the deblurring method provided in at least one embodiment. Results show that, without employing cross-frame motion flow, the PSNR of the deblurring method provided in the exemplary embodiment of this application decreases by 0.63 dB.

[0147] Table 4

[0148]

[0149] Table 4 shows the decomposition ablation based on the GoPro dataset.

[0150] As shown in Table 4, by changing multiple conditions or parameters in ablation experiments, the effectiveness of the deblurring method provided in at least one embodiment of this application can be determined, and it can be used to evaluate the key components in the deblurring method and model architecture provided in this application. Table 4 shows the comparison of multiple deblurring methods when trained using the same training settings, wherein the ablation experiments changing multiple conditions / parameters may include: removing the bi-modal input of the point-by-point path of the BGDM (bi-modal in BGDM, denoted as "Ours-1"); removing the bi-modal input of the point-by-point path of the BGDM and the motion flow guidance (Guided in BGDM, denoted as "Ours-2").

[0151] Furthermore, as an alternative, ablation experiments with varying conditions / parameters may include alternatives to dual-path deblurring. Dual-path deblurring is compared to possible alternatives, such as sequential deblurring. Specifically, S+P performs spatial deblurring followed by pointwise deblurring; P+S performs pointwise deblurring followed by spatial deblurring, where these two networks can be represented as "Ours-3" and "Ours-4". A comparison reveals that both of these model architectures exhibit poor deblurring performance.

[0152] Furthermore, the PSNR value of "Ours" differs from "Ours-5" by only 0.60 dB, demonstrating that choosing point-based MLP for dual-mode feature fusion can improve the effectiveness of the deblurring method.

[0153] In addition, "Ours-6" uses cross-model self-attention (Attn), while "Ours-5" uses point MLP. By comparing "Ours-5" and "Ours-6", the two have similar performance.

[0154] Table 5

[0155]

[0156] Table 5 shows the decomposition and ablation of motion flow training.

[0157] As shown in Table 5, ablation experiments that modify multiple conditions or parameters may include: using implicit representation training instead of motion flow training (denoted as "LBF-5"); or using spatial variation kernels instead of motion flow training (denoted as "LBF-4"). Both of these changes resulted in a 0.50 dB decrease in PSNR.

[0158] Furthermore, ablation experiments with varying conditions or parameters included: learning motion flow using only a single image (Img) (denoted as "LBF-3"); and removing the reblur loss (denoted as "LBF-2"). Both of these changes resulted in a decrease in PSNR, by 0.28 dB and 0.30 dB, respectively. Additionally, ablation experiments with varying conditions / parameters included removing downsampling and upsampling in the reblurring network Gr (Down, denoted as "LBF-1"), which resulted in a 0.22 dB decrease in PSNR.

[0159] Table 6

[0160]

[0161] Table 6 shows a comparison of the computational complexity of event-based deblurring methods.

[0162] Furthermore, as shown in Table 6, this application also tested the deblurring method provided according to exemplary embodiments of this application. Compared with other deblurring methods (EFNet, The computational cost of ).

[0163] For example, taking a blurred image of size 3×256×256 and its corresponding event camera signal as an example, for event-based deblurring, the deblurring method provided according to the exemplary embodiments of this application... It has similar computational cost to EFNet; for event-based video deblurring, the deblurring method provided according to the exemplary embodiments of this application... Compared to UEVD, it costs approximately 65% ​​of the computational cost of UEVD.

[0164] Therefore, the deblurring method provided according to the exemplary embodiments of this application can provide valuable guidance for deblurring processing by modeling the explicit relationship between the event camera signal of the blurred image and the action that causes the blurring, thereby obtaining a clearer result image and making the deblurring method and system applicable to more extreme blurred scenarios.

[0165] Figure 5 This is a deblurring system 2000 according to one embodiment of this application.

[0166] Specifically, with Figure 5For example, the deblurring system 2000 may include a reblurring network 100 and a deblurring network 200. The reblurring network 100 is configured to generate a motion stream based on an event camera signal corresponding to the blurred image, wherein the motion stream characterizes the explicit relationship between the event camera signal and the action that causes the blurred image to be blurred. The deblurring network 200 is configured to perform feature extraction on the event camera signal and the blurred image to obtain event features and image features; and to restore the blurred image to a sharp result image based on the motion stream, event features, and image features.

[0167] Optionally, in one embodiment of this application, the re-blurring network 100 is further configured to: divide the exposure time of the blurred image into multiple intervals; segment and preprocess the event camera signal into event voxel grids corresponding to each of the multiple intervals based on the multiple intervals; extract features from the event voxel grids to obtain a motion flow; and train the motion flow. The motion flow includes a feature map corresponding to each of the multiple intervals; and the feature map includes the displacement between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0168] Optionally, in one embodiment of this application, the re-blurring network 100 is further configured to: divide the exposure time of the blurred image into multiple intervals; segment and preprocess the event camera signal into event voxel grids corresponding to each of the multiple intervals based on the multiple intervals; extract features from the blurred image and the event voxel grids to obtain a motion flow; and train the motion flow through a self-supervised re-blurring process from sharp to blurred in the sharp image corresponding to the blurred image. The motion flow includes a feature map corresponding to each of the multiple intervals; and the feature map includes the displacement between pixels of the blurred image and pixels of the sharp image corresponding to the blurred image.

[0169] Alternatively, in one embodiment of this application, the reblurring network 100 is further configured to: train a motion flow through a self-supervised reblurring process of a sharp image from sharp to blurry; or train a motion flow based on a dataset of sharp images within the exposure time corresponding to the blurred image.

[0170] Furthermore, the re-blurring network 100 is further configured to: distort a sharp image based on pixel displacements of multiple feature maps to generate multiple distorted images; generate a constructed blurred image based on the average value of the multiple distorted images and the motion flow; adjust the motion flow based on the difference between the blurred image and the constructed blurred image; and repeat the above steps until the difference is less than a predetermined value.

[0171] Alternatively, the re-blurring network 100 is further configured to optimize the re-blurring network based on a loss function, wherein the loss function includes the L1 norm between the blurred image and the constructed blurred image.

[0172] Furthermore, the motion stream may include a cross-frame motion stream, and the re-blurring network 100 is further configured to generate a cross-frame motion stream based on event camera signals corresponding to the multi-frame blurred images and the multi-frame blurred images, wherein the cross-frame motion stream represents an explicit relationship between the event camera signals corresponding to the multi-frame blurred images and the actions that cause the multi-frame blurred images to blur.

[0173] Alternatively, the deblurring network 200 may include an event encoder 210, an image encoder 220, and a decoder 230, wherein the image encoder 220 may include a dual-path deblurring module 221, which may include a spatial deblurring path 221-1 and a pointwise deblurring path 221-2.

[0174] In one embodiment of this application, the pointwise deblurring path 221-2 is configured to fuse cascaded image features and event features to obtain multi-scale pointwise bimodal features; and the spatial deblurring path 221-1 is configured to spatially aggregate image features and multi-scale pointwise bimodal features using motion flow to restore the blurred image to the resulting image.

[0175] Furthermore, the spatial deblurring path 221-1 includes a motion flow-guided deblurring module, wherein the motion flow-guided deblurring module includes a deformable convolution, and the spatial deblurring path 221-1 is further configured to: determine a first spatial offset position of the dynamic kernel of the deformable convolution based on the motion flow; input cascaded image features and multi-scale point-by-point bimodal features into the deformable convolution to determine a second spatial offset position of the dynamic kernel; offset the dynamic kernel based on the first spatial offset position and the second spatial offset position; and use the offset dynamic kernel to sample and spatially aggregate the image features and multi-scale point-by-point bimodal features.

[0176] Optionally, the motion flow includes multiple feature maps, each including pixels of the blurred image and displacements between pixels of the clear image corresponding to the blurred image. The spatial deblurring path 221-1 is further configured to determine a first spatial offset position based on the pixel displacements of the feature maps.

[0177] Furthermore, in one embodiment of this application, the reblurring network 100 is further configured to train a motion flow through a self-supervised reblurring process from sharp to blurry images corresponding to the blurred images; and the deblurring network 200 is further configured to be trained based on the trained motion flow.

[0178] In addition, in one embodiment of this application, the blurred image includes at least one of a photograph image, a portion of a photograph image, a video frame image, and a portion of a video frame image.

[0179] The deblurring system provided in this application has the same beneficial effects as the deblurring method provided in this application, and will not be described in detail here.

[0180] This application also provides an electronic device. The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the aforementioned deblurring method.

[0181] In another aspect, this application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described deblurring method.

[0182] Figure 6 This is a schematic diagram of an electronic device 3000 according to one embodiment of this application.

[0183] like Figure 6 As shown, the electronic device 3000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0184] like Figure 6 The illustrated electronic device 3000 includes one or more processors 3100, a memory 3200, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take the 3100 processor as an example.

[0185] The memory 3200 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause at least one processor to perform the deblurring method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the deblurring method provided in this application.

[0186] The memory 3200, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 3100 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 3200, thereby implementing the deblurring method in the above-described method implementation.

[0187] The memory 3200 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device for quality control, etc. Furthermore, the memory 3200 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 3200 may include memory remotely located relative to the processor 3100, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0188] The electronic device 3000 may further include an input device 3300 and an output device 3400. The processor 3100, memory 3200, input device 3300, and output device 3400 may be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0189] Input device 3300 can receive input digital or character information, and generate key signal inputs related to user settings and function control of electronic equipment used for quality control, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 3400 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0190] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0191] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0192] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0193] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0194] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be servers in distributed systems or servers incorporating blockchain technology. Servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0195] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0196] The deblurring system, electronic device, computer-readable storage medium, and computer program product provided in this application have the same beneficial effects as the deblurring method provided in this application, and will not be described in detail here.

[0197] The above description is merely an illustration of the embodiments of this application and the technical principles employed. Those skilled in the art should understand that the scope of protection involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the technical concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A deblurring method, characterized in that: include: generating a motion flow based on an event camera signal corresponding to a blurred image, wherein the motion flow characterizes an explicit relationship between the event camera signal and an action that causes blurring of the blurred image; Performing feature extraction on the event camera signal and the blurred image to obtain event features and image features; as well as The blurred image is restored to a clear result image according to the motion flow, the event features and the image features.

2. The method according to claim 1, wherein: Generating a motion flow based on an event camera signal corresponding to a blurred image includes: Dividing the exposure time of the blurred image into a plurality of intervals; Based on the plurality of intervals, segmenting and preprocessing the event camera signal into an event voxel grid corresponding to each of the plurality of intervals; performing feature extraction on the event voxel grid to obtain the motion flow; and training the motion flow, wherein the motion flow includes a feature map corresponding to each of the plurality of intervals; and The feature map includes displacements between pixels of the blurred image and pixels of a sharp image corresponding to the blurred image.

3. The method according to claim 1, wherein: Generating a motion flow based on an event camera signal corresponding to a blurred image includes: Dividing the exposure time of the blurred image into a plurality of intervals; Based on the plurality of intervals, segmenting and preprocessing the event camera signal into an event voxel grid corresponding to each of the plurality of intervals; performing feature extraction on the blurred image and the event voxel grid to obtain the motion flow; and training the motion flow, wherein the motion flow includes a feature map corresponding to each of the plurality of intervals; and The feature map includes displacements between pixels of the blurred image and pixels of a sharp image corresponding to the blurred image.

4. The method according to claim 2 or 3, wherein: Training the motion flow includes: Training the motion flow by self-supervising the re-blurring process of the clear image from clear to blurred; or The motion flow is trained based on a dataset of sharp images within the exposure time corresponding to the blurred images.

5. The method according to claim 4, wherein: Training the motion flow by self-supervising the re-blurring process of the clear image from clear to blurred includes: Based on the pixel displacement of the plurality of feature maps, the clear image is distorted to generate a plurality of distorted images; generating a constructed blurred image based on an average of the plurality of distorted images and the motion flow; adjusting the motion flow based on a difference between the blurred image and the constructed blurred image; and The above steps are repeated until the difference is smaller than a predetermined value.

6. The method according to claim 5, wherein: The re-blurring process is performed using a re-blurring network, and the method further includes training the re-blurring network, wherein training the re-blurring network includes: Based on the loss function, the re-blurring network is optimized, Wherein, the loss function includes the L1 norm between the blurred image and the constructed blurred image.

7. The method according to claim 1, wherein: The motion stream includes a cross-frame motion stream, and generating the motion stream based on an event camera signal corresponding to the blurred image includes: generating the cross-frame motion flow based on event camera signals corresponding to the multiple frames of the blurred images and the multiple frames of the blurred images, The cross-frame motion flow represents an explicit relationship between event camera signals corresponding to multiple frames of the blurred images and actions that cause blurring of the multiple frames of the blurred images.

8. The method according to claim 1, wherein: The method further comprises the steps of obtaining event features and image features by using a deblurring network and restoring the blurred image to a clear result image, wherein the deblurring network comprises: event encoder, image encoder and decoder, The image encoder includes a dual-path deblurring module, and the dual-path deblurring module includes a point-by-point deblurring path and a spatial deblurring path.

9. The method according to claim 8, wherein: Restoring the blurred image to a clear result image according to the motion flow, the event feature and the image feature comprises: fusing the cascaded image features and the event features through the point-by-point deblurring path to obtain multi-scale point-by-point bimodal features; and The image features and the multi-scale point-by-point bimodal features are spatially aggregated by using the motion flow through the spatial deblurring path to restore the blurred image to the result image.

10. The method according to claim 9, wherein: The spatial deblurring path includes a motion flow guided deblurring module, the motion flow guided deblurring module includes a deformable convolution, and spatially aggregating the image features and the multi-scale point-by-point bimodal features using the motion flow includes: Based on the motion flow, determining a first spatial offset position of a dynamic kernel of the deformable convolution; Inputting the cascaded image features and the multi-scale point-by-point bimodal features into the deformable convolution to determine a second spatial offset position of the dynamic kernel; offsetting the dynamic kernel based on the first spatial offset position and the second spatial offset position; and The image features and the multi-scale point-by-point bimodal features are sampled and spatially aggregated using the shifted dynamic kernel.

11. The method according to claim 10, wherein: The motion flow includes a plurality of feature maps, the feature maps include displacements between pixels of the blurred image and pixels of a clear image corresponding to the blurred image, and based on the motion flow, determining a first spatial offset position of the dynamic kernel of the deformable convolution includes: The first spatial offset position is determined based on the pixel displacement of the feature map.

12. The method according to claim 8, wherein: The method further includes training the deblurring network, wherein training the deblurring network includes: training the motion flow by self-supervising a re-blurring process of a sharp image corresponding to the blurred image from sharp to blurred; and The deblurring network is trained based on the trained motion flow.

13. The method according to any one of claims 1 to 12, wherein: The blurred image includes at least one of a photo image, a portion of a photo image, a video frame image, and a portion of a video frame image.

14. A deblurring system, characterized in that: The system comprises: a reblurring network configured to generate a motion flow based on an event camera signal corresponding to a blurred image, wherein the motion flow characterizes an explicit relationship between the event camera signal and an action that causes blurring of the blurred image; A deblurring network is configured to perform feature extraction on the event camera signal and the blurred image to obtain event features and image features; and to restore the blurred image to a clear result image according to the motion flow, the event features and the image features.

15. The system of claim 14, wherein: The re-fuzzification network is further configured as: Dividing the exposure time of the blurred image into a plurality of intervals; Based on the plurality of intervals, segmenting and preprocessing the event camera signal into an event voxel grid corresponding to each of the plurality of intervals; performing feature extraction on the event voxel grid to obtain the motion flow; as well as training the motion flow, wherein the motion flow includes a feature map corresponding to each of the plurality of intervals; and The feature map includes displacements between pixels of the blurred image and pixels of a sharp image corresponding to the blurred image.

16. The system of claim 14, wherein: The re-fuzzification network is further configured as: Dividing the exposure time of the blurred image into a plurality of intervals; Based on the plurality of intervals, segmenting and preprocessing the event camera signal into an event voxel grid corresponding to each of the plurality of intervals; performing feature extraction on the blurred image and the event voxel grid to obtain the motion flow; as well as training the motion flow, wherein the motion flow includes a feature map corresponding to each of the plurality of intervals; and The feature map includes displacements between pixels of the blurred image and pixels of a sharp image corresponding to the blurred image.

17. The system according to claim 15 or 16, wherein: The re-fuzzification network is further configured as: Training the motion flow by self-supervising the re-blurring process of the clear image from clear to blurred; or The motion flow is trained based on a dataset of sharp images within the exposure time corresponding to the blurred images.

18. The system of claim 17, wherein: The re-fuzzification network is further configured as: Based on the pixel displacement of the plurality of feature maps, the clear image is distorted to generate a plurality of distorted images; generating a constructed blurred image based on an average of the plurality of distorted images and the motion flow; adjusting the motion flow based on a difference between the blurred image and the constructed blurred image; and The above steps are repeated until the difference is smaller than a predetermined value.

19. The system of claim 18, wherein: The re-fuzzification network is further configured as: Based on the loss function, the re-blurring network is optimized, Wherein, the loss function includes the L1 norm between the blurred image and the constructed blurred image.

20. The system of claim 14, wherein: The motion flow comprises a cross-frame motion flow, and the re-blurring network is further configured to: generating the cross-frame motion flow based on event camera signals corresponding to the multiple frames of the blurred images and the multiple frames of the blurred images, The cross-frame motion flow represents an explicit relationship between event camera signals corresponding to multiple frames of the blurred images and actions that cause blurring of the multiple frames of the blurred images.

21. The system of claim 14, wherein: The deblurring network comprises: event encoder, image encoder and decoder, The image encoder includes a dual-path deblurring module, and the dual-path deblurring module includes a point-by-point deblurring path and a spatial deblurring path.

22. The system of claim 21, wherein: The point-by-point deblurring path is configured to fuse the cascaded image features and the event features to obtain multi-scale point-by-point bimodal features; and The spatial deblurring path is configured to spatially aggregate the image features and the multi-scale point-by-point bimodal features using the motion flow to restore the blurred image into the result image.

23. The system of claim 22, wherein: The spatial deblurring path includes a motion flow guided deblurring module, wherein the motion flow guided deblurring module includes a deformable convolution, Wherein, the spatial deblurring path is further configured as follows: Based on the motion flow, determining a first spatial offset position of a dynamic kernel of the deformable convolution; Inputting the cascaded image features and the multi-scale point-by-point bimodal features into the deformable convolution to determine a second spatial offset position of the dynamic kernel; offsetting the dynamic kernel based on the first spatial offset position and the second spatial offset position; and The image features and the multi-scale point-by-point bimodal features are sampled and spatially aggregated using the shifted dynamic kernel.

24. The system of claim 23, wherein: The motion flow includes a plurality of feature maps, the feature maps including displacements between pixels of the blurred image and pixels of a sharp image corresponding to the blurred image, and the spatial deblurring path is further configured as follows: The first spatial offset position is determined based on the pixel displacement of the feature map.

25. The system of claim 21, wherein: The reblurring network is further configured to train the motion flow by self-supervising a reblurring process from sharp to blurred of a sharp image corresponding to the blurred image; and The deblurring network is further configured to be trained based on the trained motion flow.

26. The system according to any one of claims 14 to 25, wherein: The blurred image includes at least one of a photo image, a portion of a photo image, a video frame image, and a portion of a video frame image.

27. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the deblurring method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the deblurring method according to any one of claims 1 to 13.

Images