Motion blurred image reconstruction method in low-illumination environment

Abstract

The invention relates to a motion blurred image reconstruction method in a low-illumination environment, and the method comprises the steps: S1, collecting an original image through an RGB camera, and obtaining a motion blurred image under the conditions of long exposure, high-speed motion or camera shaking; s2, constructing an event-frame fusion algorithm based on an event camera principle, and jointly utilizing high time domain motion information of an event stream and the pixel intensity of a frame image; s3, a joint strategy of edge alignment constraint and fuzzy alignment constraint is provided, the edge alignment constraint guarantees the authenticity of the geometric contour, and the fuzzy alignment constraint is kept consistent with the physical imaging process; s4, designing a lightweight double-branch feature extraction network (a local branch, a global branch and a fusion algorithm), and capturing fine-grained features and a global structure of the sample at the same time; and S5, completing reconstruction and outputting a clear image. According to the method, the quality of blurred image reconstruction is effectively improved under the scenes of low illumination, high-speed target motion and the like, a large amount of noise and artifacts are inhibited, and the method has good robustness and generalization.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and data processing, and particularly relates to a method for reconstructing motion-blurred images under low-light conditions. Background Technology

[0002] In low-light scenarios such as security monitoring, driving recording, and mobile phone night scene, the exposure will be passively lengthened. At the same time, camera shake and high-speed movement of the target will produce strong noise and motion blur. Most traditional deblurring methods assume sufficient lighting, static background or linear degradation conditions, making it difficult to simultaneously achieve clear texture and low computational cost. There is an urgent need for an image reconstruction method for low-light motion scenes that can converge stably when weak light noise dominates, maintain sharp edges when objects undergo large displacements, be compatible with general hardware, and be deployable in real time. The method of this invention can be used in a variety of scenarios: (1) In urban security monitoring, motion blur is prone to occur at night and in low-light environments. This method can clearly reconstruct monitoring images under low light, enhance the feasibility of nighttime crime investigation and accident evidence collection, and improve security prevention capabilities. (2) In autonomous driving or assisted driving systems, road images under low-light conditions often exhibit motion blur due to vehicle speed or weather conditions. This method can effectively restore blurred images, improve the nighttime driving recognition accuracy of ADAS systems, and ensure driving safety. (3) When drones are inspecting at night or in low light environments, the images may be blurred due to flight speed or shaking. This method can help restore image details and improve the efficiency and accuracy of drone inspections in low light conditions. It can be widely used in factories in various fields.

[0003] Chinese patent document CN120525765A discloses an image deblurring method and system, belonging to the field of image processing technology. The method includes: acquiring an image to be deblurred; inputting the image to be deblurred into a pre-constructed deblurring model to acquire a deblurred image; the deblurring model is based on the U-Net model, constructed by adding feature fusion extraction modules before the downsampling module in the encoder and before the upsampling module in the decoder of the U-Net model. This invention can efficiently model the global context, directional details, and channel correlations of an image, effectively deblurring the image.

[0004] Chinese patent document CN120672626A discloses an image restoration method, apparatus, and storage medium, comprising: real-time acquisition of multidimensional inertial data; frequency domain analysis of the multidimensional inertial data using a sliding window short-time Fourier transform to obtain vibration intensity; if the vibration intensity does not exceed a preset threshold, acquiring an image; calculating the image sharpness index; determining whether the image is a blurred image based on a preset sharpness standard and the image sharpness index; if the image is determined to be a blurred image, classifying the blurred image into motion-blurred images and focus-blurred images; constructing point spread function models for motion-blurred images and focus-blurred images respectively; performing deconvolution processing or depth reconstruction on the blurred image using the point spread function model to obtain a sharp image; recalculating the sharpness index of the sharp image, and if it does not exceed the sharpness threshold, triggering re-acquisition or switching the restoration model to perform secondary restoration.

[0005] The current research focus remains on the "frame-based deblurring" framework, which has inherent limitations when dealing with high-speed motion and strong noise in low-light conditions: (1) loss of trajectory details under long exposure; (2) oversmoothing and artifacts in extremely dark scenes; and (3) reliance on deep learning networks, which leads to latency and energy consumption. To address the problems of low accuracy and high computational resource consumption in traditional methods, it is necessary to propose a motion-blurred image reconstruction method under low-light conditions. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes the following technical solution: a method for reconstructing motion-blurred images under low-light conditions, the specific technical solution including the following:

[0007] To solve the above problems, the technical solution provided by the present invention includes the following steps:

[0008] S1 Image Acquisition: Raw images are acquired using an RGB camera, and then motion-blurred images are obtained through long exposure, high-speed motion, and camera shake operations.

[0009] S2 Event-Frame Fusion Algorithm: Based on the principle of event camera, an event-frame fusion algorithm is proposed for joint reconstruction. It utilizes the rich motion information in the event and the intensity information of each pixel in the image branch to achieve joint reconstruction in blurred scenes.

[0010] S3 Dual Alignment Constraint Joint Strategy: A joint strategy of "edge alignment constraint + fuzzy alignment constraint" is proposed. The former ensures the authenticity of the geometric contour, while the latter ensures consistency with the physical imaging process.

[0011] S4 Dual-branch Extraction Network: A lightweight dual-branch feature extraction network with "local branch + global branch + fusion" is designed to capture fine-grained features and then achieve consistency between local details and global structure through fusion.

[0012] S5 Image Reconstruction: Obtain the reconstructed motion-deblurred image.

[0013] Step S2 event – ​​The frame fusion algorithm includes the following steps:

[0014] S21: Convert the events into a continuous brightness map, with each event e i =(x i ,y i ,t i ,p i ), where: (x i ,y i ) represents pixel coordinates, t i It is the event trigger time, polarity p i ∈{-1,+1} represents an increase or decrease in brightness, p i As constants, an event can be represented as E(x,y,t); the grayscale image in the image is represented as I. f (x,y), by fusing E(x,y,t) and I f Reconstruct the grayscale image at a given time (x,y). A value above the threshold C represents increased brightness, termed a positive event; conversely, a value below the threshold C represents decreased brightness, termed a negative event. The value of C varies across different sensors, where L(x,y,t) represents the signal brightness, and L(x,y,t) = logI(x,y,t), with C being the brightness change threshold in the logarithmic domain.

[0015] The brightness change model of the event camera is shown in formula (2):

[0016] L(x,y,t)=log I(x,y,t) (2)

[0017] The event triggering condition formula (3) is shown below:

[0018] ΔL(x,y,t)=L(x,y,t)-L(x,y,t last ) = p i C (3)

[0019] Where C is the event threshold, t last The time of the last event is given. The event camera only triggers the condition when the brightness change exceeds the threshold, so integrating the event will yield the cumulative value of the brightness change, as shown in formula (4):

[0020]

[0021] The event image is within the time window [t0, t1].

[0022]

[0023] As shown in formula (5), by "pixel accumulation" of the event stream over a certain period of time, a 2D image is obtained, similar to an edge map, which reflects the direction of brightness change under different scenes.

[0024] S22: Fusion with the frame image is achieved through incremental brightness updates, as shown in formula (6):

[0025]

[0026] Among them, I f (x,y) is the gray level of the most recent frame, ∑p i C is from frame time t f The event integral up to the current time t. The most recent frame provides initial brightness information, and exponential accumulation is performed on the frame image. Since the event is based on log brightness, a real-time reconstructed brightness map can be obtained.

[0027] S23: Event-Frame Weighted Fusion, assuming events on pixel (x,y) are ordered by time:

[0028]

[0029] As shown in formula (7), the calculation starts from the most recent frame, i.e., the initial value. For each event received within a certain time series, the brightness of the corresponding pixel is updated. By performing incremental event reconstruction in this way, high frame rate continuous images can be obtained.

[0030] Because image frames and event images contain noise, this invention employs a weighted fusion method. (±1) controls the frame image weights, which are adjusted adaptively based on frame image quality or event density. This prevents excessive event accumulation from causing brightness drift. The fusion method combines the global brightness provided by the frames with the high-frequency information provided by the events, ultimately outputting a continuous image sequence.

[0031]

[0032] Step S3, the joint strategy for double alignment constraints, includes the following steps:

[0033] S31: The "edge alignment constraint" strategy addresses the issue that traditional cameras, during long exposures, integrate the motion trajectories of objects into a single blurred image—the cumulative result of the same scene over a certain exposure time. Events, on the other hand, record the sparse changes in brightness edges during the exposure process. The blurred image reconstruction process utilizes the extremely high-resolution edge information provided by the event camera to constrain the blurred image, ensuring its gradient aligns with the direction of the accumulated event edges.

[0034]

[0035] in E represents the gradient of the reconstructed image in the x-direction, which is also the direction of change in spatial illumination intensity. boundary () represents the edge direction information observed by the event camera, which is the actual direction of light intensity change, i.e., (±1). sgn() is used to compare whether the gradient direction of the reconstructed image is consistent with the light intensity direction observed by the event camera. If the two terms are consistent, the dot product result is sgn = +1; otherwise, the result is sgn = -1. -sgn() represents the penalty for inconsistent directions, which is combined with the loss function to correct the gradient direction. A loss function is employed to avoid overfitting and to align the reconstructed image with the real event data. This invention chooses to use the logistic loss function, which is expressed as: Substitute this into formula (9) to obtain the result.

[0036]

[0037] When the gradient direction of the reconstructed image is consistent with the light intensity direction observed by the event camera (sgn = +1 → -1), the penalty approaches 0; when they are inconsistent (sgn = -1 → +1), the penalty increases. This discrimination method is helpful in distinguishing the credibility of information, thereby determining whether to learn, and this smooth probability discrimination method has a gradient that is always non-zero, resulting in good convergence performance after optimization.

[0038] S32: "Fuzzy Alignment Constraint" strategy, known The objective function becomes:

[0039]

[0040] Where B represents the blurred image and K represents the feature extraction network. For a clear image, To predict a sharp image, the first term aims to ensure that the reconstructed image remains consistent with the input image after degradation; the second term, as an edge alignment constraint, ensures that the reconstruction result is consistent with the edge information. A sharp reconstructed image can be obtained through multiple iterative calculations.

[0041] Step S4, the dual-branch feature extraction structure, includes the following steps:

[0042] S41: A dual-branch strategy, consisting of a global feature branch and a local feature branch. The feature information extracted by the branch structure at different scales is different. The given input features in this invention are... After convolution operation, we obtain Local features are used In other words, global features are represented by This means that adding the two results together yields...

[0043] Local branching strategy: Using unfold and reshape, F′ is segmented according to P=4, where P represents a spatially continuous patch. The resulting shape is (p×p, H′ / p, W′ / p, C). After averaging the channels, we get (p×p, H′ / p, W′ / p). FFN (Feed-Forward Network) is used to increase the model's expressive power, where the input vector is... Then the output It can be represented as:

[0044]

[0045] in: First, the input vector Projecting to a higher dimension in It is the model dimension. As the intermediate dimension of the feedforward network, it can capture more complex patterns and relationships in the input data in a high-dimensional space, making better use of feature information; secondly, it uses a non-linear activation function to perform non-linear transformation on the features in the high-dimensional space, which facilitates the processing of more complex feature information; finally, it projects the high-dimensional features back to the source. Under normal circumstances d low <d high The projected features undergo dimensionality reduction to remove redundant information while retaining the most important features. The three-channel feature information obtained through the above operations is then fed into four 3×3 convolutional filters to obtain...

[0046] Global branching strategy: First, for After performing depthwise convolution, pointwise convolution is then performed. First, spatial features are extracted using depthwise convolution, and then grouped into convolutions along the feature dimension. Each channel undergoes independent depthwise convolution. The kernel size of the depthwise convolution is D. K *D K *1, the number of convolution kernels is M, so the number of parameters is: D K *D K *M, computational complexity: D K *D K *M*D F *D F Then, pointwise convolution is used to extract channel features. Before output, N convolutional kernels of size 1×1*M are used to fuse all channels to obtain the overall global features. The number of parameters is M*N, and the computational cost is M*N*D. F *D F The computational cost of global branches: D K *D K *M*D F*D F +M*N*D F *D F The computational cost of ordinary convolution: D K *D K The ratio of *M*N is *M*N. In this invention, D K =3, As N increases, the dimension becomes closer to the original dimension. It runs faster and requires less computation, is easier to implement and deploy on different platforms, and its streamlined computation can achieve high-precision operations on smaller devices, indicating that the global branching strategy has high generalization and robustness.

[0047] S42: Branch Feature Fusion. After feature extraction using a dual-branch extraction structure, this invention first utilizes the combined effect of spatial attention and channel attention to obtain F. sa Then, after passing through DBS, the final output is... In this process It will pass through a one-dimensional channel attention map in sequence. A two-dimensional spatial attention map Expressed as a formula:

[0048]

[0049] in F represents element-wise multiplication. ca and F sa These represent the features selected by channel and spatial attention mechanisms, respectively. DBS = Dropout + BN + Swish is represented, where Dropout() represents random deactivation. During training, some elements are randomly set to zero with probability ρ. To maintain the expectation, the remaining elements are adjusted according to... Scaling reduces feature dependency, preventing overfitting and improving model generalization. Batch normalization (BN) performs channel normalization on each batch, which, when combined with Dropout (Dropout), balances variance, preventing gradient vanishing or exploding, thus accelerating training convergence.

[0050]

[0051] Compared to ReLU, which sets all negative values ​​to zero, Swish() retains a portion of the negative range, exhibiting a smaller response and preserving fine-grained features when subjected to attention suppression. The reciprocal of Swish is continuous, which is beneficial for maintaining gradients during deep fusion. When combined with Dropout(), Swish retains more weak signals, even though Dropout() randomly masks some elements, still yielding an overall feature representation. When combined with BN(), the smooth nonlinearity of Swish(), after BN()'s distribution coordination, can more precisely map the Gaussian distribution to the nonlinear space, improving feature discriminative power. sa The F″ obtained after random sparsification, distribution normalization, and Swish() activation has high robustness and expressiveness.

[0052] Compared with the prior art, the technical solution provided by this invention has the following advantages:

[0053] (1) The event-frame fusion algorithm in this invention makes up for the lack of dynamic information in the traditional frame method and overcomes the shortcomings of the pure event method in brightness restoration, thus improving the temporal consistency of sample information.

[0054] (2) The combined strategy of “edge alignment constraint + fuzzy alignment constraint” in this invention is different from the existing single constraint or pure data-driven method. It constructs a dual mechanism that simultaneously meets “geometric consistency” and “imaging consistency”, thereby improving the accuracy of sample information.

[0055] (3) The lightweight dual-branch feature extraction network of “local branch + global branch + fusion” in this invention is different from the single path of existing fuzzy reconstruction networks. It can adaptively handle multiple types of fuzziness and has high generalization. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the overall process framework of a motion-blurred image reconstruction method under low-light conditions according to the present invention.

[0057] Figure 2 This is a diagram illustrating the event signal mechanism in a motion-blurred image reconstruction method under low-light conditions according to the present invention.

[0058] Figure 3 This is a schematic diagram of a dual-branch extraction network in a motion-blurred image reconstruction method under low illumination conditions according to the present invention.

[0059] Figure 4 This is a schematic diagram of the global branching strategy in a motion-blurred image reconstruction method under low-light conditions according to the present invention;

[0060] Figure 5This is a schematic diagram of the swish function in a motion-blurred image reconstruction method under low-light conditions according to the present invention;

[0061] Figure 6 This presents the comparative experimental results of a motion-blurred image reconstruction method under low-light conditions proposed in this invention with four other algorithms on different datasets. Detailed Implementation

[0062] The technical solution will be further described below with reference to the accompanying drawings.

[0063] Example: Figure 1 As shown, the specific steps of this motion-blurred image reconstruction method under low-light conditions are as follows:

[0064] Example 1

[0065] S1 Image Acquisition: Raw images are acquired using an RGB camera, and then motion-blurred images are obtained through long exposure, high-speed motion, and camera shake operations.

[0066] S2 Event-Frame Fusion Algorithm: Based on the principle of event camera, an event-frame fusion algorithm is proposed for joint reconstruction. It utilizes the rich motion information in the event and the intensity information of each pixel in the image branch to achieve joint reconstruction in blurred scenes.

[0067] S3 Dual Alignment Constraint Joint Strategy: A joint strategy of "edge alignment constraint + fuzzy alignment constraint" is proposed. The former ensures the authenticity of the geometric contour, while the latter ensures consistency with the physical imaging process.

[0068] S4 Dual-branch Extraction Network: A lightweight dual-branch feature extraction network with "local branch + global branch + fusion" is designed to capture fine-grained features and then achieve consistency between local details and global structure through fusion.

[0069] S5 Image Reconstruction: Obtain the reconstructed motion-deblurred image.

[0070] Step S2 event – ​​The frame fusion algorithm includes the following steps:

[0071] S21: Convert the events into a continuous brightness map, with each event e i =(x i ,y i ,t i ,p i ), where: (x i ,y i ) represents pixel coordinates, t i It is the event trigger time, polarity p i ∈{-1,+1} represents an increase or decrease in brightness, p iAs constants, an event can be represented as E(x,y,t); the grayscale image in the image is represented as I. f (x,y), by fusing E(x,y,t) and I f Reconstruct the grayscale image at a given time (x,y). like Figure 2 As shown, values ​​above the threshold C represent increased brightness, termed a positive event (upward arrow), while values ​​below the threshold C represent decreased brightness, termed a negative event (downward arrow). The value of C varies across different sensors, where L(x,y,t) represents the signal brightness, and L(x,y,t) = logI(x,y,t), with C being the brightness change threshold in the logarithmic domain.

[0072]

[0073] The brightness change model of the event camera is shown in formula (2):

[0074] L(x,y,t)=log I(x,y,t) (2)

[0075] The event triggering condition formula (3) is shown below:

[0076] ΔL(x,y,t)=L(x,y,t)-L(x,y,t last ) = p i C (3)

[0077] Where C is the event threshold, t last The time of the last event is given. The event camera only triggers the condition when the brightness change exceeds the threshold, so integrating the event will yield the cumulative value of the brightness change, as shown in formula (4):

[0078]

[0079] The event image is within the time window [t0, t1].

[0080]

[0081] As shown in formula (5), by "pixel accumulation" of the event stream over a certain period of time, a 2D image is obtained, similar to an edge map, which reflects the direction of brightness change under different scenes.

[0082] S22: Fusion with the frame image is achieved through incremental brightness updates, as shown in formula (6):

[0083]

[0084] Among them, I f (x,y) is the grayscale value of the most recent frame, and ∑piC is the grayscale value from frame time t. fThe event integral up to the current time t. The most recent frame provides initial brightness information, and exponential accumulation is performed on the frame image. Since the event is based on log brightness, a real-time reconstructed brightness map can be obtained.

[0085] S23: Event-Frame Weighted Fusion, assuming events on pixel (x,y) are ordered by time:

[0086]

[0087] As shown in formula (7), the calculation starts from the most recent frame, i.e., the initial value. For each event received within a certain time series, the brightness of the corresponding pixel is updated. By performing incremental event reconstruction in this way, high frame rate continuous images can be obtained.

[0088] Because image frames and event images contain noise, this invention employs a weighted fusion method. (±1) controls the frame image weights, which are adjusted adaptively based on frame image quality or event density. This prevents excessive event accumulation from causing brightness drift. The fusion method combines the global brightness provided by the frames with the high-frequency information provided by the events, ultimately outputting a continuous image sequence.

[0089]

[0090] Step S3, the joint strategy for double alignment constraints, includes the following steps:

[0091] S31: The "edge alignment constraint" strategy addresses the issue that traditional cameras, during long exposures, integrate the motion trajectories of objects into a single blurred image—the cumulative result of the same scene over a certain exposure time. Events, on the other hand, record the sparse changes in brightness edges during the exposure process. The blurred image reconstruction process utilizes the extremely high-resolution edge information provided by the event camera to constrain the blurred image, ensuring its gradient aligns with the direction of the accumulated event edges.

[0092]

[0093] in E represents the gradient of the reconstructed image in the x-direction, which is also the direction of change in spatial illumination intensity. boundary () represents the edge direction information observed by the event camera, which is the actual direction of light intensity change, i.e., (±1). sgn() is used to compare whether the gradient direction of the reconstructed image is consistent with the light intensity direction observed by the event camera. If the two terms are consistent, the dot product result is sgn = +1; otherwise, the result is sgn = -1. -sgn() represents the penalty for inconsistent directions, which is combined with the loss function to correct the gradient direction. A loss function is employed to avoid overfitting and to align the reconstructed image with the real event data. This invention chooses to use the logistic loss function, which is expressed as: Substitute this into formula (9) to obtain the result.

[0094]

[0095] When the gradient direction of the reconstructed image is consistent with the light intensity direction observed by the event camera (sgn = +1 → -1), the penalty approaches 0; when they are inconsistent (sgn = -1 → +1), the penalty increases. This discrimination method is helpful in distinguishing the credibility of information, thereby determining whether to learn, and this smooth probability discrimination method has a gradient that is always non-zero, resulting in good convergence performance after optimization.

[0096] S32: "Fuzzy Alignment Constraint" strategy, known The objective function becomes:

[0097]

[0098] Where B represents the blurred image and K represents the feature extraction network. For a clear image, To predict a sharp image, the first term aims to ensure that the reconstructed image remains consistent with the input image after degradation; the second term, as an edge alignment constraint, ensures that the reconstruction result is consistent with the edge information. A sharp reconstructed image can be obtained through multiple iterative calculations.

[0099] Step S4, the dual-branch feature extraction structure, includes the following steps:

[0100] S41: Two-branch strategy, such as Figure 3 As shown, the dual-branch structure consists of a global feature branch and a local feature branch. The feature information extracted by the branch structure at different scales is different. The given input feature in this invention is... After convolution operation, we obtain Local features are used In other words, global features are represented by This means that adding the two results together yields...

[0101] Local branching strategy: Using unfold and reshape, F′ is segmented according to P=4, where P represents a spatially continuous patch. The resulting shape is (p×p, H′ / p, W′ / p, C). After averaging the channels, we get (p×p, H′ / p, W′ / p). FFN (Feed-Forward Network) is used to increase the model's expressive power, where the input vector is... Then the output It can be represented as:

[0102]

[0103] in: First, the input vector Projecting to a higher dimension in It is the model dimension. As the intermediate dimension of the feedforward network, it can capture more complex patterns and relationships in the input data in a high-dimensional space, making better use of feature information; secondly, it uses a non-linear activation function to perform non-linear transformation on the features in the high-dimensional space, which facilitates the processing of more complex feature information; finally, it projects the high-dimensional features back to the source. Under normal circumstances d low <d high The projected features undergo dimensionality reduction to remove redundant information while retaining the most important features. The three-channel feature information obtained through the above operations is then fed into four 3×3 convolutional filters to obtain...

[0104] Global branching strategy: such as Figure 4 As shown, firstly... After performing depthwise convolution, pointwise convolution is then performed. First, spatial features are extracted using depthwise convolution, and then grouped into convolutions along the feature dimension. Each channel undergoes independent depthwise convolution. The kernel size of the depthwise convolution is D. K *D K *1, the number of convolution kernels is M, so the number of parameters is: D K *D K *M, computational complexity: D K *D K *M*D F *D F Then, pointwise convolution is used to extract channel features. Before output, N convolutional kernels of size 1×1*M are used to fuse all channels to obtain the overall global features. The number of parameters is M*N, and the computational cost is M*N*D. F *D F The computational cost of global branches: D K *D K *M*D F *D F +M*N*D F *D F The computational cost of ordinary convolution: D K *D K The ratio of *M*N is *M*N. In this invention, D K =3, As N increases, the dimension becomes closer to the original dimension. It runs faster and requires less computation, is easier to implement and deploy on different platforms, and its streamlined computation can achieve high-precision operations on smaller devices, indicating that the global branching strategy has high generalization and robustness.

[0105] S42: Branch Feature Fusion. After feature extraction using a dual-branch extraction structure, this invention first utilizes the combined effect of spatial attention and channel attention to obtain F. sa Then, after passing through DBS, the final output is... In this process It will pass through a one-dimensional channel attention map in sequence. A two-dimensional spatial attention map Expressed as a formula:

[0106]

[0107] in F represents element-wise multiplication. ca and F sa These represent the features selected by channel and spatial attention mechanisms, respectively. DBS = Dropout + BN + Swish is represented, where Dropout() represents random deactivation. During training, some elements are randomly set to zero with probability ρ. To maintain the expectation, the remaining elements are adjusted according to... Scaling reduces feature dependency, preventing overfitting and improving model generalization. Batch normalization (BN) performs channel normalization on each batch, which, when combined with Dropout (Dropout), balances variance, preventing gradient vanishing or exploding, thus accelerating training convergence.

[0108]

[0109] Compared to ReLU, which sets all negative values ​​to zero, Swish() retains a portion of the negative range, exhibiting a smaller amplitude response and preserving fine-grained features when subjected to attentional suppression. For example... Figure 5 As shown, the reciprocal of Swish is continuous, which is beneficial for maintaining gradients during deep fusion. When combined with Dropout(), Swish retains more weak signals, and although Dropout() randomly masks some elements, the overall feature representation can still be obtained. When combined with BN(), after BN() coordinates the distribution, the smooth nonlinearity of Swish() can more precisely map the Gaussian distribution to the nonlinear space, improving feature discriminativeness. sa The F″ obtained after random sparsification, distribution normalization, and Swish() activation exhibits high robustness and expressiveness. For example... Figure 6 As shown, tests on various public datasets demonstrate that the algorithm of this invention has higher robustness than other deblurring algorithms.

[0110] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, such as changes to a material or reaction conditions, should be included within the scope of protection of the present invention.

Claims

1. A method for motion-blurred image reconstruction in low-illumination environment, characterized in that, The specific steps are: S1 image acquisition: using an RGB camera for original image acquisition, through long exposure, high-speed motion, camera shaking and other operations to obtain a motion blurred image; S2 event-frame fusion algorithm: an event-frame fusion algorithm is proposed to jointly reconstruct by using the principle of an event camera, using the rich motion information in the event and the intensity information of each pixel in the image branch to realize joint reconstruction in a blurred scene; S3 double alignment constraint joint strategy: a joint strategy of "edge alignment constraint + blur alignment constraint" is proposed, the former guarantees the authenticity of the geometric outline, and the latter guarantees consistency with the physical imaging process; S4 double-branch extraction network: a lightweight double-branch feature extraction network of "local branch + global branch + fusion" is designed to capture fine-grained features and realize the consistency of local details and global structure through fusion; S5 image reconstruction: a reconstructed motion deblurring image is obtained.

2. The method of claim 1, wherein, The acquisition device in step S1 is an RGB device.

3. The method of claim 1, wherein, The specific steps of step S2 are: S21: convert events into a continuous intensity map, each event e i = (x i ,y i ,t i ,p i ), where: (x i ,y i ) are pixel coordinates, t i is the event trigger time, polarity p i ∈ {-1, +1} represents an intensity increase or decrease, p i is a constant, the event can be represented as E(x,y,t); the grayscale image in the image is represented as I f (x,y), and the grayscale image at a certain moment is reconstructed by fusing E(x,y,t) and I f (x,y) If the value is higher than the threshold C, it represents an intensity increase and is called a positive event. Conversely, if the value is lower than the threshold C, it represents an intensity decrease and is called a negative event. The value of C is different on different sensors, where L(x,y,t) represents the signal intensity, L(x,y,t) = log I(x,y,t), and C is the intensity change threshold in the log domain. The luminance change model of the event camera is shown in formula (2): L(x,y,t)=logI(x,y,t) (2) The event triggering condition is shown in formula (3): AL(x, y, t) = L(x, y, t) - L(x, y, t last ) = p i C (3) where C is the event threshold, t last is the time of the last event. The event camera only triggers the condition when the luminance change exceeds the threshold, so integrating the events gives the accumulated value of the luminance change, as shown in equation (4): The event image is in a time window [t0, t1], As shown in formula (5), a 2D image is obtained by "pixel accumulation" of the event stream in a certain period of time, which is similar to an edge map and reflects the luminance change direction in different scenes. S22: fusion with frame images, fusion is realized by updating the incremental luminance, as shown in formula (6): where I f (x, y) is the gray scale of the latest frame, ∑p i C is the event integral from frame time t f to the current time t. The latest frame provides the initial intensity information, and the frame image is exponentially accumulated, because the event is based on the log intensity, so the real-time reconstructed intensity map can be obtained. S23: event-frame weighted fusion, assuming that the events on pixel (x, y) are sorted by time: As shown in equation (7), the calculation starts from the most recent frame, i.e. the initial value The luminance of the corresponding pixel is updated every time an event is received within a certain time sequence. Through the above event incremental reconstruction, continuous images with high frame rate can be obtained. Because there is noise in the image frame and the event image, the application adopts a weighted fusion method, in which (±1) controls the frame image weight, and the adaptive weight is adjusted according to the frame image quality or event density. Prevent luminance drift caused by excessive event accumulation. The global luminance provided by the fusion frame and the high-frequency information provided by the event are finally output as a continuous image sequence.

4. The method of claim 3, wherein, The specific steps of step S3 are: S31: "edge alignment constraint" strategy, a traditional camera will integrate the motion trajectory of an object into a blurred image under long exposure, that is, the cumulative result of the same scene in a certain exposure time, while the event is recording the sparse case of luminance edge change. The blur image reconstruction process is: using the high-resolution edge information provided by the event camera to constrain the blur image, so that the gradient direction of the reconstructed image is consistent with the light intensity direction observed by the event camera. wherein represents the gradient of the reconstructed image in the x direction, i.e. the direction of the spatial variation of the light intensity, E boundary () represents the edge direction information observed by the event camera, i.e. the true direction of the light intensity variation, i.e. (±1), and sgn() compares whether the gradient direction of the reconstructed image and the light intensity direction observed by the event camera are consistent. If the two items are consistent, the point multiplication result is sgn = +1; Otherwise the result is sgn = -1: -sgn() represents a penalty for the case of inconsistent direction, combined with the loss function to correct the gradient direction. The loss function is used to avoid overfitting, and the reconstructed image is aligned with the real event data. The logistic loss function is selected in the application, which is represented as: Bring it into formula (9). When the gradient direction of the reconstructed image is consistent with the light intensity direction observed by the event camera (sgn=+1→-1), the penalty is close to 0; when they are inconsistent (sgn=-1→+1), the penalty becomes large. This discrimination method is beneficial to distinguish the credibility of information, so as to determine whether to learn, and the gradient of this smooth probability discrimination method is always non-zero, and the convergence effect after optimization is good. S32: "fuzzy alignment constraints" strategy, known The objective function becomes: wherein B is a blurred image, K is a feature extraction network, is a clear image, is a predicted clear image, the first term aims to ensure that the reconstructed image is consistent with the input image after degradation; the second term as an edge alignment constraint can achieve consistency of the reconstructed result with edge information. After multiple iterations of calculation, a clear reconstructed image can be obtained.

5. The method of claim 4, wherein, The specific steps of step S4 are: S41: double-branch strategy, the double-branch is composed of a global feature branch and a local feature branch, the feature information extracted by the branch structure under different scales is different, and the input feature given by the application is After the convolution operation, the following is obtained Wherein the local feature is represented by The global feature is represented by The two results are added to obtain Local branch strategy: using unfold and reshape, F' is divided into P=4, where P represents the spatially continuous patch, the shape after division is (p×p, H' / p, W' / p, C), and the channel is averaged to obtain (p×p, H' / p, W' / p). The expression ability of the model is increased using FFN (Feed-Forward Network), where the input vector is The output of can be represented as: wherein: Firstly, the input vector is projected to a higher dimensional wherein is the model dimension, is the intermediate dimension of the feedforward network, which can capture more complex patterns and relationships in the input data and better utilize the feature information; secondly, the features in the high-dimensional space are nonlinearly transformed using a nonlinear activation function, which facilitates the processing of more complex feature information; finally, the high-dimensional features are projected back In general, d low < d high After projection, the features are processed by dimensionality reduction to remove redundant information and retain the most important features. The three-channel feature information obtained through the above operations is placed in four convolutional layers with 3x3 filters to obtain Global branching strategy: First, for After performing depthwise convolution, pointwise convolution is then performed. First, spatial features are extracted using depthwise convolution, and then grouped into convolutions along the feature dimension. Each channel undergoes independent depthwise convolution. The kernel size of the depthwise convolution is D. K *D K *1, the number of convolution kernels is M, so the number of parameters is: D K *D K *M, computational complexity: D K *D K *M*D F *D F Then, pointwise convolution is used to extract channel features. Before output, N convolutional kernels of size 1×1*M are used to fuse all channels to obtain the overall global features. The number of parameters is M*N, and the computational cost is M*N*D. F *D F The computational cost of global branches: D K *D K *M*D F *D F +M*N*D F *D F The computational cost of ordinary convolution: D K *D K The ratio of *M*N is *M*N. In this invention, D K =3, As N increases, the dimension becomes closer to the original dimension. It runs faster and requires less computation, is easier to implement and deploy on different platforms, and its streamlined computation can achieve high-precision operations on smaller devices, indicating that the global branching strategy has high generalization and robustness. S42: Branch feature fusion, after feature extraction by the double branch extraction structure, the application first utilizes the combined action of spatial attention and channel attention to obtain F sa , and finally outputs by DBS In this process will sequentially pass through a one-dimensional channel attention map and a two-dimensional spatial attention map Expressed by formula: wherein represents element-wise multiplication, F ca and F sa respectively represent the features selected by channel and spatial attention mechanism, DBS = Dropout + BN + Swish represents, wherein Dropout() represents random inactivation, in training, part of the elements are randomly set to zero with a probability p, in order to keep the expectation unchanged, the remaining elements are scaled according to , so as to reduce feature dependence, avoid overfitting phenomenon, and improve the generalization ability of the model. BN() performs channel standardization processing on each batch, which balances the variance when combined with Dropout(), avoids gradient disappearance or explosion, and thus speeds up the training convergence. Compared with ReLU, Swish() preserves a part of the negative value interval, has a small amplitude response, and can preserve fine-grained features when inhibited by attention. The reciprocal of Swish is continuous, which is conducive to maintaining the gradient when fusing in the deep layer. When used with Dropout(), Swish preserves more weak signals, and although Dropout() randomly masks some element units, the overall feature expression can still be obtained. When used with BN(), after the distribution is coordinated, the smooth nonlinearity of Swish() can more meticulously map the Gaussian distribution to the nonlinear space, improving the feature discrimination. sa After random sparsification, distribution standardization, and Swish() activation, F" has high robustness and expressiveness.

Images