An image deblurring method driven by motion perception event data representation

Abstract

The application discloses an image deblurring method based on motion-aware event data representation driving, and the steps include: 1, obtaining a blurred image to be processed and an event sequence in a corresponding exposure process; 2, processing the event sequence according to a proposed deviation accumulation method; 3, extracting event features with motion information through a cyclic motion feature extraction module; 4, extracting features of image data through an encoding network; 5, aligning and fusing the event features and the image features through a feature alignment and fusion module; 6, generating a deblurred image through a decoding network. The application excavates the motion information contained in the event stream data, and assists in enhancing the image deblurring effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image deblurring, and in particular to an image deblurring method based on motion-aware event data representation driving. BACKGROUND

[0002] Due to the limitation of exposure time, frame-based cameras often produce blurred images due to camera shake or object motion, etc. This blur is a common form of image degradation that occurs in various scenarios. Not only do photographers usually pursue clear and focused photos, but for many computer vision tasks, clear inputs are also crucial. Therefore, image deblurring is a fundamental task in the field of computer vision, which not only enhances image quality but also provides support for the normal operation of various algorithms.

[0003] Traditional image-based techniques usually try to derive a sharp image from a single blurred image. They achieve this goal by exploiting prior knowledge of natural images or making assumptions about the blurring operation. However, a lot of motion information is lost when capturing a single blurred image. The lack of motion information during exposure can lead to the appearance of artifacts in the deblurred results that we do not expect, especially in complex motion scenes.

[0004] Unlike frame-based cameras, event cameras inspired by biological systems provide a promising solution to this problem. Event cameras have microsecond-level temporal resolution and asynchronous output architecture, which can record fast motion. This ability enables event streams to store pixel intensity changes and provide accurate motion information to deblur corresponding frames. However, these methods mainly utilize event streams through polarity accumulation. In essence, they only consider the final cumulative result of the relative intensity change between adjacent events, ignoring the absolute intensity change caused by historical events. Therefore, this method cannot distinguish between different motion patterns, introducing unnecessary ambiguity in the deblurring process, leading to the appearance of edge artifacts, which limits the development of image deblurring research based on event data. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the present application provides an image deblurring method based on motion-aware event data representation driving, in order to achieve better deblurring performance in image deblurring tasks under various motion scenes, thereby effectively improving the deblurring effect.

[0006] In order to achieve the above application purposes, the present application adopts the following technical solutions:

[0007] The image deblurring method based on motion-aware event data representation driving of the present application is characterized in that the following steps are performed:

[0008] Step 1: Obtain the training image dataset and the event data within its corresponding exposure time:

[0009] Step 1.1, obtain a real blurred image dataset, denoted as X={x1,x2,…,x i ,…,x N}, where x i represents the i-th blurred image, i=1,2,…,N, N is the number of blurred images;

[0010] Get a real clear image dataset, denoted as Y={y1,y2,…,y i ,…,y N}, where y i represents the i-th clear image; let I={X,Y} represent the training image dataset;

[0011] Step 1.2: Generate a synthetic event dataset E={e1,e2,…,e i ,…,e N}, where e i Represents the i-th blurred image x i During its exposure time [t start ,t end ] generated within the synthetic event data, and , where x i,j ,y i,j ,t i,j ,p i,j Respectively represent the i-th synthetic event data e i The horizontal coordinate, vertical coordinate, timestamp and polarity of the j-th event in Q i Represents the i-th synthetic event data e i The total number of events included in ;

[0012] Step 2: Blurred image x i The exposure time [t start ,t end ] is divided into 2×A segments, and the time span of each segment is , using the deviation accumulation method to calculate the timestamps in [ ( a − 1 ) t e n d − t s t a r t 2 A + t s t a r t , t e n d − t s t a r t 2 ] and [ t e n d − t s t a r t 2 , t e n d − ( a − 1 ) t e n d − t s t a r t 2 A ] Process the events within the range and get x i The ath event represents c i a ;

[0013] Step 3: Construct an image deblurring network driven by motion perception event representation, including: S cyclic motion extraction modules, encoding network, feature alignment and fusion module, and decoding network:

[0014] Step 3.1, S cyclic motion extraction modules are sequentially applied to x i Process and get x i A event features {g i,s a |a=1,2,…,A}; among them, g i,s a represents the ath event feature output by the sth cyclic motion extraction module;

[0015] Step 3.2: The encoding network is used to calculate x i and the Ath event feature g i,s A Process and get x i The encoding feature F i,ST ;

[0016] Step 3.3, the decoding network is F i,ST Process and get x i Predicted clear image ;

[0017] Step 4: Use equations (9) and (10) to construct the back propagation loss function :

[0018] (9)

[0019] (10)

[0020] In formula (9), K is The number of pixels in for The kth pixel in y i k y i The kth pixel in ; MSE stands for mean square error;

[0021] Step 5: Based on the real blurred image dataset X and its corresponding synthetic event dataset E and the real clear image dataset Y, the image deblurring network is trained and the loss function L is calculated. The network parameters and weights are updated using the backpropagation algorithm. The training is terminated when the number of training iterations reaches the set number, thereby obtaining the optimal image deblurring model, which is used to deblur the blurred image to obtain the corresponding clear image.

[0022] The image deblurring method based on motion perception event data representation drive according to the present invention is also characterized in that step 2 is performed as follows:

[0023] Step 2.1, count x iThe timestamp of any pixel with coordinates (p, q) is in [ ( a − 1 ) t e n d − t s t a r t 2 A + t s t a r t , t e n d − t s t a r t 2 ] the number of events M within

[0024] Step 2.2, initialize m = 1;

[0025] The deviation value between the mth event occurring within [ ( a − 1 ) t e n d − t s t a r t 2 A + t s t a r t , t e n d − t s t a r t 2 ] and the initial time is recorded as D, and the cumulative amount of deviation value is recorded as R, and both D and R are initialized as 0;

[0026] Step 2.3, the polarity p i,m of the mth event is superimposed with the deviation value D of the mth event, to obtain the updated deviation value D ’ ; the updated deviation value D ’ is superimposed with the cumulative amount of deviation value R of the mth event, to obtain the updated cumulative amount of deviation value R ’ ;

[0027] Step 2.4, the updated deviation value D ’ is taken as the deviation value of the (m+1)th event, the updated cumulative amount of deviation value R ’ is taken as the cumulative amount of deviation value of the (m+1)th event, and m+1 is assigned to m, then return to step 2.3 for sequential execution until m > M, thereby obtaining the final updated deviation value D * and the cumulative amount of deviation value R * ;

[0028] Step 2.5, calculate the event representation value V = R i / M of any pixel with coordinates (p, q) on x * ;

[0029] Step 2.6, after processing each pixel on x i according to the process of step 2.1-step 2.5, the ath forward event representation c i i a- ;

[0030] Step 2.7, after processing the events with timestamps within [ t e n d − t s t a r t 2 , t e n d − ( a − 1 ) t e n d − t s t a r t 2 A ] according to the process of step 2.1-step 2.6, the ath backward event representation c i i a+ ;

[0031] Step 2.8, take the opposite of each element on c i a- , and superimpose it with c​​i a+ After splicing, we get x i The ath event represents c i a .

[0032] The step 3.1 is performed as follows:

[0033] When s=1, the sth cyclic motion extraction module uses formula (1) to obtain x i A event features {g i,s a |a=1,2,…,A}:

[0034] (1)

[0035] In formula (1), f ex and f re Respectively represent the two basic convolution modules in the cyclic motion extraction module, is the weight parameter to be trained; g i,s a-1 represents the a-1th event feature output by the sth cyclic motion extraction module; g i,s a represents the ath event feature output by the sth cyclic motion extraction module;

[0036] For g i,s a After downsampling, we get x i The event features after the ath downsampling ;

[0037] When s=2,3,…,S, Input the sth cyclic motion extraction module and use formula (2) to get x i A event features {g i,s a |a=1,2,…,A}:

[0038] (2)

[0039] In formula (2), It represents the ath downsampled event feature output by the s-1th cyclic motion extraction module.

[0040] The encoding network in step 3.2 has a total of S layers of encoding structure, wherein the first S-1 layers of encoding structure are sequentially composed of K basic convolution modules, 1 feature alignment and fusion module, and 1 downsampling layer; the Sth layer of encoding structure is sequentially composed of K basic convolution modules, 1 feature alignment and fusion module, 1 downsampling layer, and T basic convolution modules;

[0041] When s=1, the i-th blurred image x i After processing by the K basic convolution modules of the s-th layer encoding structure, the image feature u of the s-th layer structure is obtained. i,s ;

[0042] The feature alignment fusion module of the s-th layer coding structure uses formulas (3), (4) and (5) to align u i,s and the Ath event feature g output by the sth cyclic motion extraction module i,s A Processing is performed to obtain the fusion feature F of the s-th layer coding structure i,s :

[0043] (3)

[0044] (4)

[0045] F i , s = C o n v 3 ( C o n c a t ( [ A t t e n t i o n ( Q u ′ i , s , K g ′ i , s A , V g ′ i , s A ) , A t t e n t i o n ( Q g ′ i , s A , K u ′ i , s , V u ′ i , s ) ] ) ) (5)

[0046] In formula (3), formula (4) and formula (5), u i,s ’ and g i,s ’A are the aligned image features and event features of the s-th layer coding structure, and Indicated by u i,s ’ and g i,s ’A The generated query matrix, and Indicated by u i,s ’ and g i,s ’A The resulting bond matrix, and Indicated by u i,s ’ and g i,s ’A The generated value matrix, Attention represents the multi-head attention operation, Concat represents the splicing operation along the channel dimension; Conv1, Conv2, and Conv3 represent three convolution operations respectively;

[0047] The downsampling layer of the s-th layer coding structure is F i,s After downsampling, the downsampled fusion features of the s-th layer coding structure are obtained ;

[0048] When s = 2, 3, …, S, the down-sampled fusion feature of the s-1th layer encoding structure is obtained by the down-sampling layer of the s-1th layer encoding structure In the input of the s th layer encoding structure, after the processing of the K basic convolution modules, the compressed feature Z of the s th layer encoding structure is obtained i,s ;

[0049] The feature alignment fusion module of the s th layer encoding structure obtains the fusion feature F of the s th layer encoding structure by formula (6), formula (7) and formula (8) i,s :

[0050] (6)

[0051] (7)

[0052] F i , s = C o n v 3 ( C o n c a t ( [ A t t e n t i o n ( Q Z ′ i , s , K g ′ i , s A , V g ′ i , s A ) , A t t e n t i o n ( Q g ′ i , s A , K Z ′ i , s , V Z ′ i , s ) ] ) ) (8)

[0053] In formula (6), formula (7) and formula (8), Z i,s ’ is the aligned compressed feature of the s th layer encoding structure, 、 and respectively represent the query matrix, the key matrix and the value matrix generated by Z i,s ’ ;

[0054] The down-sampling layer of the s th layer encoding structure performs down-sampling processing on F i,s to obtain the down-sampled fusion feature of the s th layer encoding structure ; so that the down-sampled fusion feature of the S th layer encoding structure is output by the down-sampling layer of the S th layer encoding structure ;

[0055] When s = S, the T basic convolution modules of the S th layer encoding structure sequentially process the down-sampled fusion feature of the S th layer encoding structure to obtain the encoding feature F i of x i,ST .

[0056] The decoding network in step 3.3 has S layers, and each layer decoding structure is composed of one up-sampling layer and K basic convolution modules;

[0057] When s = 1, F i,ST firstly passes through the up-sampling layer of the s th layer decoding structure to obtain the up-sampled encoding feature output by the s th layer decoding structure ;

[0058] and the fusion feature Fi,S+1-s After adding, after the processing of the K basic convolution modules of the s-th decoding structure in turn, the restored feature H output by the s-th decoding structure is obtained i,s

[0059] When s=2, 3, …, S, the restored feature H output by the s-1-th decoding structure is i,s-1 After the processing of the up-sampling layer of the s-th decoding structure, the up-sampled restored feature output by the s-th decoding structure is obtained

[0060] The fusion feature F output by the S+1-s-th encoding structure i,S+1-s After adding, after the processing of the K basic convolution modules of the s-th decoding structure in turn, the restored feature H output by the s-th decoding structure is obtained i,s , so that the final restored feature H output by the S-th decoding structure is obtained i,S , and after a convolution layer operation, the predicted clear image x i of the image is generated .

[0061] The electronic device comprises a memory and a processor, and the memory is used to store a program supporting the processor to execute the image deblurring method, and the processor is configured to execute the program stored in the memory.

[0062] The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image deblurring method are executed.

[0063] Compared with the prior art, the present application has the following advantages:

[0064] 1. The intensity deviation at the initial exposure time carries information about the sequence of event generation, so the present application introduces a deviation accumulation method specially designed for deblurring problems. This method can respond differently to various motion patterns, thereby supplementing accurate motion information for the image deblurring task.

[0065] 2. Since event representations with longer time spans can capture global motion, and event representations with shorter time spans are better at depicting local fast motion, the present application designs a network with a cyclic motion extraction module. This network processes event representations with time spans from long to short through the cyclic motion extraction module, which can better extract motion features on local and global time scales. This design enables the network to effectively handle various complex motion scenes.

[0066] ​​3. Although the information captured by the event camera is highly correlated with that captured by the frame camera during equivalent exposure, there are still inevitable discrepancies due to the modal differences between them. Therefore, this paper proposes a feature alignment fusion module. This module aligns the two modalities into a more consistent space by injecting information from the other modality during the feature transformation process, ensuring that the motion information provided by the event representation more effectively guides the image deblurring process. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flow chart of the method of the present invention;

[0068] Figure 2 This is a structural diagram of the image deblurring method based on motion-aware temporal representation drive of the present invention;

[0069] Figure 3 This is a structural diagram of the cyclic motion extraction module of the present invention;

[0070] Figure 4 This is a structural diagram of the basic convolution module of the present invention;

[0071] Figure 5 This is a structural diagram of the feature alignment and fusion module of the present invention. Specific implementation methods

[0072] In this embodiment, an image deblurring method based on motion perception event representation is provided. For the specific process, see Figure 1 , the deviation accumulation method is used to make different responses to the time series generated by different motion modes, and the motion information is extracted by cyclic motion extraction to assist the deblurring process. The algorithm structure diagram of the whole method can be found in Figure 2 Specifically, the method is performed in the following steps:

[0073] Step 1: Obtain the training image dataset and the event data within its corresponding exposure time:

[0074] Step 1.1, obtain a real blurred image dataset, denoted as X={x1,x2,…,x i ,…,x N}, where x i represents the i-th blurred image, i=1,2,…,N, N is the number of blurred images;

[0075] Get a real clear image dataset, denoted as Y={y1,y2,…,y i ,…,y N}, where y i represents the i-th clear image; let I={X,Y} represent the training image dataset;

[0076] Step 1.2: Generate a synthetic event dataset E={e1,e2,…,e i ,…,e N}, where e i Represents the i-th blurred image x i During its exposure time [t start ,t end ] generated within the synthetic event data, and , where x i,j ,y i,j ,t i,j ,p i,j Respectively represent the i-th synthetic event data e i The horizontal coordinate, vertical coordinate, timestamp and polarity of the j-th event in Q i Represents the i-th synthetic event data e i The total number of events contained in .

[0077] Step 2: Blurred image x i The exposure time [t start ,t end ] is divided into 2×A segments. In this embodiment, A is 3, and the time span of each segment is , using the deviation accumulation method to calculate the timestamps in [ ( a − 1 ) t e n d − t s t a r t 2 A + t s t a r t , t e n d − t s t a r t 2 ] and [ t e n d − t s t a r t 2 , t e n d − ( a − 1 ) t e n d − t s t a r t 2 A ] Process the events within the range and get the ath event representation c i a :

[0078] Step 2.1, count x i At any pixel with coordinates (p,q) in [ ( a − 1 ) t e n d − t s t a r t 2 A + t s t a r t , t e n d − t s t a r t 2 ] The number of events in M;

[0079] Step 2.2, initialize m=1;

[0080] will occur in [ ( a − 1 ) t e n d − t s t a r t 2 A + t s t a r t , t e n d − t s t a r t 2 ] The mth event in the initial moment The deviation value between them is recorded as D, the cumulative deviation value is recorded as R, and both D and R are initialized to 0;

[0081] Step 2.3: Set the polarity of the mth event p i,m After superimposing the deviation value D of the mth event, the updated deviation value D is obtained. ’ ; Update the deviation value D ’ After superimposing the cumulative deviation value R of the mth event, the updated cumulative deviation value R is obtained. ’ .

[0082] Step 2.4: Update the deviation value D ’ As the deviation value of the m+1th event, the updated deviation value accumulation amount R ’ As the cumulative deviation value of the m+1th event, assign m+1 to m and return to step 2.3 to execute sequentially until m>M, thus obtaining the final updated deviation value D * and the cumulative deviation R * ;

[0083] Step 2.5, calculate x i The event representation value V=R on any pixel with coordinates (p,q) * / M;

[0084] Step 2.6: Follow the process from step 2.1 to step 2.5 to calculate x. i After processing each pixel, we get x i The ath forward event representation c i a- ;

[0085] Step 2.7: Follow the process from step 2.1 to step 2.6 to check the timestamp in [ t e n d − t s t a r t 2 , t e n d − ( a − 1 ) t e n d − t s t a r t 2 A ] After processing the events within the range, we get x i The ath backward event representation c i a+ ;

[0086] Step 2.8, for c i a- Take the opposite of the elements at each position in c and add them together. i a+ After splicing, we get x i The ath event represents c i a .

[0087] The advantage of selecting events of various time spans to process into event representations is that event representations of different time spans have different abilities to capture motion at different scales, which allows for better characterization of both global and local motion. In this example, the GoPro dataset was used to train and evaluate the model. This dataset includes video sequences of 33 different scenes, 22 of which were selected for model training, and the remaining for model evaluation.

[0088] Step 3: Construct an image deblurring network driven by motion perception event representation, including: S cyclic motion extraction modules, encoding network, feature alignment and fusion module, and decoding network:

[0089] Step 3.1: When s=1, the sth cyclic motion extraction module uses formula (1) to obtain xi A event features of x i,s a |a=1,2,…,A}:

[0090] (1)

[0091] In formula (1), f ex and f re respectively represent two basic convolution modules in the cycle motion extraction module, is a weight parameter to be trained; g i,s a-1 represents the (a-1)th event feature output by the s th cycle motion extraction module; g i,s a represents the ath event feature output by the s th cycle motion extraction module; the specific structures of the cycle motion extraction module and the basic convolution module are respectively shown in Figure 3 and Figure 4 .

[0092] After down-sampling processing is performed on g i,s a , the ath down-sampled event feature x i of x is obtained.

[0093] When s=2,3,…,S, x is input into the s th cycle motion extraction module, and A event features of x i are obtained by using formula (2): i,s a |a=1,2,…,A}:

[0094] (2)

[0095] In formula (2), x represents the ath down-sampled event feature output by the (s-1)th cycle motion extraction module; in this embodiment, the cycle motion extraction module has three.

[0096] Step 3.2 The encoding network has S layers of encoding structure, wherein the first S-1 layers of encoding structure are sequentially composed of K basic convolution modules, one feature alignment fusion module and one down-sampling layer; the S th layer of encoding structure is sequentially composed of K basic convolution modules, one feature alignment fusion module, one down-sampling layer and T basic convolution modules; in this embodiment, as shown in Figure 2 , S is 3, K is 1, T is 3, the number of channels input by the basic convolution module of each layer of encoding structure is 64, 128 and 256, and the specific structure of the feature alignment fusion module is shown in Figure 5 .

[0097] When s=1, the i-th blurred image x i After processing by the K basic convolution modules of the s-th layer encoding structure, the image feature u of the s-th layer structure is obtained. i,s ;

[0098] The feature alignment fusion module of the s-th layer coding structure uses formulas (3), (4) and (5) to align u i,s and the Ath event feature g output by the sth cyclic motion extraction module i,s A Processing is performed to obtain the fusion feature F of the s-th layer coding structure i,s :

[0099] (3)

[0100] (4)

[0101] F i , s = C o n v 3 ( C o n c a t ( [ A t t e n t i o n ( Q u ′ i , s , K g ′ i , s A , V g ′ i , s A ) , A t t e n t i o n ( Q g ′ i , s A , K u ′ i , s , V u ′ i , s ) ] ) ) (5)

[0102] In formula (3), formula (4) and formula (5), u i,s ’ and g i,s ’A are the aligned image features and event features of the s-th layer coding structure, and Indicated by u i,s ’ and g i,s ’A The generated query matrix, and Indicated by u i,s ’ and g i,s ’A The resulting bond matrix, and Indicated by u i,s ’ and g i,s ’A The generated value matrix, Attention represents the multi-head attention operation, Concat represents the splicing operation along the channel dimension; Conv1, Conv2, and Conv3 represent three convolution operations respectively.

[0103] The downsampling layer of the s-th layer coding structure is F i,s After downsampling, the downsampled fusion features of the s-th layer coding structure are obtained ;

[0104] When s=2,3,…,S, the fusion features of the downsampled encoding structure of the s-1 layer are The input is processed by K basic convolution modules to obtain the compression feature Z of the s-th layer coding structure. i,s ;

[0105] The feature alignment fusion module of the s-th layer coding structure uses formula (6), formula (7) and formula (8) to obtain the fusion feature F of the s-th layer coding structure i,s :

[0106] (6)

[0107] (7)

[0108] F i , s = C o n v 3 ( C o n c a t ( [ A t t e n t i o n ( Q Z ′ i , s , K g ′ i , s A , V g ′ i , s A ) , A t t e n t i o n ( Q g ′ i , s A , K Z ′ i , s , V Z ′ i , s ) ] ) ) (8)

[0109] In formula (6), formula (7) and formula (8), Z i,s ’ is the aligned compressed feature of the s-th layer encoding structure, 、 and Respectively represented by Z i,s ’ The resulting query matrix, key matrix, and value matrix.

[0110] The downsampling layer of the s-th layer coding structure is F i,s After downsampling, the downsampled fusion features of the s-th layer coding structure are obtained ; Thus, the down-sampling layer of the S-th layer encoding structure outputs the down-sampled fusion feature ;

[0111] When s=S, the T basic convolution modules of the S-th layer coding structure sequentially fused the downsampled features of the S-th layer coding structure. After processing, we get x i The encoding feature F i,ST .

[0112] Step 3.3 The decoding network has a total of S layers. Each decoding structure consists of one upsampling layer and K basic convolution modules. In this embodiment, S is 3 and K is 1. The number of input channels of the basic convolution module of each decoding structure is 256, 128, and 64, respectively.

[0113] When s=1, F i,ST After being processed by the upsampling layer of the s-th layer decoding structure, the upsampled encoding features output by the s-th layer decoding structure are obtained. ;

[0114] The fusion feature F output by the S+1-s layer encoding structurei,S+1-s After adding, the recovered feature H output by the s-th decoding structure is obtained after sequentially passing through the K basic convolution modules of the s-th decoding structure i,s ; by sharing the features corresponding to the encoding structure with the decoding structure, the details in the original features are preserved.

[0115] When s = 2, 3, …, S, the recovered feature H output by the s-1-th decoding structure is i,s-1 After passing through the up-sampling layer of the s-th decoding structure, the up-sampled recovered feature H output by the s-th decoding structure is obtained ;

[0116] and the fusion feature F output by the S+1-s-th encoding structure i,S+1-s After adding, the recovered feature H output by the s-th decoding structure is obtained after sequentially passing through the K basic convolution modules of the s-th decoding structure i,s , so that the final recovered feature H output by the S-th decoding structure is obtained i,S , and after a convolution layer operation, the predicted clear image x i is generated ; the convolution operation has a convolution kernel size of and a step size of 1.

[0117] Step 4: Use formula (9) and formula (10) to construct the loss function of back propagation :

[0118] (9)

[0119] (10)

[0120] In formula (9), K is the number of pixel points in , y is the k-th pixel point in , and y i k is the k-th pixel point in y i ; MSE represents the mean square error.

[0121] Step 5: Based on the real blurred image dataset X and its corresponding synthetic event dataset E and the real clear image dataset Y, the image deblurring network is trained, and the loss function L is calculated, the network parameters and weights are updated using the back propagation algorithm, the learning rate is adjusted using the cosine annealing strategy, in this example, the initial learning rate is taken as 0.0001, and when the training iteration number reaches the set number, the training is terminated, so as to obtain the optimal image deblurring model, which is used for deblurring the blurred image to obtain the corresponding clear image.

[0122] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0123] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed. Example

[0124] In order to verify the effectiveness of the method of the present invention in various scenarios, the GoPro dataset, HS-ERGB dataset, and REBlur dataset were selected for training and testing in this embodiment.

[0125] The method was trained on the GoPro dataset, which includes 2,059 blurry and clear image pairs from 22 scenes. The test set includes 1,089 blurry-clear image pairs from 11 scenes. To obtain event data, the ESIM event simulator was used to generate the corresponding event sequences. For evaluation on real-world event data, the HS-ERGB and REBlur datasets were used. These datasets contain event data captured in real scenes using an event camera.

[0126] The present invention adopts peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indicators.

[0127] In this embodiment, ten methods are selected for comparison with the proposed method, namely LEVS (see paper: Learning to extract a video sequence from a single motion-blurred image.), EDI (see paper: Bringing a blurry frame alive at high frame-rate with an event camera.), SRN (see paper: Scale-recurrent network for deep image deblurring.), EVDI (see paper: Unifying motion deblurring and frame interpolation with events.), HINet (see paper: Half instance normalization network for image restoration.), Restormer (see paper: Efficient transformer for high-resolution image restoration.), MSDI-Net (see paper: Learning degradation representations for image deblurring.), NAFNet (see paper: Simple baselines for image restoration.), UFPNet (see paper: Self-supervised non-uniform kernel estimation with flow-based motion prior for blind image deblurring.), EFNet (see paper: Event-based fusion for motion deblurring with cross-modal The experimental results are shown in Table 1, where Ours represents the method of the present invention:

[0128]

[0129] Experimental results show that our method performs better than ten other methods on three different datasets, demonstrating the feasibility of our method. The experiments demonstrate that our method is highly capable of capturing diverse motion patterns, effectively extracting motion information, and assisting in image deblurring tasks.

Claims

1. An image deblurring method driven by motion perception event data representation, characterized in that: The steps are as follows: Step 1: Obtain the training image dataset and the event data within its corresponding exposure time: Step 1.1, obtain a real blurred image dataset, denoted as X={x1,x2,…,x i ,…,x N }, where x i represents the i-th blurred image, i=1,2,…,N, N is the number of blurred images; Get a real clear image dataset, denoted as Y={y1,y2,…,y i ,…,y N }, where y i represents the i-th clear image; let I={X,Y} represent the training image dataset; Step 1.2: Generate a synthetic event dataset E={e1,e2,…,e i ,…,e N }, where e i Represents the i-th blurred image x i During its exposure time [t start ,t end ] generated within the synthetic event data, and , where x i,j ,y i,j ,t i,j ,p i,j Respectively represent the i-th synthetic event data e i The horizontal coordinate, vertical coordinate, timestamp and polarity of the j-th event in Q i Represents the i-th synthetic event data e i The total number of events included in ; Step 2: Blurred image x i The exposure time [t start ,t end ] is divided into 2×A segments, and the time span of each segment is , using the deviation accumulation method to calculate the timestamps in and Process the events within the range and get x i The ath event represents c i a ; Step 3: Construct an image deblurring network driven by motion perception event representation, including: S cyclic motion extraction modules, encoding network, feature alignment and fusion module, and decoding network: Step 3.1, S cyclic motion extraction modules are sequentially applied to x i Process and get x i A event features {g i,s a |a=1,2,…,A}; among them, g i,s a represents the ath event feature output by the sth cyclic motion extraction module; Step 3.2: The encoding network is used to calculate x i and the Ath event feature g i,s A Process and get x i The encoding feature F i,ST ; Step 3.3, the decoding network is F i,ST Process and get x i Predicted clear image ; Step 4: Use equations (9) and (10) to construct the back propagation loss function : (9) (10) In formula (9), K is The number of pixels in for The kth pixel in y i k y i The kth pixel in ; MSE stands for mean square error; Step 5: Based on the real blurred image dataset X and its corresponding synthetic event dataset E and the real clear image dataset Y, the image deblurring network is trained and the loss function L is calculated. The network parameters and weights are updated using the backpropagation algorithm. The training is terminated when the number of training iterations reaches the set number, thereby obtaining the optimal image deblurring model, which is used to deblur the blurred image to obtain the corresponding clear image.

2. The image deblurring method based on motion perception event data representation drive according to claim 1, characterized in that: Described step 2 is carried out as follows: Step 2.1, count x i At any pixel with coordinates (p,q) in The number of events in M; Step 2.2, initialize m=1; will occur in The mth event in the initial moment The deviation value between them is recorded as D, the cumulative deviation value is recorded as R, and both D and R are initialized to 0; Step 2.3: Set the polarity of the mth event p i,m After superimposing the deviation value D of the mth event, the updated deviation value D is obtained. ’ ; Update the deviation value D ’ After superimposing the deviation value cumulative amount R of the mth event, the updated deviation value cumulative amount R is obtained ’ ; Step 2.4: Update the deviation value D ’ As the deviation value of the m+1th event, the updated deviation value accumulation amount R ’ As the cumulative deviation value of the m+1th event, assign m+1 to m and return to step 2.3 to execute sequentially until m>M, thus obtaining the final updated deviation value D * and the cumulative deviation R * ; Step 2.5, calculate x i The event representation value V=R on any pixel with coordinates (p,q) * / M; Step 2.6: Follow the process from step 2.1 to step 2.5 to calculate x. i After processing each pixel, we get x i The ath forward event representation c i a- ; Step 2.7: Follow the process from step 2.1 to step 2.6 to check the timestamp in After processing the events within the range, we get x i The ath backward event representation c i a+ ; Step 2.8, for c i a- Take the opposite of the elements at each position in c and add them together. i a+ After splicing, we get x i The ath event represents c i a .

3. The image deblurring method based on motion perception event data representation drive according to claim 2, characterized in that: The step 3.1 is performed as follows: When s=1, the sth cyclic motion extraction module uses formula (1) to obtain x i A event features {g i,s a |a=1,2,…,A}: (1) In formula (1), f ex and f re Respectively represent the two basic convolution modules in the cyclic motion extraction module, is the weight parameter to be trained; g i,s a-1 represents the a-1th event feature output by the sth cyclic motion extraction module; g i,s a represents the ath event feature output by the sth cyclic motion extraction module; For g i,s a After downsampling, we get x i The event features after the ath downsampling ; When s=2,3,…,S, Input the sth cyclic motion extraction module and use formula (2) to get x i A event features {g i,s a |a=1,2,…,A}: (2) In formula (2), It represents the ath downsampled event feature output by the s-1th cyclic motion extraction module.

4. The image deblurring method based on motion perception event data representation drive according to claim 3, characterized in that: The encoding network in step 3.2 has a total of S layers of encoding structure, wherein the first S-1 layers of encoding structure are sequentially composed of K basic convolution modules, 1 feature alignment and fusion module, and 1 downsampling layer; the Sth layer of encoding structure is sequentially composed of K basic convolution modules, 1 feature alignment and fusion module, 1 downsampling layer, and T basic convolution modules; When s=1, the i-th blurred image x i After processing by the K basic convolution modules of the s-th layer encoding structure, the image feature u of the s-th layer structure is obtained. i,s ; The feature alignment fusion module of the s-th layer coding structure uses formulas (3), (4) and (5) to align u i,s and the Ath event feature g output by the sth cyclic motion extraction module i,s A Processing is performed to obtain the fusion feature F of the s-th layer coding structure i,s : (3) (4) (5) In formula (3), formula (4) and formula (5), u i,s ’ and g i,s ’A are the aligned image features and event features of the s-th layer coding structure, and Indicated by u i,s ’ and g i,s ’A The generated query matrix, and Indicated by u i,s ’ and g i,s ’A The resulting bond matrix, and Indicated by u i,s ’ and g i,s ’A The generated value matrix, Attention represents the multi-head attention operation, Concat represents the splicing operation along the channel dimension; Conv1, Conv2, and Conv3 represent three convolution operations respectively; The downsampling layer of the s-th layer coding structure is F i,s After downsampling, the downsampled fusion features of the s-th layer coding structure are obtained ; When s=2,3,…,S, the fusion features after downsampling of the s-1th layer encoding structure are The input is processed by K basic convolution modules to obtain the compression feature Z of the s-th layer coding structure. i,s ; The feature alignment fusion module of the s-th layer coding structure uses formula (6), formula (7) and formula (8) to obtain the fusion feature F of the s-th layer coding structure i,s : (6) (7) (8) In formula (6), formula (7) and formula (8), Z i,s ’ is the aligned compressed feature of the s-th layer encoding structure, 、 and Respectively represented by Z i,s ’ The generated query matrix, key matrix and value matrix; The downsampling layer of the s-th layer coding structure is F i,s After downsampling, the downsampled fusion features of the s-th layer coding structure are obtained. ; Thus, the downsampling layer of the S-th layer encoding structure outputs the downsampled fusion features ; When s=S, the T basic convolution modules of the S-th layer coding structure sequentially fused the downsampled features of the S-th layer coding structure. After processing, we get x i The encoding feature F i,ST .

5. The image deblurring method based on motion perception event data representation drive according to claim 3, characterized in that: The decoding network in step 3.3 has a total of S layers, and each decoding structure consists of 1 upsampling layer and K basic convolution modules; When s=1, F i,ST After being processed by the upsampling layer of the s-th layer decoding structure, the upsampled encoding features output by the s-th layer decoding structure are obtained. ; The fusion feature F output by the S+1-s layer encoding structure i,S+1-s After addition, the K basic convolution modules of the s-th layer decoding structure are processed in sequence to obtain the restored feature H output by the s-th layer decoding structure. i,s ; When s=2,3,…,S, the restored feature H output by the s-1th layer decoding structure i,s-1 After being processed by the upsampling layer of the s-th layer decoding structure, the upsampled restored features output by the s-th layer decoding structure are obtained. ; The fusion feature F output by the S+1-s layer encoding structure i,S+1-s After addition, the K basic convolution modules of the s-th layer decoding structure are processed in sequence to obtain the restored feature H output by the s-th layer decoding structure. i,s , so that the final restored feature H is output by the S-th layer decoding structure i,S , and after a convolution layer operation, generate x i Predicted clear image .

6. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the image deblurring method according to any one of claims 1 to 5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image deblurring method according to any one of claims 1 to 5 are executed.

Images