Image deblurring method and device, equipment and storage medium

Abstract

The invention provides an image deblurring method and device, equipment and a storage medium, and relates to the technical field of image processing, and the specific implementation scheme is as follows: obtaining an original APS image collected by a fusion image sensor and first EVS data synchronously collected within the exposure time corresponding to the original APS image; performing alignment operation on the first EVS data and the original APS image to obtain aligned second EVS data; performing feature extraction and fusion processing on the second EVS data to obtain third EVS data; and performing deblurring processing on the original APS image based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image. Through the implementation of the scheme of the invention, the EVS data in the bimodal data synchronously collected by the fusion image sensor is adopted to guide the deblurring processing of the APS image, and the EVS data can provide richer motion information, so that the deblurring processing effect of the APS image is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to the field of image deblurring processing technology, and can be applied to the deblurring scene of APS (Active-Pixel Sensor) images. More specifically, the present application discloses an image deblurring method, device, equipment and storage medium. Background Art

[0002] In the process of capturing images through an image capture device, if there is relative motion between the image capture device and the photographed target, for example, the image capture device and / or the photographed target are in motion, the final captured image will be blurred.

[0003] At present, image deblurring is an important research topic in the field of computer vision. The image quality of blurred images can be enhanced through image deblurring algorithms. However, the image deblurring algorithms provided by related technologies are usually deblurring algorithms based on APS images. The guidance information provided when deblurring blurred APS images is relatively limited, resulting in poor final image deblurring effect.

[0004] It is worth noting that the techniques described in this section are not necessarily techniques that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any technique described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The main purpose of the present application is to provide an image deblurring method, apparatus, device and storage medium, which can at least solve the problem that the image deblurring effect of the APS image-based deblurring algorithm provided by the related art is poor.

[0006] In a first aspect, the present application provides an image deblurring method, comprising: acquiring an original APS image acquired by a fusion image sensor and first EVS data synchronously acquired within an exposure time corresponding to the original APS image; wherein the fusion image sensor is configured with APS pixels and EVS pixels; aligning the first EVS data with the original APS image to obtain aligned second EVS data; performing feature extraction and fusion processing on the second EVS data to obtain third EVS data; and deblurring the original APS image based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image.

[0007] According to a second aspect of the present application, there is provided an image deblurring device, comprising: an acquisition module, for acquiring an original APS image acquired by a fusion image sensor and first EVS data synchronously acquired within an exposure time corresponding to the original APS image; wherein the fusion image sensor is configured with APS pixels and EVS pixels; an alignment module, for aligning the first EVS data with the original APS image to obtain aligned second EVS data; a first processing module, for performing feature extraction and fusion processing on the second EVS data to obtain third EVS data; and a second processing module, for performing deblurring processing on the original APS image based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image.

[0008] The third aspect of the present application provides an electronic device, comprising: a memory and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements each step of the image deblurring method provided in the first aspect of the embodiment of the present application.

[0009] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image deblurring method provided in the first aspect of the embodiment of the present application are implemented.

[0010] As can be seen from the above, according to the image deblurring method, device, equipment and storage medium provided by the scheme of the present application, the original APS image acquired by the fusion image sensor and the first EVS data synchronously acquired within the exposure time corresponding to the original APS image are acquired; the first EVS data is aligned with the original APS image to obtain the aligned second EVS data; the second EVS data is feature extracted and fused to obtain the third EVS data; the original APS image is deblurred based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image. Through the implementation of the scheme of the present application, the EVS data in the dual-modal data synchronously acquired by the fusion image sensor is used to guide the deblurring of the APS image. Since the EVS data can provide richer motion information, the effect of the APS image deblurring can be effectively improved.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The drawings shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram of the basic process of an image deblurring method provided in one embodiment of the present application;

[0014] Figure 2 A schematic diagram of the principle of a gated recurrent unit provided in one embodiment of the present application;

[0015] Figure 3 A feature extraction schematic diagram provided for an embodiment of the present application;

[0016] Figure 4 A schematic diagram of the principle of a defuzzification processing network provided in one embodiment of the present application;

[0017] Figure 5 A schematic diagram of a detailed process of an image deblurring method provided in one embodiment of the present application;

[0018] Figure 6 A schematic diagram of functional modules of an image deblurring device provided in one embodiment of the present application;

[0019] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0021] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the term "plurality" means two or more, unless otherwise clearly and specifically defined.

[0022] In order to solve the problem that the image deblurring algorithm based on the APS image provided by the related art has poor image deblurring effect, an embodiment of the present application provides an image deblurring method, such as Figure 1 This is a basic flow chart of the image deblurring method provided in this embodiment. The image deblurring method includes the following steps:

[0023] Step 101: Acquire an original APS image acquired by a fusion image sensor and first EVS data synchronously acquired within an exposure time corresponding to the original APS image.

[0024] Specifically, in this embodiment, the fusion image sensor is configured with APS pixels and EVS pixels, that is, the fusion image sensor is integrated with an APS (Active Pixel Sensor) pixel array and an EVS (Event-based Vision Sensor) pixel array, which are distributed in different array areas of the image sensor. In actual applications, the layout and quantity of the two types of pixels in the overall pixel array of the sensor can be determined according to the actual application scenario, and this embodiment does not make a sole limitation on this.

[0025] It is worth noting that the EVS pixel generates event data in response to brightness changes. When the brightness change exceeds a certain threshold, it outputs event data including pixel coordinates (x, y), timestamp (t) and event polarity (p, with values ​​of +1 and -1, indicating an increase and decrease in brightness, respectively). Each event data is represented in the form of e = (t, x, y, p).

[0026] In practical applications, the fusion image sensor uses rolling exposure to synchronously collect APS images and event data streams. The exposure form of APS is rolling exposure, the exposure time of each row of data is equal, and the exposure start time of each row of data is arithmetically distributed (for example, the start exposure time of the Nth row of APS data is t, the start exposure time of the N+1th row is t+α, and the start exposure time of the N+2th row is t+2α).

[0027] In addition, this embodiment describes the imaging model of the APS image and the mathematical model for generating event data as follows:

[0028] In this embodiment, the imaging model of the APS image can be expressed as:

[0029]

[0030] Where I represents the APS image, the exposure time interval is [t0, t1], and L(t) represents the brightness value at time t.

[0031] In addition, the mathematical model for generating event data can be expressed as:

[0032] S=(x,y,p,t)

[0033]

[0034]

[0035] Among them, S represents event data, which is a four-element array, (x, y) represents the pixel coordinate position, t represents the timestamp, p represents the event polarity, when the brightness becomes stronger, p = 1, when the brightness becomes weaker, p = -1, c represents the comparison threshold, L x,y (t) represents the brightness value of point (x, y) at time t.

[0036] Step 102: align the first EVS data with the original APS image to obtain aligned second EVS data.

[0037] In a preferred embodiment, first, the exposure time intervals corresponding to different rolling shutter exposure rows within the corresponding exposure time of the original APS image are respectively obtained; secondly, the fourth EVS data corresponding to each exposure time interval is respectively extracted from the first EVS data; finally, all the extracted fourth EVS data are combined to obtain the aligned second EVS data.

[0038] For an APS image originally acquired, the total exposure time is T_exposure, and the exposure time difference between rows is T_line. In this embodiment, the exposure start time of the first row is Timestamp, and the exposure end time is Timestamp+T_exposure. In the first EVS data, the fourth event data E1_line_1 with a timestamp in the exposure time interval [Timestamp, Timestamp+T_exposure] of the first row is taken out, the exposure start time of the nth row is Timestamp+(n-1)*T_line, and the exposure end time is Timestamp+(n-1)*T_line+T_exposure. In the first EVS data, the fourth event data E1_line_n with a timestamp in the exposure time interval [Timestamp+(n-1)*T_line, Timestamp+(n-1)*T_line+T_exposure] of the nth row is taken out. Finally, E1_line_1, E1_line_2, ..., E1_line_n data are combined to obtain the second EVS data after rolling alignment.

[0039] Step 103: perform feature extraction and fusion processing on the second EVS data to obtain third EVS data.

[0040] In practical applications, since the APS exposure time may not be the same and the span will be relatively large, ranging from microseconds (us) to seconds (s), and EVS data has a sparse characteristic, at the same time, some pixels have events, while some pixels do not. Moreover, APS is a rolling shutter, and the time recorded by each row of APS will be different, that is, the EVS frames corresponding to each row will be different, and the interval time between each row of different sensors may also be different. Based on this, this embodiment provides a time-space network for feature extraction and fusion processing of the second EVS data.

[0041] In an optional implementation of the present embodiment, the above-mentioned step of extracting features and fusing the second EVS data to obtain the third EVS data includes: inputting the EVS data corresponding to different EVS pixels in the second EVS data into the gated cycle unit for feature extraction to obtain corresponding implicit features; fusing all implicit features to obtain the third EVS data.

[0042] Furthermore, the step of inputting the EVS data corresponding to different EVS pixels in the second EVS data into the gated loop unit for feature extraction to obtain the corresponding implicit features includes: for the EVS data corresponding to a single EVS pixel in the second EVS data, inputting different EVS data frames in the EVS data into a plurality of cascaded gated loop units in time sequence for feature extraction to obtain the corresponding implicit features.

[0043] like Figure 2 FIG. 1 is a schematic diagram showing the principle of a gated cycle unit provided by this embodiment, where r represents a reset gate and z represents an update gate. Figure 3 The figure shows a feature extraction schematic diagram provided by this embodiment, which shows that a pixel is passed through a series of gated recurrent units to obtain the final output. Assuming that an image has 2 million pixels, there will be 2 million such outputs. The aforementioned third EVS data can be obtained by fusing these 2 million outputs. It should be noted that the mathematical expression for feature extraction based on the gated recurrent unit in this embodiment is expressed as follows:

[0044]

[0045]

[0046] r (i,j) =σ(W xr x (i,j) +b xr +W hr h (i,j) +b hr )

[0047] z (i,j) =σ(W xz x (i,j) +b xz +W hr h (i,j) +b hz )

[0048] n (i,j) =tanh(W xn x (i,j) +b xn +r (i,j) *(W hn h (i,j) +b hn ))

[0049] h′ (i,j) =(1-z (i,j) )*n (i,j) +z (i,j) *h (i,j)

[0050]

[0051] h′ (i,j) =h (i,j)

[0052] Among them, x (i,j) and h (i,j) Both represent the input corresponding to the pixel position (i, j), W hr and W hn Both represent weights, b hr and b hn Both represent bias, r (i,j) and z (i,j) Both represent the output corresponding to pixel position (i, j).

[0053] It is worth noting that the time-space network of this embodiment uses RNN input for the input of each pixel point, and the RNN weights at each point are shared, and then the output of the RNN is connected to other CNN networks. The input of ordinary RNNs are all features in the form of one-dimensional vectors, and the time-space network input solves several pain points of the current EVS input to the network very well: the input length may be uncertain, the EVS data is sparse, the input time points of different pixels may be different, and it is adapted to the processing of different input resolution sizes. It should be understood that the time-space network of this embodiment has good spatiotemporal feature learning capabilities and is suitable for feature extraction of temporal data.

[0054] Step 104 : Deblurring the original APS image based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image.

[0055] Specifically, in this embodiment, the third EVS data and the original APS image are first input into the first downsampling network branch and the second downsampling network branch for downsampling processing, respectively, to obtain the first downsampling feature and the second downsampling feature; then the first downsampling feature and the second downsampling feature are both input into the upsampling network branch for upsampling processing, and a deblurred APS image corresponding to the original APS image is output.

[0056] In an optional implementation of this embodiment, the first downsampling network branch and the second downsampling network branch both include N cascaded downsampling modules, and the upsampling network branch includes N cascaded upsampling modules, where the value of N is greater than 1.

[0057] The step of inputting the third EVS data and the original APS image into the first downsampling network branch and the second downsampling network branch for downsampling processing to obtain the first downsampling feature and the second downsampling feature comprises: sequentially downsampling the third EVS data through N cascaded downsampling modules of the first downsampling network branch to obtain N first downsampling features, and sequentially downsampling the original APS image through N cascaded downsampling modules of the second downsampling network branch to obtain N second downsampling features;

[0058] Correspondingly, the above-mentioned step of inputting both the first down-sampled features and the second down-sampled features into the upsampling network branch for upsampling processing and outputting the deblurred APS image corresponding to the original APS image includes: upsampling the first down-sampled features and the second down-sampled features of the same level through the upsampling module of the deepest level of the upsampling network branch, and then sequentially using the upsampling modules of other levels to upsample the output features of the upsampling module of the next level and the first down-sampled features and the second down-sampled features of the same level, and finally outputting the deblurred APS image corresponding to the original APS image through the upsampling module of the shallowest level.

[0059] like Figure 4 The figure shows a schematic diagram of the principle of a deblurring processing network provided by this embodiment. The deblurring processing network is a three-branch network, wherein the two branches correspond to the original APS image (ie Figure 4 I1) and the third EVS data (i.e. Figure 4The down-sampling network branch of EF) is the remaining one branch, and the up-sampling network branch is the up-sampling network branch. The up-sampling network branch and the down-sampling network branch of this embodiment have the same scale consistency, that is, they have the same number of network modules, and the two down-sampling modules of the same scale (that is, Figure 4 The output of Block_down is also connected to the upsampling module of this scale (i.e. Figure 4 In practical applications, the downsampling network branch performs multi-level downsampling processing on the input features from shallow to deep layers, while the upsampling network branch performs multi-level upsampling processing on the input features from deep to shallow layers, and the final output feature is the deblurred APS image (that is, Figure 4 (middle I2).

[0060] It is worth mentioning that, in this embodiment, the downsampling module may include a cascaded convolution layer, a batch normalization (BN) layer, a Relu activation function layer, and a downsampling convolution layer; and / or, the upsampling module includes a cascaded convolution layer, a batch normalization layer, a Relu activation function layer, and an upsampling convolution layer. In practical applications, the step size of the downsampling convolution layer can be set to 2, and the upsampling convolution layer can be implemented based on deconvolution.

[0061] Figure 5 The method in is a refined image deblurring method provided by an embodiment of the present application, which specifically includes the following process:

[0062] Step 501, acquiring an original APS image acquired by a fusion image sensor and first EVS data synchronously acquired within an exposure time corresponding to the original APS image;

[0063] Step 502, respectively obtaining exposure time intervals corresponding to different rolling exposure rows within the exposure time of the original APS image;

[0064] Step 503: extracting EVS data units corresponding to each exposure time interval from the first EVS data;

[0065] Step 504: combine all extracted EVS data units to obtain aligned second EVS data;

[0066] Step 505: input the EVS data corresponding to different EVS pixels in the second EVS data into the gated cycle unit for feature extraction to obtain corresponding implicit features, and fuse all implicit features to obtain third EVS data;

[0067] Step 506: downsample the third EVS data sequentially through the N cascaded downsampling modules of the first downsampling network branch to obtain N first downsampled features, and downsample the original APS image sequentially through the N cascaded downsampling modules of the second downsampling network branch to obtain N second downsampled features;

[0068] Step 507: upsample the first downsampled features and the second downsampled features of the same level through the upsampling module at the deepest level of the upsampling network branch, and then use the upsampling modules at other levels to upsample the output features of the upsampling module at the next level and the first downsampled features and the second downsampled features of the same level in turn, and finally output the deblurred APS image corresponding to the original APS image through the upsampling module at the shallowest level.

[0069] It should be understood that the size of the serial number of each step in this embodiment does not mean the order of execution of the steps. The execution order of each step should be determined by its function and internal logic, and should not constitute a sole limitation on the implementation process of the embodiment of this application.

[0070] Figure 6 An image deblurring device is provided in one embodiment of the present application. The image deblurring device can be used to implement the image deblurring method in the above embodiment, and mainly includes:

[0071] An acquisition module 601 is used to acquire an original APS image acquired by a fusion image sensor and first EVS data synchronously acquired within an exposure time corresponding to the original APS image; wherein the fusion image sensor is configured with APS pixels and EVS pixels;

[0072] An alignment module 602 is used to align the first EVS data with the original APS image to obtain aligned second EVS data;

[0073] The first processing module 603 is used to perform feature extraction and fusion processing on the second EVS data to obtain third EVS data;

[0074] The second processing module 604 is configured to perform a deblurring process on the original APS image based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image.

[0075] In an optional implementation of the present embodiment, the alignment module is specifically used to: respectively obtain the exposure time intervals corresponding to different rolling shutter exposure rows within the corresponding exposure time of the original APS image; respectively extract the fourth EVS data corresponding to each exposure time interval from the first EVS data; and combine all the extracted fourth EVS data to obtain aligned second EVS data.

[0076] In an optional implementation of this embodiment, the first processing module is specifically used to: input the EVS data corresponding to different EVS pixels in the second EVS data into the gated cycle unit for feature extraction to obtain corresponding implicit features; and fuse all implicit features to obtain third EVS data.

[0077] In an optional implementation of the present embodiment, when the first processing module executes the function of inputting the EVS data corresponding to different EVS pixels in the second EVS data into the gated loop unit respectively for feature extraction to obtain the corresponding implicit features, it is specifically used for: for the EVS data corresponding to a single EVS pixel in the second EVS data, inputting different EVS data frames in the EVS data into a plurality of cascaded gated loop units in a time sequence for feature extraction to obtain the corresponding implicit features.

[0078] In an optional implementation of the present embodiment, the second processing module is specifically used to: input the third EVS data and the original APS image into the first downsampling network branch and the second downsampling network branch for downsampling processing, respectively, to obtain the first downsampling feature and the second downsampling feature; input the first downsampling feature and the second downsampling feature into the upsampling network branch for upsampling processing, and output a deblurred APS image corresponding to the original APS image.

[0079] In an optional implementation of the present embodiment, the first downsampling network branch and the second downsampling network branch both include N cascaded downsampling modules, and the upsampling network branch includes N cascaded upsampling modules, where the value of N is greater than 1. Accordingly, the second processing module is specifically used to: sequentially downsample the third EVS data through the N cascaded downsampling modules of the first downsampling network branch to obtain N first downsampling features, and sequentially downsample the original APS image through the N cascaded downsampling modules of the second downsampling network branch to obtain N second downsampling features; upsample the first downsampling features and the second downsampling features of the same level through the upsampling module of the deepest level of the upsampling network branch, and then sequentially use the upsampling modules of other levels to upsample the output features of the upsampling module of the next level and the first downsampling features and the second downsampling features of the same level, and finally output the deblurred APS image corresponding to the original APS image through the upsampling module of the shallowest level.

[0080] It should be noted that the image deblurring methods in the aforementioned embodiments can all be implemented based on the image deblurring device provided in this embodiment. Ordinary technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the image deblurring device described in this embodiment can be implemented by referring to the corresponding working process in the aforementioned method embodiment, and will not be repeated here.

[0081] Based on the technical solution of the embodiment of the present application described above, the original APS image acquired by the fusion image sensor and the first EVS data synchronously acquired within the exposure time corresponding to the original APS image are acquired; the first EVS data is aligned with the original APS image to obtain the aligned second EVS data; the second EVS data is feature extracted and fused to obtain the third EVS data; the original APS image is deblurred based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image. Through the implementation of the solution of the present application, the EVS data in the dual-modal data synchronously acquired by the fusion image sensor is used to guide the deblurring of the APS image. Since the EVS data can provide richer motion information, the effect of the deblurring of the APS image can be effectively improved.

[0082] Figure 7 An electronic device is provided in one embodiment of the present application. The electronic device can be used to implement the image deblurring method in the aforementioned embodiment, and mainly includes: a memory 701 and a processor 702. The memory 701 stores a computer program 703 that can be run on the processor 702. The memory 701 and the processor 702 are in communication connection. When the processor 702 executes the computer program 703, the image deblurring method in the aforementioned embodiment is implemented. The number of processors 702 can be one or more.

[0083] The memory 701 may be a high-speed random access memory (RAM) memory, or a non-volatile memory, such as a disk memory. The memory 701 is used to store executable program codes, and the processor 702 is coupled to the memory 701 .

[0084] Furthermore, the present application also provides a computer-readable storage medium, which may be provided in the electronic device in the above embodiments. Figure 7 Memory in the illustrated embodiment.

[0085] The computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the image deblurring method in the aforementioned embodiment is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0086] It should be understood that the devices and methods disclosed in the embodiments provided in the present application can also be implemented in any other equivalent manner. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0087] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0089] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a readable storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0090] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0091] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0092] The above is a description of the image deblurring method, device, equipment and storage medium provided in the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image deblurring method, characterized in that: include: Acquire an original APS image acquired by a fusion image sensor and first EVS data synchronously acquired within an exposure time corresponding to the original APS image; wherein the fusion image sensor is configured with APS pixels and EVS pixels; Aligning the first EVS data with the original APS image to obtain aligned second EVS data; Performing feature extraction and fusion processing on the second EVS data to obtain third EVS data; The original APS image is deblurred based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image.

2. The image deblurring method according to claim 1, characterized in that: The step of aligning the first EVS data with the original APS image to obtain aligned second EVS data includes: Respectively obtain exposure time intervals corresponding to different rolling exposure rows within the exposure time corresponding to the original APS image; Extracting fourth EVS data corresponding to each of the exposure time intervals from the first EVS data respectively; All the extracted fourth EVS data are combined to obtain aligned second EVS data.

3. The image deblurring method according to claim 1, characterized in that: The step of performing feature extraction and fusion processing on the second EVS data to obtain the third EVS data includes: Inputting the EVS data corresponding to different EVS pixels in the second EVS data into a gated recurrent unit for feature extraction to obtain corresponding implicit features; All the implicit features are fused to obtain the third EVS data.

4. The image deblurring method according to claim 3, characterized in that: The step of inputting the EVS data corresponding to different EVS pixels in the second EVS data into a gated recurrent unit for feature extraction to obtain corresponding implicit features includes: For the EVS data corresponding to a single EVS pixel in the second EVS data, different EVS data frames in the EVS data are respectively input into a plurality of cascaded gated recurrent units in a time sequence for feature extraction to obtain corresponding implicit features.

5. The image deblurring method according to claim 1, characterized in that: The step of performing a deblurring process on the original APS image based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image comprises: Inputting the third EVS data and the original APS image into the first downsampling network branch and the second downsampling network branch for downsampling processing respectively, to obtain a first downsampling feature and a second downsampling feature; The first down-sampled features and the second down-sampled features are both input into an up-sampling network branch for up-sampling processing, and a deblurred APS image corresponding to the original APS image is output.

6. The image deblurring method according to claim 5, characterized in that: The first downsampling network branch and the second downsampling network branch each include N cascaded downsampling modules, and the upsampling network branch includes N cascaded upsampling modules, where the value of N is greater than 1; The step of inputting the third EVS data and the original APS image into the first downsampling network branch and the second downsampling network branch for downsampling processing to obtain the first downsampling feature and the second downsampling feature comprises: Downsampling the third EVS data sequentially through the N cascaded downsampling modules of the first downsampling network branch to obtain N first downsampling features, and downsampling the original APS image sequentially through the N cascaded downsampling modules of the second downsampling network branch to obtain N second downsampling features; The step of inputting both the first down-sampled feature and the second down-sampled feature into an up-sampling network branch for up-sampling processing, and outputting a deblurred APS image corresponding to the original APS image comprises: The first down-sampled features and the second down-sampled features of the same level are up-sampled by the up-sampling module at the deepest level of the up-sampling network branch, and then the output features of the up-sampling module at the next level and the first down-sampled features and the second down-sampled features of the same level are up-sampled in turn by using the up-sampling modules at other levels, and finally a deblurred APS image corresponding to the original APS image is outputted by the up-sampling module at the shallowest level.

7. The image deblurring method according to claim 6, characterized in that: The downsampling module includes a cascaded convolutional layer, a batch normalization layer, a Relu activation function layer and a downsampling convolutional layer; And / or, the upsampling module includes a cascaded convolutional layer, a batch normalization layer, a Relu activation function layer and an upsampling convolutional layer.

8. An image deblurring device, characterized in that: include: An acquisition module, used to acquire an original APS image acquired by a fusion image sensor and first EVS data synchronously acquired within an exposure time corresponding to the original APS image; wherein the fusion image sensor is configured with APS pixels and EVS pixels; An alignment module, used for aligning the first EVS data with the original APS image to obtain aligned second EVS data; A first processing module, configured to perform feature extraction and fusion processing on the second EVS data to obtain third EVS data; The second processing module is configured to perform a deblurring process on the original APS image based on the third EVS data to obtain a deblurred APS image corresponding to the original APS image.

9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps of the image deblurring method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image deblurring method according to any one of claims 1 to 7 are implemented.

Images