Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The method comprises the steps that an image collected by an image sensor and an event frame collected by a dynamic vision sensor are acquired; wherein the event frame comprises brightness change information of pixel points in the image; determining whether the image has motion blur according to the brightness change information in the event frame; generating a plurality of sub-images according to the image under the condition that the motion blur exists in the image; wherein the resolution of the sub-image is consistent with the resolution of the event frame; and performing motion blur removal on the image according to the event frame and the plurality of sub-images. According to the scheme, the motion blur of the image can be removed with a small calculation amount.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, electronic equipment, and storage medium. Background Art

[0002] This includes electronic devices such as cameras, mobile phones, tablets, drones, cars, and wearable devices. During the image exposure process, the movement of the electronic device or the subject may cause aliasing of pixel information at different spatial positions, resulting in motion blur in the captured image.

[0003] Traditional methods for removing motion blur, such as coded exposure technology, motion invariant photography technology, inverse filtering, Wiener filtering, etc., reduce motion blur in images by eliminating pixel information superimposed on pixel points. However, the effect of motion blur removal by such methods is poor and it is difficult to meet user needs. Summary of the invention

[0004] The present disclosure provides an image processing method and device, an electronic device, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Acquire an image captured by an image sensor and an event frame captured by a dynamic vision sensor; wherein the event frame includes brightness change information of pixel points in the image;

[0007] Determining whether the image has motion blur according to brightness change information in the event frame;

[0008] In the case where the image has motion blur, a plurality of sub-images are generated according to the image; wherein the resolution of the sub-images is consistent with the resolution of the event frame;

[0009] Motion blur removal is performed on the image according to the event frame and the multiple sub-images.

[0010] In some embodiments, generating a plurality of sub-images according to the image includes:

[0011] The image is traversed using a sliding window, and pixel values ​​of pixels in the window in a preset color channel are obtained; wherein the preset color channels of adjacent pixels in the window are different;

[0012] The pixel values ​​of the pixel points at the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixel points included in the sub-image.

[0013] In some embodiments, the number of traversals of the window is the resolution of the event frame, and the pixels in the window do not overlap at different traversal times.

[0014] In some embodiments, removing motion blur from the image according to the event frame and the plurality of sub-images includes:

[0015] The event frame and the multiple sub-images are processed using a preset motion blur removal model to obtain an image after removing motion blur; wherein the preset motion blur removal model is trained based on historical image samples and historical event frame samples.

[0016] In some embodiments, the method further comprises:

[0017] Acquire multiple historical image samples collected by an image sensor and multiple historical event frame samples collected by a dynamic vision sensor; wherein the historical event frame samples include brightness change information of pixel points in the corresponding historical image samples; and the resolution of the historical image samples is greater than a preset resolution threshold;

[0018] Performing fuzzy processing on a plurality of historical image samples to obtain a plurality of fuzzy image samples;

[0019] A preset initial model is trained using a plurality of blurred image samples, a plurality of historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model.

[0020] In some embodiments, the method of training a preset initial model using a plurality of blurred image samples, a plurality of historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model includes:

[0021] For each blurred image sample, determining a motion feature of an object in the blurred image sample according to a historical event frame sample corresponding to a historical image sample associated with the blurred image sample;

[0022] Performing motion blur removal on blurred image samples according to the motion characteristics;

[0023] According to the difference between each image after motion blur removal and a historical image sample associated with the blurred image sample before motion blur removal, the parameters of the preset initial model are adjusted to obtain the preset motion blur removal model.

[0024] In some embodiments, determining whether the image has motion blur according to the brightness change information in the event frame includes:

[0025] The brightness change information in the event frame is processed using a preset motion blur detection model to obtain a result of whether the image has motion blur; wherein the preset motion blur detection model is obtained based on training of a neural network model.

[0026] In some embodiments, the method further comprises:

[0027] Preprocessing the brightness change information in the event frame; wherein the preprocessing includes at least one of the following: noise elimination processing and delay elimination processing;

[0028] The determining, according to the brightness change information in the event frame, whether the image has motion blur comprises:

[0029] It is determined whether the image has motion blur according to the brightness change information in the preprocessed event frame.

[0030] In some embodiments, the method further comprises:

[0031] Correcting the image; wherein the correction includes at least one of the following: bad pixel correction, black level correction, lens shading correction;

[0032] Generating a plurality of sub-images according to the image comprises:

[0033] Based on the corrected image, multiple sub-images are generated.

[0034] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0035] A first acquisition module is used to acquire an image acquired by an image sensor and an event frame acquired by a dynamic vision sensor; wherein the event frame includes brightness change information of pixels in the image;

[0036] A determination module, used for determining whether the image has motion blur according to brightness change information in the event frame;

[0037] A generating module, configured to generate a plurality of sub-images according to the image when the image has motion blur; wherein the resolution of the sub-images is consistent with the resolution of the event frame;

[0038] A removal module is used to remove motion blur from the image according to the event frame and the multiple sub-images.

[0039] In some embodiments, the generation module is used to traverse the image using a sliding window and obtain pixel values ​​of pixels in the window in a preset color channel; wherein the preset color channels of adjacent pixels in the window are different; and the pixel values ​​of pixels at the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixels included in the sub-image.

[0040] In some embodiments, the number of traversals of the window is the resolution of the event frame, and the pixels in the window do not overlap at different traversal times.

[0041] In some embodiments, the removal module is used to process the event frame and the multiple sub-images using a preset motion blur removal model to obtain an image after removing motion blur; wherein the preset motion blur removal model is trained based on historical image samples and historical event frame samples.

[0042] In some embodiments, the apparatus further comprises:

[0043] A second acquisition module is used to acquire a plurality of historical image samples collected by an image sensor and a plurality of historical event frame samples collected by a dynamic vision sensor; wherein the historical event frame samples include brightness change information of pixels in corresponding historical image samples; and the resolution of the historical image samples is greater than a preset resolution threshold;

[0044] The first obtaining module is used to perform fuzzy processing on a plurality of historical image samples to obtain a plurality of fuzzy image samples;

[0045] The second obtaining module is used to train a preset initial model using multiple blurred image samples, multiple historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model.

[0046] In some embodiments, the second obtaining module is used to determine, for each blurred image sample, the motion characteristics of the object in the blurred image sample based on the historical event frame samples corresponding to the historical image samples associated with the blurred image sample; remove motion blur from the blurred image sample based on the motion characteristics; and adjust the parameters of the preset initial model based on the difference between each image after motion blur removal and a historical image sample associated with the blurred image sample before motion blur removal, so as to obtain the preset motion blur removal model.

[0047] In some embodiments, the determination module is used to process the brightness change information in the event frame using a preset motion blur detection model to obtain a result of whether the image has motion blur; wherein the preset motion blur detection model is obtained based on training of a neural network model.

[0048] In some embodiments, the apparatus further comprises:

[0049] A preprocessing module, used for preprocessing the brightness change information in the event frame; wherein the preprocessing includes at least one of the following: noise elimination processing, delay elimination processing;

[0050] The determination module is used to determine whether the image has motion blur according to the brightness change information in the preprocessed event frame.

[0051] In some embodiments, the apparatus further comprises:

[0052] A correction module, used to correct the image; wherein the correction includes at least one of the following: bad pixel correction, black level correction, lens shading correction;

[0053] The generating module is used to generate a plurality of sub-images according to the corrected image.

[0054] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0055] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method as described in the first aspect above.

[0056] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0057] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in the first aspect above.

[0058] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0059] The present disclosure determines whether there is motion blur in an image based on brightness change information in an event frame, and generates multiple sub-images with resolutions consistent with the resolution of the event frame based on the image if there is motion blur in the image, and then removes motion blur from the image based on the event frame and the multiple sub-images. On the one hand, since the resolution of an event frame is usually much lower than the resolution of an image, the present disclosure embodiment has a high accuracy in removing image motion blur based on event frames and sub-images with the same resolution. On the other hand, compared to the problem of increased computational complexity and greater difficulty in super-resolution caused by super-resolution of an event frame, the present disclosure embodiment decomposes the image into multiple sub-images so that the resolution of the sub-images is consistent with the resolution of the event frame. On the basis of being able to remove motion blur from the image based on sub-images and event frames with the same resolution, no new pixel information is introduced, which greatly reduces the computational complexity and has the advantage of being easy to implement.

[0060] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0062] Figure 1 An example of an image processing method flow shown in the embodiment of the present disclosure Figure 1 .

[0063] Figure 2 This is a diagram showing an example of a motion blur removal model structure according to an embodiment of the present disclosure.

[0064] Figure 3 An example of an image processing method flow shown in the embodiment of the present disclosure Figure 2 .

[0065] Figure 4 is a diagram of an image processing device shown in an embodiment of the present disclosure.

[0066] Figure 5 It is a block diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0068] In the related art, the image captured by the image sensor (CMOS Image Sensor, CIS) is combined with the event frame captured by the dynamic vision sensor (Dynamic Vision Sensor, DVS) to remove the motion blur in the image. Due to the limitations of the hardware design of the image sensor and the dynamic vision sensor, the resolution of the image captured by the image sensor is usually much greater than the resolution of the event frame captured by the dynamic vision sensor. In the related art, super-resolution technology is used, such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc., to improve the resolution of the event frame to be consistent with the resolution of the image, so as to process the event frame and the image with the same resolution and remove the motion blur in the image.

[0069] However, the amount of information in the event frame after super-resolution in this scheme increases, resulting in an increase in the amount of computational effort in image processing; and, since the event frame includes multi-dimensional information, the super-resolution technology for event frames is not yet mature, resulting in poor results in removing image motion blur.

[0070] In this regard, an embodiment of the present disclosure provides an image processing method, the execution subject of which may be an image processing device. For example, the image processing method may be executed by a terminal device or a server or other electronic device, wherein the terminal device may be a mobile phone, a camera, a tablet computer, a vehicle-mounted device, a wearable device, etc. If the execution subject is a server or other electronic device, after the terminal device acquires the image and the event frame, it sends them to the server or other electronic device, so that the server or other electronic device removes motion blur from the image based on the image and the event frame. In some possible implementations, the image processing method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0071] Figure 1 This is an example of an image processing method flow shown in the embodiment of the present disclosure Figure 1 ,Depend on Figure 1 It can be seen that the following steps are included:

[0072] S101, acquiring an image captured by an image sensor and an event frame captured by a dynamic vision sensor; wherein the event frame includes brightness change information of pixel points in the image;

[0073] S102, determining whether the image has motion blur according to brightness change information in the event frame;

[0074] S103, when there is motion blur in the image, generating a plurality of sub-images according to the image; wherein the resolution of the sub-images is consistent with the resolution of the event frame;

[0075] S104: Remove motion blur from the image according to the event frame and the multiple sub-images.

[0076] In step S101, the image processing device obtains an image captured by an image sensor and an event frame captured by a dynamic vision sensor, wherein the image sensor and the dynamic vision sensor may be separate or integrated (Hybrid DVS), which is not limited in the present disclosure.

[0077] It should be noted that the image sensor and the dynamic vision sensor in the embodiment of the present disclosure respectively collect images and event frames in the same scene, such as exposing an image in the same scene and collecting event frames during the exposure of the image. If the image sensor and the dynamic vision sensor are separate, the image sensor and the dynamic vision sensor need to be aligned in field of view and time calibrated.

[0078] Since there may be errors in field of view alignment and time calibration, or there may be situations where field of view alignment cannot be achieved, if the embodiment of the present disclosure adopts Hybrid DVS, the captured images and event frames are strictly aligned, so there is no need for calibration and registration, which helps to improve the effect of subsequent motion blur removal of images based on images captured by the image sensor and event frames captured by the dynamic vision sensor.

[0079] The event frame of the embodiment of the present disclosure includes brightness change information of pixels in the image, such as pixel coordinates, polarity of brightness change, and time information. As can be seen from the foregoing, due to the limitations of the hardware design of the image sensor and the dynamic vision sensor, the resolution of the image captured by the image sensor is usually greater than the resolution of the event frame captured by the dynamic vision sensor, such as the ratio of the image resolution to the event frame resolution is 16:1 or 8:1. Therefore, the event frame of the embodiment of the present disclosure only includes brightness change information of some pixels in the image.

[0080] As can be seen from the foregoing, due to the movement of the image sensor and / or the dynamic vision sensor, or the movement of the subject during the shooting process, the pixel information at different times will be mixed together, resulting in motion blur in the captured image. Therefore, relative to the still scene where the brightness of the pixels in the image remains unchanged or the brightness of a few pixels changes, in step S102, the image processing device determines whether the image has motion blur based on the brightness change information in the event frame. In some embodiments, the image processing device calculates the proportion of the number of pixels that have changed in brightness in the total number of pixels based on the event frame statistics, and determines that the image has motion blur when the proportion is greater than a preset proportion threshold.

[0081] In some embodiments, determining whether the image has motion blur according to the brightness change information in the event frame includes:

[0082] The brightness change information in the event frame is processed using a preset motion blur detection model to obtain a result of whether the image has motion blur; wherein the preset motion blur detection model is obtained based on training of a neural network model.

[0083] In the disclosed embodiment, the event frame is input into the motion blur detection model, and after the motion blur detection model processes the brightness change information in the event frame, it outputs the result of whether the image associated with the event frame has motion blur; wherein the preset motion blur detection model is obtained based on the training of the neural network model. For example, based on the event frame training samples and label values, the motion blur detection model is obtained after training and adjusting the parameters of networks such as Convolutional Neural Networks (CNN) and Deep Neural Networks (DNN); the label value is the preset existence of motion blur or the absence of motion blur.

[0084] It can be understood that since the motion blur detection model is obtained after multiple training convergence of the neural network, compared with determining whether an image has motion blur based on the proportion of the number of pixels that have changed in brightness based on event frame statistics in the total number of pixels, the embodiment of the present disclosure is based on the motion blur detection model. It can improve the accuracy of determining whether an image has motion blur, thereby increasing the necessity of subsequent motion blur removal of the image.

[0085] In addition, the image processing device of the disclosed embodiment can first determine whether there is motion of the subject in the image based on an optical flow detection algorithm, or based on whether the proportion of the number of pixels whose brightness has changed in the event frame statistics in the total number of pixels in the image is greater than a preset motion proportion threshold, etc.; and when there is motion of the subject in the image, use the aforementioned motion blur detection model to determine whether there is motion blur in the image, thereby increasing the necessity of detecting motion blur.

[0086] In step S103, when there is motion blur in the image, the image processing device generates multiple sub-images according to the image; wherein the resolution of the sub-images is consistent with the resolution of the event frame. In some embodiments, the image processing device divides the image into multiple regions according to the resolution of the event frame, the number of pixels in each region is consistent with the resolution of the event frame, and each region is used as a sub-image.

[0087] In some embodiments, the image processing device determines the ratio between the image resolution and the event frame resolution, and uses the pixel value of each consecutive pixel of the ratio in the image as the pixel value of a pixel in each sub-image; based on this scheme, the sub-images of the ratio can be obtained. For example, the ratio between the image resolution and the event frame resolution is 16:1; each consecutive 16 pixels in the image are divided into one portion, and for each portion, the pixel values ​​of the 16 pixels in one portion are used as the pixel values ​​of a pixel in 16 sub-images, such as using the pixel value of the first pixel in the 16 pixels as the pixel value of a pixel in the first sub-image, the pixel value of the second pixel as the pixel value of a pixel in the second sub-image, the pixel value of the third pixel as the pixel value of a pixel in the third sub-image, and so on; after the pixel values ​​of each pixel are processed as described above, 16 sub-images are obtained, and the resolution of each sub-image is consistent with the resolution of the event frame.

[0088] It can be understood that, compared with the aforementioned embodiment in which an area of ​​an image is taken as a sub-image, and each generated sub-image only represents a certain part of the image, the pixel values ​​of the pixels in each sub-image in the embodiment of the present disclosure are more evenly distributed relative to the image, so that the sub-image can carry pixel information of different areas of the image, thereby helping to improve the effect of subsequent motion blur removal of the image based on multiple sub-images and event frames.

[0089] After generating multiple sub-images according to the image, in step S104, the image processing device removes motion blur from the image according to the event frame and the multiple sub-images. In some embodiments, it can be seen from the above that motion blur is caused by the aliasing of pixel information at different spatial positions due to the movement of the image sensor and / or the dynamic visual sensor or the movement of the subject during the exposure process. In this regard, the embodiment of the present disclosure determines the target pixel point whose brightness changes in each sub-image according to the brightness change information of each pixel point in the sub-image included in the event frame, and the target pixel point is more likely to be the pixel point causing motion blur; thus, for each target pixel point, the pixel value of any neighboring pixel point of the target pixel point is used as the pixel value of the target pixel point, or the average pixel value of each neighboring pixel point of the target pixel point is used as the pixel value of the target pixel point, to obtain a sub-image after removing motion blur; and then, according to each sub-image after removing motion blur, an image after removing motion blur with a resolution consistent with the resolution of the image before processing is obtained. Among them, if the sub-image is generated based on a certain area of ​​the image, the sub-images after removing motion blur can be spliced ​​according to the various areas of the image to obtain the image after removing motion blur; if the sub-image is generated based on the pixel values ​​of pixel points that are relatively evenly distributed in the image, the pixel values ​​of the pixel points of each sub-image after removing motion blur can be evenly distributed based on the inverse process of generating the sub-image to obtain the image after removing motion blur.

[0090] In some embodiments, since the resolution of the event frame is consistent with the resolution of the sub-image, the brightness change information included in the event frame corresponds to each pixel in the sub-image. In this regard, the image processing device of the disclosed embodiment determines the target pixel of brightness change in the sub-image for each sub-image based on the brightness change information of each pixel in the sub-image included in the event frame, and the target pixel is more likely to be a pixel at the motion blur position; thus, for each target pixel, the pixel value of any pixel in other sub-images that is consistent with the coordinates of the target pixel is used as the pixel value of the target pixel, or the average of the pixel values ​​of each pixel in other sub-images that is consistent with the coordinates of the target pixel is used as the pixel value of the target pixel, to obtain a sub-image after removing motion blur; and then, based on each sub-image after removing motion blur, an image after removing motion blur with a resolution consistent with the resolution of the image before processing is obtained.

[0091] It can be understood that the embodiment of the present disclosure determines whether there is motion blur in the image based on the brightness change information in the event frame, and generates multiple sub-images with resolutions consistent with the resolution of the event frame based on the image when there is motion blur in the image, and then removes the motion blur from the image based on the event frame and the multiple sub-images. On the one hand, since the resolution of the event frame is usually much lower than the resolution of the image, the embodiment of the present disclosure has a high accuracy in removing the motion blur of the image based on the event frame and sub-images with the same resolution. On the other hand, compared with the problem of increased computational complexity and greater difficulty in super-resolution caused by super-resolution of the event frame, the embodiment of the present disclosure decomposes the image into multiple sub-images so that the resolution of the sub-images is consistent with the resolution of the event frame. On the basis of satisfying the requirement of removing the motion blur from the image based on the sub-images and event frames with the same resolution, no new pixel information is introduced, which greatly reduces the computational complexity and has the advantage of being easy to implement.

[0092] In some embodiments, generating a plurality of sub-images according to the image includes:

[0093] The image is traversed using a sliding window, and pixel values ​​of pixels in the window in a preset color channel are obtained; wherein the preset color channels of adjacent pixels in the window are different;

[0094] The pixel values ​​of the pixel points at the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixel points included in the sub-image.

[0095] In an embodiment of the present disclosure, an image processing device uses a sliding window to traverse an image, wherein the sliding window may be software-generated, such as a window filter, or may be hardware-based, such as a filter. The sliding window may include only one window or multiple windows, and the present disclosure does not limit this.

[0096] In the process of sliding the window through the image, the pixel values ​​of the pixels in the window in the preset color channel are obtained. For example, the preset color channel is red, and based on the sliding window, the pixel values ​​of the pixels in the window in the red channel are filtered. It should be noted that in the embodiments of the present disclosure, the preset color channels may be different for pixels at different positions, and it is necessary to satisfy the inconsistency of the preset color channels of adjacent pixels in the window. For example, the distribution of the preset color channels of pixels at different positions in the window is such as the Bayer format, or other distributions that satisfy the inconsistency of the preset color channels of adjacent pixels in the window.

[0097] It can be understood that the embodiment of the present disclosure traverses the image based on a preset sliding window to obtain pixel values ​​of each pixel point at adjacent positions on different color channels.

[0098] As the sliding window is traversed, the pixel values ​​of the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixels included in the sub-image. Taking the generation of the first sub-image as an example, if the preset color channel corresponding to the first pixel in the window is the red channel, in the first traversal, the pixel value of the first pixel in the window in the red channel is used as the pixel value of the first pixel in the first sub-image; in the second traversal, the pixel value of the first pixel in the window in the red channel is used as the pixel value of the second pixel in the first sub-image; in the third traversal, the pixel value of the first pixel in the window in the red channel is used as the pixel value of the third pixel in the first sub-image, and so on, to obtain the first sub-image, and the pixel values ​​of each pixel in the first sub-image are all pixel values ​​of the red channel. Since the preset color channels of adjacent pixels in the window are different, there are pixel values ​​of pixels in different color channels between each sub-image.

[0099] It can be understood that the embodiment of the present disclosure uses a sliding window to traverse the image and obtain the pixel values ​​of the pixels in the window in the preset color channel, so that the pixel values ​​of the pixels at the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixels included in the sub-image. On the one hand, compared with the motion blur removal based on each sub-image and event frame including pixel values ​​of multiple color channels, the sub-images generated by the embodiment of the present disclosure, such as RAW images, that is, the pixel values ​​of each pixel in the sub-image are the pixel values ​​of a single color channel, can further reduce the amount of calculation for motion blur removal based on each sub-image and event frame. On the other hand, it can be seen from the above that there are pixel values ​​of pixels in different color channels between each sub-image, which helps to improve the imaging effect of generating an image after motion blur removal based on the pixel values ​​of each pixel in different color channels after motion blur removal.

[0100] In some disclosed embodiments, the number of traversals of the window is the resolution of the event frame, and the pixels in the window do not overlap at different traversal numbers.

[0101] In the disclosed embodiment, when the preset sliding window slides through the image, the pixels in the window do not overlap at different traversal times, and the traversal times of the window are consistent with the resolution of the event frame. Taking the ratio of the image resolution to the event frame resolution as 16:1 as an example, the preset sliding window includes 16 sub-windows, and the number of non-overlapping traversals of the window is consistent with the resolution of the event frame, thereby obtaining 16 sub-images, and each sub-image is consistent with the resolution of the event frame. Through this solution, on the basis of ensuring that the resolution of the sub-image is consistent with the resolution of the event frame, since the traversed pixels do not overlap, the number of sliding traversals of the window can be reduced, thereby improving the efficiency of removing motion blur from the image.

[0102] In some embodiments, removing motion blur from the image according to the event frame and the plurality of sub-images includes:

[0103] The event frame and the multiple sub-images are processed using a preset motion blur removal model to obtain an image after removing motion blur; wherein the preset motion blur removal model is trained based on historical image samples and historical event frame samples.

[0104] In an embodiment of the present disclosure, an image processing device uses a preset motion blur removal model, inputs an event frame and multiple sub-images into the motion blur removal model, and obtains an image after removing motion blur; wherein the preset motion blur removal model is trained based on historical image samples and historical event frame samples.

[0105] The embodiment of the present disclosure may also convert the event frame into a voxel grid and then input it into a motion blur removal model; and / or input multiple sub-images into the motion blur removal model through different channels respectively.

[0106] Figure 2 This is an example diagram of a motion blur removal model structure shown in an embodiment of the present disclosure, comprising Figure 2 It can be seen that the motion blur removal model includes a convolution layer, a maximum pooling layer, a fully connected layer and an activation function; where, for example, the size of the first convolution layer is 224*224*64, the size of the second convolution layer is 112*112*128, the size of the third convolution layer is 56*56*256, the size of the fourth convolution layer is 28*28*512, the size of the fifth convolution layer is 14*14*512, the size of the maximum pooling layer is 7*7*512, and the size of the fully connected layer is 1*1*4096. It should be noted that Figure 2 The structure of the motion blur removal model shown is only an example. The motion blur removal model of the embodiment of the present disclosure may also be a neural network model with other structures.

[0107] The following describes how to use the motion blur removal model to process the image and sub-image, and compares the output size of the convolutional layer after processing the image and sub-image. Figure 2 Taking the structure shown in the figure as an example, if the size of the R (red) G (green) B (blue) image is 4096*3680*3 (3 represents RGB three channels), the size of the sub-image (such as the original image, RAWImage) is 1024*920*16 (16 represents 16 sub-images, corresponding to the 16 channels of the input motion blur removal model), assuming that the stride of the convolution layer is 1 and the number of pixel padding is 0, the output result size N after the convolution layer processing can be expressed by the following formula (1):

[0108]

[0109] Among them, W represents the input data size, F is the convolution kernel size, P is the number of pixel padding, and S is the step size of the convolution layer.

[0110] According to the above formula (1), the convolutional layer is Figure 2 In the motion blur removal model shown in the figure, the size of the first convolutional layer is 224*224*64, that is, taking F as 224 as an example, the size of the output feature map after the first convolutional layer processes the image is That is 3873; the size of the output feature map after the first convolutional layer processes the sub-image is That is, 801; it can be seen that the size of the output feature map after the first convolutional layer processes the image is about 4.8 times the size of the output feature map after processing the sub-image.

[0111] It can be understood that, on the one hand, after the first convolutional layer processes the sub-image, the size of the output feature map is greatly reduced; since the number of channels of the output feature map of each convolutional layer is only determined by the number of convolution kernels of the layer, that is, the number of channels of the output feature map is equal to the number of convolution kernels; and from the above, it can be seen that after the first convolutional layer processes the sub-image, the size of the output feature map is greatly reduced, so when the feature map output by the first convolutional layer is used as the input of the subsequent convolutional layer, the amount of calculation of the subsequent convolutional layer can be greatly reduced. Therefore, by performing multi-channel low-resolution processing on the sub-image, the size of the feature map output by each convolutional layer will be reduced due to the reduction in the size of the input sub-image, thereby greatly reducing the amount of calculation of the entire motion blur removal model.

[0112] On the other hand, after the pixel values ​​of the pixels in the image are generated into multiple sub-images according to the sliding window, each sub-image contains the pixel information of the pixels that are widely distributed in the image. Since the receptive field of the same network model is determined by the size of the convolution kernel of each layer and the network structure, the low-resolution sub-image has a larger receptive field than the image, so that the global features of the image can be extracted more abundantly, which helps to improve the effect of removing motion blur from the image.

[0113] In some embodiments, the method further comprises:

[0114] Acquire multiple historical image samples collected by an image sensor and multiple historical event frame samples collected by a dynamic vision sensor; wherein the historical event frame samples include brightness change information of pixel points in the corresponding historical image samples; and the resolution of the historical image samples is greater than a preset resolution threshold;

[0115] Performing fuzzy processing on a plurality of historical image samples to obtain a plurality of fuzzy image samples;

[0116] A preset initial model is trained using a plurality of blurred image samples, a plurality of historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model.

[0117] In the embodiment of the present disclosure, a plurality of historical image samples collected by an image sensor and a plurality of historical event frame samples collected by a dynamic vision sensor are obtained; wherein the image sensor is a high-resolution image sensor, so that it can be considered that there is no motion blur in the historical image samples collected by the high-resolution image sensor. The historical event frame samples and the historical image samples are collected in the same scene, so that the historical event frame samples include brightness change information of the pixels in the corresponding historical image samples.

[0118] The embodiments of the present disclosure perform blur processing on multiple historical image samples, such as performing blur processing on multiple historical image samples based on a Gaussian blur image algorithm, a salt and pepper blur image algorithm, a weighted average method, etc., to obtain multiple blurred image samples; wherein, one historical image sample may be blurred to obtain a blurred image sample; or multiple historical image samples may be blurred to obtain a blurred image sample, etc., and the present disclosure does not limit this.

[0119] After obtaining a plurality of blurred image samples, a preset initial model is trained using the plurality of blurred image samples, a plurality of historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model. For example, the plurality of blurred image samples and a plurality of historical event frame samples are used as training samples, and the self-training of the model is supervised by using at least one historical image sample; or based on a preset loss function, the loss between the output image of the initial model and at least one historical image sample is calculated, and then the initial model parameters are adjusted according to the loss to obtain the preset motion blur removal model.

[0120] It can be understood that the embodiment of the present disclosure generates blurred image samples based on historical image samples without motion blur, and then trains a preset initial model based on multiple blurred image samples, multiple historical event frame samples, and at least one historical image sample to obtain a preset motion blur removal model. This does not require manual labeling of label values, and can improve the efficiency of motion blur removal model training.

[0121] In some embodiments, the method of training a preset initial model using a plurality of blurred image samples, a plurality of historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model includes:

[0122] For each blurred image sample, determining a motion feature of an object in the blurred image sample according to a historical event frame sample corresponding to a historical image sample associated with the blurred image sample;

[0123] Performing motion blur removal on blurred image samples according to the motion characteristics;

[0124] According to the difference between each image after motion blur removal and a historical image sample associated with the blurred image sample before motion blur removal, the parameters of the preset initial model are adjusted to obtain the preset motion blur removal model.

[0125] In the embodiment of the present disclosure, the historical event frame samples corresponding to the historical image samples associated with the blurred image samples are event frames collected by the dynamic vision sensor during the process of exposing the historical image samples used to generate the blurred image samples.

[0126] In the embodiment of the present disclosure, for each blurred image sample, the initial model is used to extract features of the historical event frame samples corresponding to the historical image samples associated with the blurred image sample, and obtain the motion features of the objects in the blurred image sample; and the blurred image sample is deblurred according to the motion features to obtain an image after the blurred image sample is deblurred. The initial model is a neural network such as CNN, DNN, etc.

[0127] In some embodiments, the motion characteristics of the object include motion speed, so that the possible position of the object in the image can be determined based on the motion speed of the object, and then based on the position, combined with edge detection algorithms, foreground detection algorithms, etc., the motion blur of the blurred image sample can be removed.

[0128] In some embodiments, the motion characteristics of the object include angular velocity of motion, so that the angle of the object after change in the image is determined based on the angular velocity of motion of the object, and then based on this angle, combined with edge detection algorithms, foreground detection algorithms, etc., the motion blur of the blurred image sample is removed.

[0129] After performing motion blur removal on each blurred image, the disclosed embodiment determines, for each blurred image, the difference between each image after motion blur removal and a historical image sample associated with the blurred image sample before motion blur removal, wherein the difference can be characterized by a loss value determined based on a preset loss function; or it can be characterized by the percentage of pixel points at the same position whose pixel value difference is greater than a preset pixel value threshold between a historical image sample associated with the blurred image sample and the image after motion blur removal on the blurred image sample, etc.

[0130] The embodiment of the present disclosure adjusts the parameters of the preset initial model according to the differences determined based on the above method, so that the model gradually converges to obtain a preset motion blur removal model; wherein the parameters include weights, step sizes, etc.

[0131] It can be understood that in the process of training the model, the embodiment of the present disclosure determines, for each blurred image sample, the motion characteristics of the object in the blurred image sample based on the historical event frame samples corresponding to the historical image samples associated with the blurred image sample, and then performs motion blur removal on the blurred image sample based on the motion characteristics. This can improve the accuracy of locating the pixels that produce motion blur in the blurred image sample, and thus improve the effect of motion blur removal on the image based on the trained motion blur removal model.

[0132] In some embodiments, the method further comprises:

[0133] Preprocessing the brightness change information in the event frame; wherein the preprocessing includes at least one of the following: noise elimination processing and delay elimination processing;

[0134] The determining, according to the brightness change information in the event frame, whether the image has motion blur comprises:

[0135] It is determined whether the image has motion blur according to the brightness change information in the preprocessed event frame.

[0136] In an embodiment of the present disclosure, the image processing device preprocesses the brightness change information in the event frame before determining whether the image has motion blur based on the brightness change information in the event frame; wherein the preprocessing includes at least one of the following: noise elimination processing and delay elimination processing.

[0137] In some embodiments, the image processing device performs noise elimination processing on the brightness change information in the event frame. For example, the brightness change information in the event frame is grouped into a preset number, and the time neighborhood threshold is calculated; whether the newly arrived event is noise is determined based on the spatiotemporal characteristics of the arrived event, and the density matrix is ​​calculated to remove the brightness change information that is noise in the event frame. Alternatively, the brightness change information that is noise in the event frame is eliminated based on an enhanced deep learning super-resolution reconstruction network (Enhanced Deep Super-Resolutionnetwork, EDSR), etc.

[0138] In some embodiments, the image processing device performs delay elimination processing on the brightness change information in the event frame. For example, based on a preset delay compensation value, the time information in the brightness change information is compensated; wherein the preset delay compensation value can be the time delay of the dynamic visual sensor collecting the event frame determined based on multiple experiments.

[0139] It can be understood that the embodiments of the present disclosure perform noise elimination processing and / or delay elimination processing on the brightness change information in the event frame, which can improve the accuracy of the brightness change information in the event frame, and further improve the accuracy of determining whether there is motion blur in the image based on the brightness change information in the pre-processed event frame.

[0140] In some embodiments, the method further comprises:

[0141] Correcting the image; wherein the correction includes at least one of the following: bad pixel correction, black level correction, lens shading correction;

[0142] Generating a plurality of sub-images according to the image comprises:

[0143] Based on the corrected image, multiple sub-images are generated.

[0144] In the embodiment of the present disclosure, before generating multiple sub-images from the image, the image processing device performs at least one of the following correction processes on the image: bad pixel correction, black level correction, and lens shading correction. Through this solution, the accuracy of pixel information of pixel points in the image can be improved, thereby improving the accuracy of removing motion blur from the multiple sub-images generated from the corrected image and the event frame.

[0145] Figure 3 An example of an image processing method flow shown in the embodiment of the present disclosure Figure 2 ,Depend on Figure 3 It can be seen that the following steps are included:

[0146] S301, acquiring images and event frames;

[0147] In the embodiment of the present disclosure, an image captured by an image sensor and an event frame captured by a dynamic vision sensor are acquired based on Hybrid DVS; wherein the event frame includes brightness change information of pixels in the image;

[0148] S302, pre-processing the event frame;

[0149] In an embodiment of the present disclosure, the brightness change information in the event frame is preprocessed; wherein the preprocessing includes at least one of the following: noise elimination processing and delay elimination processing.

[0150] S303, detecting whether the image has motion blur; if so, executing step S306; if not, ending the process;

[0151] In the embodiment of the present disclosure, it is determined whether the image has motion blur according to the brightness change information in the event frame. For example, a preset motion blur detection model is used to process the brightness change information in the event frame to obtain a result of whether the image has motion blur; wherein the preset motion blur detection model is obtained based on the training of a neural network model.

[0152] If there is motion blur in the image, execute step S306; if there is no motion blur in the image, end the process.

[0153] S304, correcting the image;

[0154] In an embodiment of the present disclosure, the image is corrected; wherein the correction includes at least one of the following: bad pixel correction, black level correction, and lens shading correction.

[0155] S305, generating a plurality of sub-images according to the image;

[0156] In the disclosed embodiment, multiple sub-images are generated based on the corrected image. For example, a sliding window such as a Bayer format is used to traverse the corrected image, and the pixel values ​​of the pixels in the window in the preset color channel are obtained; wherein the preset color channels of adjacent pixels in the window are different; the pixel values ​​of the pixels at the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixels included in the sub-image. wherein the number of traversal times of the window is the resolution of the event frame, and the pixels in the window do not overlap under different traversal times, so that the resolution of each sub-image is consistent with the resolution of the event frame.

[0157] S306, motion blur removal;

[0158] In the disclosed embodiment, motion blur removal is performed on an image based on an event frame and a plurality of sub-images. For example, a preset motion blur removal model is used to process the event frame and the plurality of sub-images to obtain an image after motion blur removal; wherein the preset motion blur removal model is obtained by training based on historical image samples and historical event frame samples.

[0159] S307, image post-processing;

[0160] In the embodiments of the present disclosure, the image after the motion blur is removed may be subjected to image post-processing. For example, the size of the image after the motion blur is removed may be processed to match the image preview requirements; or the image after the motion blur is removed may be beautified to improve the image display effect, etc., to finally obtain a clear image after the motion blur is removed.

[0161] Figure 4 is a diagram of an image processing device shown in an embodiment of the present disclosure, comprising Figure 4 It is known that, including:

[0162] The first acquisition module 401 is used to acquire an image acquired by an image sensor and an event frame acquired by a dynamic vision sensor; wherein the event frame includes brightness change information of pixels in the image;

[0163] A determination module 402, configured to determine whether the image has motion blur according to brightness change information in the event frame;

[0164] A generating module 403 is used to generate a plurality of sub-images according to the image when the image has motion blur; wherein the resolution of the sub-images is consistent with the resolution of the event frame;

[0165] The removal module 404 is used to remove motion blur from the image according to the event frame and the multiple sub-images.

[0166] In some embodiments, the generation module 403 is used to traverse the image using a sliding window and obtain pixel values ​​of pixels in the window in a preset color channel; wherein the preset color channels of adjacent pixels in the window are different; and the pixel values ​​of pixels at the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixels included in the sub-image.

[0167] In some embodiments, the number of traversals of the window is the resolution of the event frame, and the pixels in the window do not overlap at different traversal times.

[0168] In some embodiments, the removal module 404 is used to process the event frame and the multiple sub-images using a preset motion blur removal model to obtain an image after removing motion blur; wherein the preset motion blur removal model is trained based on historical image samples and historical event frame samples.

[0169] In some embodiments, the apparatus further comprises:

[0170] The second acquisition module 405 is used to acquire a plurality of historical image samples collected by the image sensor and a plurality of historical event frame samples collected by the dynamic vision sensor; wherein the historical event frame samples include brightness change information of pixel points in the corresponding historical image samples; and the resolution of the historical image samples is greater than a preset resolution threshold;

[0171] A first obtaining module 406 is used to perform fuzzy processing on a plurality of historical image samples to obtain a plurality of fuzzy image samples;

[0172] The second obtaining module 407 is used to train a preset initial model using multiple blurred image samples, multiple historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model.

[0173] In some embodiments, the second obtaining module 407 is used to determine, for each blurred image sample, the motion characteristics of the object in the blurred image sample based on the historical event frame samples corresponding to the historical image samples associated with the blurred image sample; remove motion blur from the blurred image sample based on the motion characteristics; and adjust the parameters of the preset initial model based on the difference between each image after motion blur removal and a historical image sample associated with the blurred image sample before motion blur removal, so as to obtain the preset motion blur removal model.

[0174] In some embodiments, the determination module 402 is used to process the brightness change information in the event frame using a preset motion blur detection model to obtain a result of whether the image has motion blur; wherein the preset motion blur detection model is obtained based on training of a neural network model.

[0175] In some embodiments, the apparatus further comprises:

[0176] A preprocessing module 408 is used to preprocess the brightness change information in the event frame; wherein the preprocessing includes at least one of the following: noise elimination processing, delay elimination processing;

[0177] The determination module 402 is used to determine whether the image has motion blur according to the brightness change information in the pre-processed event frame.

[0178] In some embodiments, the apparatus further comprises:

[0179] A correction module 409, used to correct the image; wherein the correction includes at least one of the following: bad pixel correction, black level correction, lens shading correction;

[0180] The generating module 403 is used to generate a plurality of sub-images according to the corrected image.

[0181] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0182] Figure 5 8 is a block diagram of an electronic device 800 shown in an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a mobile computer, etc.

[0183] Reference Figure 5, the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0184] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0185] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0186] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0187] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0188] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0189] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0190] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0191] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0192] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0193] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0194] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the aforementioned image processing method.

[0195] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0196] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image captured by an image sensor and an event frame captured by a dynamic vision sensor; wherein the event frame includes brightness change information of pixels in the image; Determining whether the image has motion blur according to brightness change information in the event frame; In the case where the image has motion blur, a plurality of sub-images are generated according to the image; wherein the resolution of the sub-images is consistent with the resolution of the event frame; Motion blur removal is performed on the image according to the event frame and the multiple sub-images.

2. The method according to claim 1, characterized in that Generating a plurality of sub-images according to the image comprises: The image is traversed using a sliding window, and pixel values ​​of pixels in the window in a preset color channel are obtained; wherein the preset color channels of adjacent pixels in the window are different; The pixel values ​​of the pixel points at the same position in the window in the same preset color channel under different traversal times are used as the pixel values ​​of the pixel points included in the sub-image.

3. The method according to claim 2, characterized in that The number of traversal times of the window is the resolution of the event frame, and the pixels in the window do not overlap under different traversal times.

4. The method according to claim 1, characterized in that The removing motion blur from the image according to the event frame and the multiple sub-images comprises: The event frame and the multiple sub-images are processed using a preset motion blur removal model to obtain an image after removing motion blur; wherein the preset motion blur removal model is trained based on historical image samples and historical event frame samples.

5. The method according to claim 4, characterized in that The method further comprises: Acquire multiple historical image samples collected by an image sensor and multiple historical event frame samples collected by a dynamic vision sensor; wherein the historical event frame samples include brightness change information of pixel points in the corresponding historical image samples; and the resolution of the historical image samples is greater than a preset resolution threshold; Performing fuzzy processing on a plurality of historical image samples to obtain a plurality of fuzzy image samples; A preset initial model is trained using a plurality of blurred image samples, a plurality of historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model.

6. The method according to claim 5, characterized in that The method of training a preset initial model using a plurality of blurred image samples, a plurality of historical event frame samples, and at least one historical image sample to obtain the preset motion blur removal model includes: For each blurred image sample, determining a motion feature of an object in the blurred image sample according to a historical event frame sample corresponding to a historical image sample associated with the blurred image sample; Performing motion blur removal on blurred image samples according to the motion characteristics; According to the difference between each image after motion blur removal and a historical image sample associated with the blurred image sample before motion blur removal, the parameters of the preset initial model are adjusted to obtain the preset motion blur removal model.

7. The method according to claim 1, characterized in that The determining whether the image has motion blur according to the brightness change information in the event frame comprises: The brightness change information in the event frame is processed using a preset motion blur detection model to obtain a result of whether the image has motion blur; wherein the preset motion blur detection model is obtained based on training of a neural network model.

8. The method according to claim 1, characterized in that The method further comprises: Preprocessing the brightness change information in the event frame; wherein the preprocessing includes at least one of the following: noise elimination processing and delay elimination processing; The determining whether the image has motion blur according to the brightness change information in the event frame comprises: It is determined whether the image has motion blur according to the brightness change information in the preprocessed event frame.

9. The method according to claim 1, characterized in that: The method further comprises: Correcting the image; wherein the correction includes at least one of the following: bad pixel correction, black level correction, lens shading correction; Generating a plurality of sub-images according to the image comprises: Based on the corrected image, multiple sub-images are generated.

10. An image processing device, characterized in that: The device comprises: A first acquisition module is used to acquire an image acquired by an image sensor and an event frame acquired by a dynamic vision sensor; wherein the event frame includes brightness change information of pixels in the image; A determination module, used for determining whether the image has motion blur according to brightness change information in the event frame; A generating module, configured to generate a plurality of sub-images according to the image when the image has motion blur; wherein the resolution of the sub-images is consistent with the resolution of the event frame; A removal module is used to remove motion blur from the image according to the event frame and the multiple sub-images.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method as claimed in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 9.

Images